When I use albertModel, it prints the following repeatedly.
Author: ghost
Pull request: no

```python
0 0
Layer index 0
0 1
Layer index 0
0 2
Layer index 0
0 3
Layer index 0
0 4
Layer index 0
0 5
Layer index 0
0 6
Layer index 0
0 7
Layer index 0
0 8
Layer index 0
0 9
Layer index 0
0 10
Layer index 0
0 11
Layer index 0


0 0
Layer index 0
0 1
Layer index 0
0 2
Layer index 0
0 3
Layer index 0
0 4
Layer index 0
0 5
Layer index 0
0 6
Layer index 0
0 7
Layer index 0
0 8
Layer index 0
0 9
Layer index 0
0 10
Layer index 0
0 11
Layer index 0

.......
```
Is this a mistake?

Comments:
I encountered this issue when using apex mixed precision, and I put `amp.initialize` after wrapping the model in `DistributedDataParallel`, and I believe reversing the order to first call `amp.initialize` fixed it

I did not use mixed precision.

+1, also having this issue for *-v1 and *-v2 models. I'm not using mixed precision either.

I encountered the same problem. This problem can be fixed by removing line 289 and line 331 in modeling_albert.py (Those lines are not existed in this repo):
```
class AlbertLayerGroup(nn.Module):
    def __init__(self, config):
        super(AlbertLayerGroup, self).__init__()

        self.output_attentions = config.output_attentions
        self.output_hidden_states = config.output_hidden_states
        self.albert_layers = nn.ModuleList([AlbertLayer(config) for _ in range(config.inner_group_num)])

    def forward(self, hidden_states, attention_mask=None, head_mask=None):
        layer_hidden_states = ()
        layer_attentions = ()

        for layer_index, albert_layer in enumerate(self.albert_layers):
            layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index])
            hidden_states = layer_output[0]

            print("Layer index", layer_index)

            if self.output_attentions:
            ...
```
```
class AlbertTransformer(nn.Module):
    def __init__(self, config):
        super(AlbertTransformer, self).__init__()

        self.config = config
        self.output_attentions = config.output_attentions
        self.output_hidden_states = config.output_hidden_states
        self.embedding_hidden_mapping_in = nn.Linear(config.embedding_size, config.hidden_size)
        self.albert_layer_groups = nn.ModuleList([AlbertLayerGroup(config) for _ in range(config.num_hidden_groups)])

    def forward(self, hidden_states, attention_mask=None, head_mask=None):
        hidden_states = self.embedding_hidden_mapping_in(hidden_states)

        all_attentions = ()

        if self.output_hidden_states:
            all_hidden_states = (hidden_states,)

        for i in range(self.config.num_hidden_layers):
            # Number of layers in a hidden group
            layers_per_group = int(self.config.num_hidden_layers / self.config.num_hidden_groups)

            # Index of the hidden group
            group_idx = int(i / (self.config.num_hidden_layers / self.config.num_hidden_groups))

            # Index of the layer inside the group
            layer_idx = int(i - group_idx * layers_per_group)

            print(group_idx, layer_idx)

            ...

```

>             print(group_idx, layer_idx)

Thanks!