First hidden state of the last layer of Bert (french version : FlauBert) only prints vectors of 0 or -0 after using it !!
Author: keloemma
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.4.2
- Platform: linux
- Python version: 3.7.6
- PyTorch version (GPU?): 1.5.0a0
- Tensorflow version (GPU?):
- Using GPU in script?: no (for on small sample ~25 sentences) but yes ( for full sample ~70K sentences/paragraphs)
- Using distributed or parallel set-up in script?: no 

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people. ! -->

@LysandreJik 
@sgugger 





# Library 
python  3.7.6
numpy == 1.18.4
pandas == 1.0.3
transformers == 4.6.1
torch :1.5.0a0
scipy == 1.4.1
sklearn== 0.22.1


Model hub:

- for issues with a model report at https://discuss.huggingface.co/ and tag the model's creator.

 -->

## Information

Model I am using (FlauBERT ...):

The problem arises when using FlauBert model to get first hidden state (I do not know if it is an error but the hidden states produce by the model download from transformers library produce the same value (0 or -0) for all my data after applying the model. maybe it is what it is supposed to produce but the behaviour is recurrent for all the sentence of my dataset ( ~70K).

* [ ] the official example scripts: (give details below)

        import torch
        from transformers import FlaubertModel, FlaubertTokenizer
        flaubert, log = FlaubertModel.from_pretrained(modelname, output_loading_info=True)
        flaubert_tokenizer = FlaubertTokenizer.from_pretrained(modelname, do_lowercase=False)
        sentence = "Le chat mange une pomme."
        token_ids = torch.tensor([flaubert_tokenizer.encode(sentence)])
        last_layer = flaubert(token_ids)[0]
        print(last_layer.shape)
        cls_embedding = last_layer[:, 0, :]

* [ ] my own modified scripts: (give details below)

        def spliterate(buf, chunk):
            for start in range(0, buf.size, chunk):
                yield buf[start:start + chunk]

          path_to_lge = "flaubert/flaubert_small_cased"

          def get_flaubert_layer(texte, path_to_lge):
          lge_size = path_to_lge[33:-1]
          print("Embeddings bert model used.................... : ", lge_size, "\n")
          flaubert = FlaubertModel.from_pretrained(path_to_lge)
          flaubert_tokenizer = FlaubertTokenizer.from_pretrained(path_to_lge)
          print(texte)
          tokenized = texte.apply(
            (lambda x: flaubert_tokenizer.encode(x, add_special_tokens=True, max_length=512, truncation=True)))
          print("Exit after applying tokenizer :" , "\n", tokenized, "\n")
          max_len = 0
          for i in tokenized.values:
            if len(i) > max_len:
                max_len = len(i)
          padded = np.array([i + [0] * (max_len - len(i)) for i in tokenized.values])
          print("Exit after padding: ", "\n", padded, "\n")
          last_layer_ = []
          for tmp in spliterate(padded, 4000):
            #print(tmp)
            if len(tmp) != 0:
                #print(len(tmp))
                token_ids = torch.tensor(tmp)
                print("Exit after torch transformation:" , "\n", token_ids, "\n")
                attention_mask = np.where(tmp != 0, 1, 0)
                attention_mask = torch.tensor(attention_mask)
                print("Exit after torch transformation for attention_mask:", "\n", attention_mask, "\n")
                with torch.no_grad():
                    layer = flaubert(token_ids, attention_mask=attention_mask)
                    layer = layer[0][:, 0, :].numpy()
                    print(" After applying model flaubert to get features :", "\n", layer, "\n")
                    last_layer_.append(layer)
            else:
                None
          last_layer_np = np.array(last_layer_)
          last_layer_np_array = np.concatenate(last_layer_np)
          # print("Total of sentences  :" , len(last_layer_np_array))
          
          return last_layer_np_array, lge_size

          #For getting hidden state (last layer)
          #Xtest is a dataframe column and look like this : 

          0     si j’ai un problème, comment je remonte l’info...
          1     des agents de maintenance ? Oui, oui. Enfin… I...
          2     Il faudrait des tiroirs qui sortent / rentrent...
          3     ROI, 5 à 10 ans. Si l’énergie explose, ça devi...
          4     Je ne vois pas cela en conception de cuisine, ...
          5     Les proverbes. C'est drôle parce qu'on a déjà ...
          6     J'ai l'impression que ça peut-être utilisable ...
          7     Ça reste… ça serait un réfrigérateur comme on ...
          8     C’est en plastique souple et on arrive vraimen...
          9     Parce que déjà, là, on évite de mettre un cong...
          10    En rénovation, il n'est pas évident de rajoute...

           Xtest_emb, s = get_hidden_state(Xtest, path_to_model_lge)

          # What data(token_ids) looks after transforming it to tensor before giving to flaubert model 
          [0,   93,  106,  ...,    0,    0,    0],
          [   0,   23, 2080,  ...,    0,    0,    0],
          [   0,   59, 2961,  ...,    0,    0,    0],
          ...,
          [   0,   55,  369,  ...,    0,    0,    0],
          [   0, 6077,   53,  ...,    0,    0,    0],
          [   0,   46,   41,  ...,    0,    0,    0]])
          #What layer[0] look like after using flaubert model on the data :
          tensor([[[-0.0000, -0.0000, -0.0000,  ..., -0.0000, -0.0000, -0.0000],
         [-0.0932,  0.7302,  1.3671,  ..., -2.5153, -2.0251,  1.2235],
         [-0.4520,  2.3935,  1.0048,  ..., -3.4742, -0.4194,  0.4428],
         ...,
         [-0.0000, -0.0000, -0.0000,  ..., -0.0000, -0.0000, -0.0000],
         [-0.0000, -0.0000, -0.0000,  ..., -0.0000, -0.0000, -0.0000],
         [-0.0000, -0.0000, -0.0000,  ..., -0.0000, -0.0000, -0.0000]],

        [[-0.0000, -0.0000,  0.0000,  ...,  0.0000, -0.0000, -0.0000],
         [ 2.0035,  0.6900, -0.5092,  ...,  0.0862, -1.6157,  0.6070],
         [-0.3516,  2.5931, -1.6113,  ..., -0.6265,  0.6559, -0.9409],
         ...,
         [-0.0000, -0.0000,  0.0000,  ...,  0.0000, -0.0000, -0.0000],
         [-0.0000, -0.0000,  0.0000,  ...,  0.0000, -0.0000, -0.0000],
         [-0.0000, -0.0000,  0.0000,  ...,  0.0000, -0.0000, -0.0000]],

        [[-0.0000, -0.0000,  0.0000,  ..., -0.0000,  0.0000, -0.0000],
         [-0.0040, -1.7643, -1.5588,  ..., -1.8786, -0.4597,  0.3843],
         [-3.4181,  0.1528,  0.6369,  ..., -2.2618, -1.0742, -0.6097],
         ...,
         [-0.0000, -0.0000,  0.0000,  ..., -0.0000,  0.0000, -0.0000],
         [-0.0000, -0.0000,  0.0000,  ..., -0.0000,  0.0000, -0.0000],
         [-0.0000, -0.0000,  0.0000,  ..., -0.0000,  0.0000, -0.0000]],
        ..,
        [[-0.0000,  0.0000, -0.0000,  ..., -0.0000, -0.0000,  0.0000],
         [-1.3317, -0.3899,  0.2560,  ..., -1.7550, -1.9626,  0.3821],
         [-1.3053, -0.2642,  0.1691,  ..., -1.8541, -2.1521,  0.6066],
         ...,
         [-0.0000,  0.0000, -0.0000,  ..., -0.0000, -0.0000,  0.0000],
         [-0.0000,  0.0000, -0.0000,  ..., -0.0000, -0.0000,  0.0000],
         [-0.0000,  0.0000, -0.0000,  ..., -0.0000, -0.0000,  0.0000]],

        [[-0.0000, -0.0000, -0.0000,  ...,  0.0000, -0.0000,  0.0000],
         [-0.8575, -2.6781,  1.0530,  ...,  0.7656, -2.3176,  0.6474],
         [ 0.5465,  0.1727, -0.8362,  ..., -0.1918, -1.5318,  1.0457],
         ...,
         [-0.0000, -0.0000, -0.0000,  ...,  0.0000, -0.0000,  0.0000],
         [-0.0000, -0.0000, -0.0000,  ...,  0.0000, -0.0000,  0.0000],
         [-0.0000, -0.0000, -0.0000,  ...,  0.0000, -0.0000,  0.0000]],
        [[ 0.0000, -0.0000,  0.0000,  ..., -0.0000, -0.0000, -0.0000],
         [-0.1900, -1.6420, -0.7254,  ..., -1.5700, -1.1521, -0.0588],
         [-0.7427, -2.5433,  0.6748,  ..., -3.1792, -1.8242,  0.4684],
         ...,
         [ 0.0000, -0.0000,  0.0000,  ..., -0.0000, -0.0000, -0.0000],
         [ 0.0000, -0.0000,  0.0000,  ..., -0.0000, -0.0000, -0.0000],
         [ 0.0000, -0.0000,  0.0000,  ..., -0.0000, -0.0000, -0.0000]]])

        What the result look like : 
        print(layer = layer[0][:, 0, :].numpy())
        [[-0. -0. -0. ... -0. -0. -0.]
        [-0. -0.  0. ...  0. -0. -0.]
        [-0. -0.  0. ... -0.  0. -0.]
        ...
        [-0.  0. -0. ... -0. -0.  0.]
        [-0. -0. -0. ...  0. -0.  0.]
        [ 0. -0.  0. ... -0. -0. -0.]]


The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)
Extractiing features  (get the hidden layer and pass it to a MLPerceptron from scikit learn for a classification task)

## To reproduce

Steps to reproduce the behavior:

1. Get the different libraries mentionned above in a virtual environnement
2. get the small example of Xtest given above in a dataframe column and pass it to the 
function "get_hidden_state" and print the results


<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
I did not expect the value of the first hidden state to be zero values , so I maybe using the flaubert the wrong way or  I did download the bad version


Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.