BertModel sometimes produces the same output during evaluation
Author: xuesong39
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
I finetune BertModel as a part of my model to produce word embeddings. I found sometimes the performance was very bad, then i run the code again without any change, the performance was normal. It is very strange. I check my code to try to find the bug. I found the word embeddings produced by BertModel were all the same. Then I followed the code of BertModel, and found the BertEncoder would make the output become similar gradually which was consisted of 12 BertLayers. I have no idea about this situation.

Comments:
Did you set a fixed seed? If you want deterministic results, you should set a fixed seed. 

> Did you set a fixed seed? If you want deterministic results, you should set a fixed seed.

No, I didn't. My problem seems that the model finetuned is totally wrong sometimes. Is it a problem related to random seed?

What do you mean by "bad"? Post some code to better help you.

> What do you mean by "bad"? Post some code to better help you.

For example, the normal performance is 140.1, but the bad performance is 1.0. As I mentioned before, this situation happened sometimes and I found the word embeddings produced by BertModel were all the same when the performance was bad.

The text encoding part of my code is as follows:
 
![image](https://user-images.githubusercontent.com/37296256/71330710-697fd300-2569-11ea-8839-583502b8bcdc.png)


I think you are accessing the CLS token embeddings and It is constant as the model you are using is trained on MLM objective. 

> I think you are accessing the CLS token embeddings and It is constant as the model you are using is trained on MLM objective.

If I change the out to self.bert(text)[0][:,1,:], the output is the same as self.bert(text)[0][:,0,:]. It seems I get the same output no matter the input that I put in.

I saved the checkpoint when this situation happened. I multiplied all the parameter values of bert by 10 and found the outputs were different, while I divided all parameter values by 10 and found the outputs were almost same. So I think the reason is that the parameter values are too small.

@xuesong0309  https://github.com/huggingface/transformers/issues/1465，Could you look at this problem?

> @xuesong0309  https://github.com/huggingface/transformers/issues/1465，Could you look at this problem?

Did you always get same output? I suggest outputing the parameter values of your model as I mentioned above.

> > @xuesong0309  [https://github.com/huggingface/transformers/issues/1465，Could](https://github.com/huggingface/transformers/issues/1465%EF%BC%8CCould) you look at this problem?
> 
> Did you always get same output? I suggest outputing the parameter values of your model as I mentioned above.

It may not be the problem you mentioned, because the model is normal for multi-class classification, and this happens only for multi-label classification.

> @xuesong0309  https://github.com/huggingface/transformers/issues/1465，Could you look at this problem?
> 
> Did you always get same output? I suggest outputing the parameter values of your model as I mentioned above.
> 
> It may not be the problem you mentioned, because the model is normal for multi-class classification, and this happens only for multi-label classification.

You could open a new issue to describe your problem in detail.

Have you solved this issue? I have facing the same issue. Output same results during evaluation.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.