text: 
  
  TITLE
  Docs suggest to use discriminator weights for ElectraForMaskedLM instead of generator
  
  COMMENTS
  2
  
  REACTIONS
  +1: 0
  -1: 0
  laugh: 0
  hooray: 0
  heart: 0
  rocket: 0
  eyes: 0
  
  BODY
  ## Environment info
  
  - `transformers` version: 4.2.2
  - Platform: Linux-5.3.0-64-generic-x86_64-with-Ubuntu-19.10-eoan
  - Python version: 3.7.5
  - PyTorch version (GPU?): 1.6.0 (False)
  - Tensorflow version (GPU?): not installed (NA)
  - Using GPU in script?: no
  - Using distributed or parallel set-up in script?: no
  
  ### Who can help
   @LysandreJik
  
  ## Information
  
  Model I am using (Bert, XLNet ...): ELECTRA
  
  The problem arises when using:
  * [x] the official example scripts: https://huggingface.co/transformers/model_doc/electra.html#codecell3
  * [ ] my own modified scripts: (give details below)
  
  The tasks I am working on is:
  * [ ] an official GLUE/SQUaD task: (give the name)
  * [x] my own task or dataset: I want to check if a some word fits into the context of a sentence, for which I use the prediction probability of that word at the MASK position
  
  ## To reproduce
  
  Steps to reproduce the behavior:
  
  1. `from transformers import ElectraForMaskedLM`
  2. `model = ElectraForMaskedLM.from_pretrained('google/electra-small-discriminator')`
  
  ## My issue
  
  I get the following warning: 
  
  `Some weights of ElectraForMaskedLM were not initialized from the model checkpoint at google/electra-small-discriminator and are newly initialized: ['generator_predictions.LayerNorm.weight', 'generator_predictions.LayerNorm.bias', 'generator_predictions.dense.weight', 'generator_predictions.dense.bias', 'generator_lm_head.weight', 'generator_lm_head.bias']`
  
  I understand that I'm loading the discriminator weights, whereas ElectraForMaskedLM needs the generator weights for the MLM output. Why do the docs tell me to use the discriminator? Am I missing something?
  
  
  