text: 
  
  TITLE
  Fill-mask pipeline removes space after token prediction when loading pre-training model based on roberta-base
  
  COMMENTS
  3
  
  REACTIONS
  +1: 0
  -1: 0
  laugh: 0
  hooray: 0
  heart: 0
  rocket: 0
  eyes: 0
  
  BODY
  ## Environment info
  - `transformers` version: 3.5.0
  - Platform: Linux-4.15.0-122-generic-x86_64-with-Ubuntu-18.04-bionic
  - Python version: 3.7.5
  - PyTorch version (GPU?): 1.7.0+cu101 (True)
  - Tensorflow version (GPU?): not installed (NA)
  - Using GPU in script?: False
  - Using distributed or parallel set-up in script?: False
  
  ### Who can help @mfuntowicz
  
  
  ## Information
  
  Model I am using (Bert, XLNet ...): **roberta-base**
  
  The problem arises when using:
  * [x] the official example scripts:
  -> continued **pre-training roberta-base** using the v3.5.0 examples/language-modeling/**run_mlm_wwm.py**
  * [x] my own modified scripts:
  To test the model during training I simply instantiate a pipeline class with target pretraining checkpoint folder and input masked strings to check the probabilities:
  ```
  unmasker = pipeline('fill-mask', model=model_checkpoint_path)
  results = unmasker(masked_text)
  print(json.dumps(results, indent=4))
  ```
  
  ## Expected behavior
  
  The expected behavior for input string "The goal of MASK is happiness." if loading model "roberta-base" would be:
  [
      {
          "sequence": "The goal of life is happiness.",
          "score": 0.07787031680345535,
          "token": 301,
          "token_str": "\u0120life"
      },
      {
          "sequence": "The goal of meditation is happiness.",
          "score": 0.040741581469774246,
          "token": 20183,
          "token_str": "\u0120meditation"
      }
  ]
  
  
  ## Observed behavior
  
  For the same input string I obtain no space following the predicted token when loading the further pre-trained model from a checkpoint folder, example result:
   [
      {
          "sequence": "The goal of Kiwis happiness.",
          "score": 0.11430764198303223,
          "token": 21472,
          "token_str": "\u0120Kiw"
      },
      {
          "sequence": "The goal of anis happiness.",
          "score": 0.04334629327058792,
          "token": 41,
          "token_str": "\u0120an"
      },
      {
          "sequence": "The goal of buis happiness.",
          "score": 0.03720756620168686,
          "token": 10306,
          "token_str": "\u0120bu"
      }
  ]
  
  As I thought it might be a tokenizer issue from the checkpoint, I tried to specify one from the roberta-base model used to continue the pre-training from. It solves the issue, so it seems like the pre-training steps have corrupted the tokenizer as loaded from the checkpoint. The results I get after 200,000 steps of additional pretraining from roberta-base:
  [
      {
          "sequence": "The goal of this is happiness.",
          "score": 0.28572556376457214,
          "token": 42,
          "token_str": "\u0120this"
      },
      {
          "sequence": "The goal of it is happiness.",
          "score": 0.10664933174848557,
          "token": 24,
          "token_str": "\u0120it"
      },
      {
          "sequence": "The goal of all is happiness.",
          "score": 0.07055338472127914,
          "token": 70,
          "token_str": "\u0120all"
      },
      {
          "sequence": "The goal of life is happiness.",
          "score": 0.056005414575338364,
          "token": 301,
          "token_str": "\u0120life"
      }
  ]
  
  -> so regardless of the quality of the result token prediciton, loading the roberta-base original tokenizer solves the issue.