Fill-mask pipeline removes space after token prediction when loading pre-training model based on roberta-base
Author: ironflood
Pull request: no
Labels: name: wontfix

## Environment info
- `transformers` version: 3.5.0
- Platform: Linux-4.15.0-122-generic-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.5
- PyTorch version (GPU?): 1.7.0+cu101 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: False
- Using distributed or parallel set-up in script?: False

### Who can help @mfuntowicz


## Information

Model I am using (Bert, XLNet ...): **roberta-base**

The problem arises when using:
* [x] the official example scripts:
-> continued **pre-training roberta-base** using the v3.5.0 examples/language-modeling/**run_mlm_wwm.py**
* [x] my own modified scripts:
To test the model during training I simply instantiate a pipeline class with target pretraining checkpoint folder and input masked strings to check the probabilities:
```
unmasker = pipeline('fill-mask', model=model_checkpoint_path)
results = unmasker(masked_text)
print(json.dumps(results, indent=4))
```

## Expected behavior

The expected behavior for input string "The goal of MASK is happiness." if loading model "roberta-base" would be:
[
    {
        "sequence": "The goal of life is happiness.",
        "score": 0.07787031680345535,
        "token": 301,
        "token_str": "\u0120life"
    },
    {
        "sequence": "The goal of meditation is happiness.",
        "score": 0.040741581469774246,
        "token": 20183,
        "token_str": "\u0120meditation"
    }
]


## Observed behavior

For the same input string I obtain no space following the predicted token when loading the further pre-trained model from a checkpoint folder, example result:
 [
    {
        "sequence": "The goal of Kiwis happiness.",
        "score": 0.11430764198303223,
        "token": 21472,
        "token_str": "\u0120Kiw"
    },
    {
        "sequence": "The goal of anis happiness.",
        "score": 0.04334629327058792,
        "token": 41,
        "token_str": "\u0120an"
    },
    {
        "sequence": "The goal of buis happiness.",
        "score": 0.03720756620168686,
        "token": 10306,
        "token_str": "\u0120bu"
    }
]

As I thought it might be a tokenizer issue from the checkpoint, I tried to specify one from the roberta-base model used to continue the pre-training from. It solves the issue, so it seems like the pre-training steps have corrupted the tokenizer as loaded from the checkpoint. The results I get after 200,000 steps of additional pretraining from roberta-base:
[
    {
        "sequence": "The goal of this is happiness.",
        "score": 0.28572556376457214,
        "token": 42,
        "token_str": "\u0120this"
    },
    {
        "sequence": "The goal of it is happiness.",
        "score": 0.10664933174848557,
        "token": 24,
        "token_str": "\u0120it"
    },
    {
        "sequence": "The goal of all is happiness.",
        "score": 0.07055338472127914,
        "token": 70,
        "token_str": "\u0120all"
    },
    {
        "sequence": "The goal of life is happiness.",
        "score": 0.056005414575338364,
        "token": 301,
        "token_str": "\u0120life"
    }
]

-> so regardless of the quality of the result token prediciton, loading the roberta-base original tokenizer solves the issue.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Could this be reopened. I'm facing the same issue.

@sk- Same. Still getting this with `distilroberta-base`

Could you solve it? 

Hi @Bachstelze, if you're still experiencing this problem, could you open a new issue? Since the original issue is old and there have been many changes to the modeling and tokenization code we can't be sure the same thing is being addressed.