Different ouputs from code and Hosted Inference API
Author: moniquebm
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 3.5.1
- Platform: Windows-10-10.0.19041-SP0
- Python version: 3.8.5
- PyTorch version (GPU?): 1.6.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

## To reproduce

Steps to reproduce the behavior:

1. Run the following code:

```python
from transformers import BertForTokenClassification, DistilBertTokenizerFast, pipeline

model = BertForTokenClassification.from_pretrained('monilouise/ner_pt_br')
tokenizer = DistilBertTokenizerFast.from_pretrained('neuralmind/bert-base-portuguese-cased', model_max_length=512, do_lower_case=False)
nlp = pipeline('ner', model=model, tokenizer=tokenizer)
result = nlp("O Tribunal de Contas da União é localizado em Brasília e foi fundado por Rui Barbosa.  Fiscaliza contratos, por exemplo com empresas como a Veigamed e a Buyerbr.")
print(result)
````

It'll ouput:

[{'word': 'Tribunal', 'score': 0.9858521819114685, 'entity': 'B-PUB', 'index': 2}, {'word': 'de', 'score': 0.9954801201820374, 'entity': 'I-PUB', 'index': 3}, {'word': 'Contas', 'score': 0.9929609298706055, 'entity': 'I-PUB', 'index': 4}, {'word': 'da', 'score': 0.9949454665184021, 'entity': 'I-PUB', 'index': 5}, {'word': 'União', 'score': 0.9913719296455383, 'entity': 'L-PUB', 'index': 6}, {'word': 'Brasília', 'score': 0.9405767321586609, 'entity': 'B-LOC', 'index': 10}, {'word': 'Rui', 'score': 0.979736328125, 'entity': 'B-PESSOA', 'index': 15}, {'word': 'Barbosa', 'score': 0.988306999206543, 'entity': 'L-PESSOA', 'index': 16}, {'word': 'Veiga', 'score': 0.9748793244361877, 'entity': 'B-ORG', 'index': 29}, {'word': '##med', 'score': 0.9309309124946594, 'entity': 'L-ORG', 'index': 30}, {'word': 'Bu', 'score': 0.9679405689239502, 'entity': 'B-ORG', 'index': 33}, {'word': '##yer', 'score': 0.6654638051986694, 'entity': 'L-ORG', 'index': 34}, {'word': '##br', 'score': 0.9575732350349426, 'entity': 'L-ORG', 'index': 35}]

including all entity types (PUB, PESSOA, ORG and LOC)

2. In Hosted Inference API, the following result is returned for the same sentence, ignoring PUB entity type and giving incorrect and incomplete results:

```json
[
  {
    "entity_group": "LOC",
    "score": 0.8127626776695251,
    "word": "bras"
  },
  {
    "entity_group": "PESSOA",
    "score": 0.7101765692234039,
    "word": "rui barbosa"
  },
  {
    "entity_group": "ORG",
    "score": 0.7679458856582642,
    "word": "ve"
  },
  {
    "entity_group": "ORG",
    "score": 0.45047426223754883,
    "word": "##igamed"
  },
  {
    "entity_group": "ORG",
    "score": 0.8467527627944946,
    "word": "bu"
  },
  {
    "entity_group": "ORG",
    "score": 0.6024420410394669,
    "word": "##yerbr"
  }
]
```

## Expected behavior

How can it be possible the same model file give different results?(!) May I be missing anything?


Comments:
Hi @moniquebm .

The hosted inference is running `nlp = pipeline('ner', model=model, tokenizer=tokenizer, grouped_entities=True)` by default.
There is currently no way to overload it.

Does that explain the difference? 

Hi @Narsil 

>Does that explain the difference?

I'm afraid it doesn't explain... I've just tested nlp = pipeline('ner', model=model, tokenizer=tokenizer, grouped_entities=True) and the following (correct) result is generated programmatically:

[{'entity_group': 'PUB', 'score': 0.9921221256256103, 'word': 'Tribunal de Contas da União'}, {'entity_group': 'LOC', 'score': 0.9405767321586609, 'word': 'Brasília'}, {'entity_group': 'PESSOA', 'score': 0.9840216636657715, 'word': 'Rui Barbosa'}, {'entity_group': 'ORG', 'score': 0.9529051184654236, 'word': 'Veigamed'}, {'entity_group': 'ORG', 'score': 0.8636592030525208, 'word': 'Buyerbr'}]

But in fact it seems the API does not group entities:

```json
[
  {
    "entity_group": "LOC",
    "score": 0.8127626776695251,
    "word": "bras"
  },
  {
    "entity_group": "PESSOA",
    "score": 0.7101765692234039,
    "word": "rui barbosa"
  },
  {
    "entity_group": "ORG",
    "score": 0.7679458856582642,
    "word": "ve"
  },
  {
    "entity_group": "ORG",
    "score": 0.45047426223754883,
    "word": "##igamed"
  },
  {
    "entity_group": "ORG",
    "score": 0.8467527627944946,
    "word": "bu"
  },
  {
    "entity_group": "ORG",
    "score": 0.6024420410394669,
    "word": "##yerbr"
  }
]
```

The tokens are also different.  One possible explaination is that the Hosted Inference API may be using English tokenizer, but my model/code used Portuguese tokenizer from this model: https://huggingface.co/neuralmind/bert-base-portuguese-cased

Does it make sense?

You need to update your tokenizer on your model in the hub: 'monilouise/ner_pt_br' to reflect this. The hosted inference can't know how to use a different tokenizer than the one you provide.

If you are simply using the one from `neuralmind/bert-base-portuguese-case`, you probably just download theirs, and reupload it as your own following this doc: https://huggingface.co/transformers/model_sharing.html

May I suggest moving the discussion here: 
https://discuss.huggingface.co/c/intermediate/6
As it's not really and transformers problem but a Hub one. 

I am closing the issue here. Feel free to comment here to show the new location of the discussion or ping me directly on discuss.