Bert uncased and Bert large giving much lower results than Bert cased base
Author: kh522
Pull request: no

Is there a reason why the Bert uncased model and the Bert large model give lower results that the cased model on downstream tasks?

Comments:
Any specific example that we could investigate?

I've implemented a version of SQuAD 2.0 on top of the current SQuAD that is similar to the way Google implemented their's on the official Bert repo. The base cased model works fine, but I noticed that uncased models tend to give worse results, even the large model.

@kh522 would love to try it out. Are you planing to share your code?

Sorry, but not quite yet. I was wondering if anyone had an intuition behind the error. If I recall correctly the SQuAD file lowercases the inputs for the tokenizer as a default. Shouldn't this mean that the pretrained uncased actually does better than the cased version?

Hi @kh522, were you carefull no to lower case the input in the case of the uncased models? By default the tokenizer will lower the input [see here in the readme](https://github.com/huggingface/pytorch-pretrained-BERT#tokenizer-berttokenizer)

Ah, I see. That would be a problem. I assume that the difference in accent markers will lead to a lower result. That being said, what is / is there a difference between lowercasing before and after the wordpiece tokenization?

I've tried running it with the --do_lower_case flag set to False, and the results are still not good yet. Is there another possible idea?

Try 10 different seeds maybe? A bigger batch-size can help too. More generally, you should try to explore the space of hyper-parameters for fine-tuning, there is often a high variance in the fine-tuning of bert so you will need to compute mean/variances of several results to get meaningful numbers.

In the run_squad.py, the seed is set to 42, therefore the results reported in the repo should be reproducible, as there would not be any other randomness.


> Hi @kh522, were you carefull no to lower case the input in the case of the uncased models? By default the tokenizer will lower the input [see here in the readme](https://github.com/huggingface/pytorch-pretrained-BERT#tokenizer-berttokenizer)

@thomwolf 
Just want to be sure (as the link does not land me in anything specific). If we are pre-training/fine tuning an uncased model still we don't need to lower case input text as tokenizer takes care of it; is this correct understanding?

In case, someone lowercases the text, what problem it can cause?

@abmitra84 
Yes, the tokenizer of the uncased model takes care of it. **Lowercasing the text before by yourself doesn't affect it at all, it's just not necessary since the tokenizer takes care of it**. Code below should clarify it.

```
# Install last Hugging Face libraries (datasets & transformers)
!pip install datasets git+https://github.com/huggingface/transformers/

from transformers import AutoModelForMaskedLM, AutoTokenizer
model = "bert-base-uncased"
tokenizer = AutoTokenizer.from_pretrained(model, use_fast=True)
model = AutoModelForMaskedLM.from_pretrained(model)

sent = "SARS-CoV-2 is a type of the Coronavirus"
sent_lower = sent.lower()

print("[1] ", sent)
print("[2] ",sent_lower)
print("[3] ",tokenizer.tokenize(sent))
print("[4] ",tokenizer.tokenize(sent_lower))
```

Output:
> [1]  SARS-CoV-2 is a type of the Coronavirus
> [2]  sars-cov-2 is a type of the coronavirus
> [3]  ['sar', '##s', '-', 'co', '##v', '-', '2', 'is', 'a', 'type', 'of', 'the', 'corona', '##virus']
> [4]  ['sar', '##s', '-', 'co', '##v', '-', '2', 'is', 'a', 'type', 'of', 'the', 'corona', '##virus']