Add CANINE
Author: NielsRogge
Pull request: yes

# What does this PR do?

It adds Google's new [CANINE](https://arxiv.org/abs/2103.06874) model. It's a tokenizer-free model, meaning you can throw away your `vocab.txt` file. The model trains at a character level, namely by turning each character into its unicode code point. In Python, this can be done using the built-in `ord()` function. This means that `input_ids` can be created simply as `[ord(char) for char in text]`. This is different from ByT5. 

However, there's still a good use for a `CanineTokenizer`, namely for padding/truncating unicode code points up to the max length of 2048. It's also handy as it let's you easily convert text (string) to ids (unicode code points) and vice versa. 

Due to the bigger sequence length (2048), the model downsamples the characters to what is called "molecules", of length 512. Then, a regular BERT encoder is applied. The `pooled_output` (which can be used for sequence classification) is then simply equal to the last hidden state of the first [CLS] token followed by a linear layer. In order to get a `sequence_output` (useful for token classification tasks) which is again of length 2048, an upsampling technique is used (details can be found in the paper). 

To do:

- [x] remove `is_decoder` logic. Is it possible to add support for `is_decoder` in a future PR? Will this be backwards compatible? Answer: yes
- [x] fix tests (currently 35 pass, 1 fail). Once the official checkpoints are on the hub under the "Google" namespace, all tests pass.
- [ ] Update namespace on the hub from nielsr to google (update the tests for this too), and add model cards.
 
A question here: the CANINE model uses 3 Transformer encoders (2 shallow ones, consisting of only a single layer, of which the first one uses local attention, and a deep one similar to BERT). Should it be possible to return the `hidden_states` and `attentions` of all of these 3 encoders in the output? Or only of the deep one? Otherwise I need to define a custom `CanineModelOutput`.

Fixes #11016 

cc @patil-suraj @patrickvonplaten 

Also tagging one of the original authors: @dhgarrette

Comments:
Hi @NielsRogge , thanks for that PR! I'm currently trying to run the token classification example and here's some initial feedback:

- AutoTokenizer is currently not working/finding the `CanineTokenizer`, so I had to manually specify it
- It seems that the token classification example only works with Fast Tokenizers: I disabled that assertion check, but then the following message is thrown:

```  File "run_ner.py", line 514, in <module>
    main()
  File "run_ner.py", line 361, in main
    load_from_cache_file=not data_args.overwrite_cache,
  File "/opt/conda/lib/python3.6/site-packages/datasets/arrow_dataset.py", line 1407, in map
    update_data = does_function_return_dict(test_inputs, test_indices)
  File "/opt/conda/lib/python3.6/site-packages/datasets/arrow_dataset.py", line 1378, in does_function_return_dict
    function(*fn_args, indices, **fn_kwargs) if with_indices else function(*fn_args, **fn_kwargs)
  File "run_ner.py", line 330, in tokenize_and_align_labels
    word_ids = tokenized_inputs.word_ids(batch_index=i)
  File "/mnt/transformers-canince/src/transformers/tokenization_utils_base.py", line 347, in word_ids
    raise ValueError("word_ids() is not available when using Python-based tokenizers")
ValueError: word_ids() is not available when using Python-based tokenizers
```

is there any chance to get this example running in the final version of PR :thinking: would be highly interesting to see the results :hugs: 

@patrickvonplaten @sgugger @LysandreJik my PR is ready for review. 

Just a question: I might define a custom `CanineModelOutput`, as CANINE consists of 3 Transformer encoders (2 shallow ones, which only consists of a single layer, and one "deep" BERT-like). If a user specifies `output_hidden_states=True` for example, then it could return the hidden states of all of these 3 encoders (however, the hidden states won't have the same shape then). Currently, I'm just using a `BaseModelOutputWithPooling`, and it only returns the hidden states of the deep encoder (which all have the same shape). Would appreciate your feedback here. 

Also @LysandreJik: would be great if you could review the tokenizer. Perhaps it would be better if a space is added after the CLS token and before the SEP token when decoding, e.g. "hello world" is currently decoded as "[CLS]hello world[SEP]". Do I need to update the `lstrip` and `rstrip` parameters of the `AddedToken` instances for that?




@NielsRogge just one question (I tried to use `CanineModel` with the latest commit):

```python
In [36]: output[-1].shape
Out[36]: torch.Size([1, 3, 768])

In [37]: encoding = tokenizer(["hello world and hugging face"], padding="longest", return_tensors="pt")

In [38]: hidden_states = model(**encoding).hidden_states

In [39]: hidden_states[-1].shape
Out[39]: torch.Size([1, 7, 768])

In [40]: encoding = tokenizer(["huggingface"], padding="longest", return_tensors="pt")

In [41]: hidden_states = model(**encoding).hidden_states

In [42]: hidden_states[-1].shape
Out[42]: torch.Size([1, 3, 768])
```

I don't really understand the shape of the hidden states from the model. I would expect one tensor per character, but why is it sometimes 3 or 7? 

Hi @stefan-it,

The reason is that CANINE downsamples the character sequence length before applying the deep Transformer encoder. The downsampling rate is by default set to 4, and the max sequence length (in terms of characters) is set to 2048. So `2048 // 4 = 512`, which is the regular length of models like BERT and RoBERTa. 

In your case, you're not padding anything, so if you just provide `"HuggingFace"`, then the character sequence length is 13 (special tokens included), and `13 // 4 = 3`, hence the hidden states of the deep encoder have length 3. If you just want the final hidden states for each character (which are upsampled by another shallow Transformer encoder), you can use `outputs.last_hidden_state`. 

But, and that's what my question is about above, I could, instead of only returning the `hidden states` of the deep encoder, also return the hidden states of the initial and final Transformer encoders. In that case, you will have `hidden_states` of the initial encoder at the character level, then `hidden_states` of the deep encoder at the downsampled level (the authors call this "molecule level"), and the `hidden_states` of the final encoder at the character level. 

Update: I'm working on a separate branch called `updating_outputs_canine` which replaces the `BaseModelOutputWithPooling` by a custom `CanineModelOutputWithPooling`, and it returns the attentions and hidden states of all 3 Transformer encoders. All tests passing :) I'll merge it with this main branch once I've got approval

Thanks a lot for your work @NielsRogge, a fantastic addition to the library once again!

Thanks! I haven't uploaded the model checkpoints yet, will do soon

Hi @NielsRogge , thanks for adding this! I'm current playing around with the model and found this corner case:

```python
from transformers import AutoConfig, CanineModel, CanineTokenizer
model_name = "google/canine-s"

config = AutoConfig.from_pretrained(model_name, output_hidden_states=True)
model = CanineModel.from_pretrained(pretrained_model_name_or_path=model_name, config=config)
tokenizer = CanineTokenizer.from_pretrained(model_name)

encoding = tokenizer(["."], padding="longest", return_tensors="pt")
hidden_states = model(**encoding).last_hidden_state
```

This happens, when input has a length of 1:

```bash
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)                                
<ipython-input-1-3719c2ae7d25> in <module>                    
      7                                                                    
      8 encoding = tokenizer(["."], padding="longest", return_tensors="pt")
----> 9 hidden_states = model(**encoding).last_hidden_state              
       
/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
   1013         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks          
   1014                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1015             return forward_call(*input, **kwargs)
   1016         # Do not call functions when jit is used
   1017         full_backward_hooks, non_full_backward_hooks = [], []

/mnt/europeana-bert/transformers/src/transformers/models/canine/modeling_canine.py in forward(self, input_ids, attention_mask, token_type_ids, position_ids, head_mask, inputs_embeds, outpu
t_attentions, output_hidden_states, return_dict)
   1185         # this, it seems that molecules and characters require a very different
   1186         # feature space; intuitively, this makes sense.
-> 1187         init_molecule_encoding = self.chars_to_molecules(input_char_encoding)
   1188
   1189         # Deep BERT encoder

/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
   1013         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1014                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1015             return forward_call(*input, **kwargs)
   1016         # Do not call functions when jit is used
   1017         full_backward_hooks, non_full_backward_hooks = [], []

/mnt/europeana-bert/transformers/src/transformers/models/canine/modeling_canine.py in forward(self, char_encoding)
    324         # We transpose it to be [batch, hidden_size, char_seq]
    325         char_encoding = torch.transpose(char_encoding, 1, 2)
--> 326         downsampled = self.conv(char_encoding)
    327         downsampled = torch.transpose(downsampled, 1, 2)
    328         downsampled = self.activation(downsampled)

/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
   1013         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1014                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1015             return forward_call(*input, **kwargs)
   1016         # Do not call functions when jit is used
   1017         full_backward_hooks, non_full_backward_hooks = [], []

/opt/conda/lib/python3.8/site-packages/torch/nn/modules/conv.py in forward(self, input)
    261
    262     def forward(self, input: Tensor) -> Tensor:
--> 263         return self._conv_forward(input, self.weight, self.bias)
    264
    265

/opt/conda/lib/python3.8/site-packages/torch/nn/modules/conv.py in _conv_forward(self, input, weight, bias)
    257                             weight, bias, self.stride,
    258                             _single(0), self.dilation, self.groups)
--> 259         return F.conv1d(input, weight, bias, self.stride,
    260                         self.padding, self.dilation, self.groups)
    261

RuntimeError: Calculated padded input size per channel: (3). Kernel size: (4). Kernel size can't be greater than actual input size
```

(I have one NER dataset, and one sentence consists of one token, and that token is the `.` 😅)

Thanks for your patience :hugs: 

Yeah so the input has length 3 (3 unicode code points, namely for "[CLS]", "." and  "[SEP]". However, CANINE uses a convolutional operation to downsample the sequence length, and the kernel size is 4. So of course you can't apply a kernel of size 4 to an input of size 3. So it's advised to pad the input up a size that's at least 4.