CamemBERT
Author: louismartin
Pull request: yes

Code for the CamemBERT model, adapted from https://github.com/pytorch/fairseq/tree/master/examples/camembert

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1822?src=pr&el=h1) Report
> Merging [#1822](https://codecov.io/gh/huggingface/transformers/pull/1822?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/155c782a2ccd103cf63ad48a2becd7c76a7d2115?src=pr&el=desc) will **decrease** coverage by `0.14%`.
> The diff coverage is `62.5%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/1822/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/1822?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #1822      +/-   ##
==========================================
- Coverage   84.16%   84.02%   -0.15%     
==========================================
  Files          94       97       +3     
  Lines       14185    14281      +96     
==========================================
+ Hits        11939    11999      +60     
- Misses       2246     2282      +36
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/1822?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [transformers/modeling\_camembert.py](https://codecov.io/gh/huggingface/transformers/pull/1822/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL21vZGVsaW5nX2NhbWVtYmVydC5weQ==) | `100% <100%> (ø)` | |
| [transformers/configuration\_camembert.py](https://codecov.io/gh/huggingface/transformers/pull/1822/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL2NvbmZpZ3VyYXRpb25fY2FtZW1iZXJ0LnB5) | `100% <100%> (ø)` | |
| [transformers/tokenization\_auto.py](https://codecov.io/gh/huggingface/transformers/pull/1822/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL3Rva2VuaXphdGlvbl9hdXRvLnB5) | `45.94% <33.33%> (-1.12%)` | :arrow_down: |
| [transformers/tokenization\_camembert.py](https://codecov.io/gh/huggingface/transformers/pull/1822/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL3Rva2VuaXphdGlvbl9jYW1lbWJlcnQucHk=) | `36.53% <36.53%> (ø)` | |
| [transformers/configuration\_auto.py](https://codecov.io/gh/huggingface/transformers/pull/1822/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL2NvbmZpZ3VyYXRpb25fYXV0by5weQ==) | `59.45% <66.66%> (+0.63%)` | :arrow_up: |
| [transformers/modeling\_tf\_pytorch\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/1822/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL21vZGVsaW5nX3RmX3B5dG9yY2hfdXRpbHMucHk=) | `92.95% <0%> (ø)` | :arrow_up: |
| [transformers/tests/modeling\_tf\_common\_test.py](https://codecov.io/gh/huggingface/transformers/pull/1822/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL3Rlc3RzL21vZGVsaW5nX3RmX2NvbW1vbl90ZXN0LnB5) | `97.08% <0%> (ø)` | :arrow_up: |
| [transformers/modeling\_roberta.py](https://codecov.io/gh/huggingface/transformers/pull/1822/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL21vZGVsaW5nX3JvYmVydGEucHk=) | `71.76% <0%> (ø)` | :arrow_up: |
| [transformers/modeling\_xlnet.py](https://codecov.io/gh/huggingface/transformers/pull/1822/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL21vZGVsaW5nX3hsbmV0LnB5) | `73.61% <0%> (ø)` | :arrow_up: |
| [transformers/modeling\_ctrl.py](https://codecov.io/gh/huggingface/transformers/pull/1822/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL21vZGVsaW5nX2N0cmwucHk=) | `96.46% <0%> (ø)` | :arrow_up: |
| ... and [4 more](https://codecov.io/gh/huggingface/transformers/pull/1822/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/1822?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1822?src=pr&el=footer). Last update [155c782...65549e3](https://codecov.io/gh/huggingface/transformers/pull/1822?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Hi @louismartin many thanks for adding this :heart: 

I tested the implementation a bit, and I got the following error message when I try to save the tokenizer object:

For RoBERTa this is working correctly:

```python
In [2]: roberta_tokenizer = RobertaTokenizer.from_pretrained("roberta-base")

In [3]: roberta_tokenizer.save_pretrained("/tmp")
Out[3]: 
('/tmp/vocab.json',
 '/tmp/merges.txt',
 '/tmp/special_tokens_map.json',
 '/tmp/added_tokens.json')
```

But for CamemBERT:

```python
In [6]: camembert_tokenizer = CamembertTokenizer.from_pretrained("camembert-base")

In [7]: camembert_tokenizer.save_pretrained("/tmp")
---------------------------------------------------------------------------
NotImplementedError                       Traceback (most recent call last)
<ipython-input-7-417ff9326aaf> in <module>
----> 1 camembert_tokenizer.save_pretrained("/tmp")

/mnt/transformers-camembert/transformers/tokenization_utils.py in save_pretrained(self, save_directory)
    463             f.write(out_str)
    464
--> 465         vocab_files = self.save_vocabulary(save_directory)
    466
    467         return vocab_files + (special_tokens_map_file, added_tokens_file)

/mnt/transformers-camembert/transformers/tokenization_utils.py in save_vocabulary(self, save_directory)
    474             Please use :func:`~transformers.PreTrainedTokenizer.save_pretrained` `()` to save the full Tokenizer state if you want to reload it using the :func:`~transformers.PreTrainedTokenizer.from_pretrained` class method.
    475         """
--> 476         raise NotImplementedError
    477
    478
```

`.save_vocabulary()` is used e.g. in the NER example script :)

> Hi @louismartin many thanks for adding this ❤️
> 
> I tested the implementation a bit, and I got the following error message when I try to save the tokenizer object:
> 
> For RoBERTa this is working correctly:
> 
> ```python
> In [2]: roberta_tokenizer = RobertaTokenizer.from_pretrained("roberta-base")
> 
> In [3]: roberta_tokenizer.save_pretrained("/tmp")
> Out[3]: 
> ('/tmp/vocab.json',
>  '/tmp/merges.txt',
>  '/tmp/special_tokens_map.json',
>  '/tmp/added_tokens.json')
> ```
> 
> But for CamemBERT:
> 
> ```python
> In [6]: camembert_tokenizer = CamembertTokenizer.from_pretrained("camembert-base")
> 
> In [7]: camembert_tokenizer.save_pretrained("/tmp")
> ---------------------------------------------------------------------------
> NotImplementedError                       Traceback (most recent call last)
> <ipython-input-7-417ff9326aaf> in <module>
> ----> 1 camembert_tokenizer.save_pretrained("/tmp")
> 
> /mnt/transformers-camembert/transformers/tokenization_utils.py in save_pretrained(self, save_directory)
>     463             f.write(out_str)
>     464
> --> 465         vocab_files = self.save_vocabulary(save_directory)
>     466
>     467         return vocab_files + (special_tokens_map_file, added_tokens_file)
> 
> /mnt/transformers-camembert/transformers/tokenization_utils.py in save_vocabulary(self, save_directory)
>     474             Please use :func:`~transformers.PreTrainedTokenizer.save_pretrained` `()` to save the full Tokenizer state if you want to reload it using the :func:`~transformers.PreTrainedTokenizer.from_pretrained` class method.
>     475         """
> --> 476         raise NotImplementedError
>     477
>     478
> ```
> 
> `.save_vocabulary()` is used e.g. in the NER example script :)

Hi Stefan, 

Thanks for trying our model!
The reason is that I haven't implemented saving the tokenizer.
Two solutions: either don't use `tokenizer.save_pretrained`, or implement it by adapting the necessary methods from [tokenization_xlnet.py](https://github.com/huggingface/transformers/blob/master/transformers/tokenization_xlnet.py) to implement `save_pretrained()`. It is probably only necessary to copy and paste `__setstate__` and `__getstate__` methods :)
I don't have much time to do it in the next few weeks, but don't hesitate if you have more questions!

Louis


Hi @louismartin!!

### **#1844**

I rebased on master, did the few discussed tweaks, and merged this: https://github.com/huggingface/transformers/pull/1844

Great work! 

@stefan-it @julien-c @louismartin 

Could you show how to get the embedding vector of a sentence please?

```python
from transformers import CamembertTokenizer
import torch

camembert_tokenizer = CamembertTokenizer.from_pretrained("camembert-base")

camembert_tokenizer.encode("Salut, ça va ?") # How to get embedding of this sentence not just the ids of tokens ? 
```

@stefan-it @julien-c @louismartin @hzitoun
Question: the following weights I've extracted with the following code are the weights of the pre-trained CamemBERT model or the sentence embeddings?

```
from transformers import CamembertTokenizer
from transformers import CamembertModel

text="J'aime le camembert !"
tokenizer = CamembertTokenizer.from_pretrained('camembert-base')
model = CamembertModel.from_pretrained('camembert-base', output_hidden_states=True) # add output_hidden_states in order to retrieve ALL hidden states of the CamemBERT model (so the embeddings layer too!)
input_ids = torch.tensor(tokenizer.encode(text)).unsqueeze(0)
output = model(input_ids)

embeddings = output[2][0]
print('embeddings: \n{}'.format(embeddings))
>>> tensor([[[ 0.0399, -0.0788,  0.0552,  ..., -0.0855, -0.0394,  0.0222],
         [-0.1678,  0.1954, -0.2219,  ...,  0.1590, -0.2967, -0.1723],
         [ 0.0699,  0.2307, -0.0723,  ...,  0.2294,  0.0463, -0.0167],
         ...,
         [ 0.0766,  0.2548,  0.0690,  ...,  0.0172, -0.3713, -0.0262],
         [ 0.1681,  0.0253, -0.0386,  ...,  0.1626, -0.1203, -0.2415],
         [ 0.4876, -0.0714, -0.0020,  ..., -0.0628, -0.2701,  0.2210]]],
       grad_fn=<NativeLayerNormBackward>)
```

It has been tested by comparing with the official Camembert model found [here](https://camembert-model.fr/#contact).
```
import torch
camembert = torch.hub.load('pytorch/fairseq', 'camembert.v0')
camembert.eval()  # disable dropout (or leave in train mode to finetune)

line = "J'aime le camembert!"
tokens = camembert.encode(line)

all_layers = camembert.extract_features(tokens, return_all_hiddens=True)
embeddings = all_layers[0]
print('embeddings: \n{}'.format(embeddings)
>>> tensor([[[ 0.0399, -0.0788,  0.0552,  ..., -0.0855, -0.0394,  0.0222],
         [-0.1678,  0.1954, -0.2219,  ...,  0.1590, -0.2967, -0.1723],
         [ 0.0699,  0.2307, -0.0723,  ...,  0.2294,  0.0463, -0.0167],
         ...,
         [ 0.0766,  0.2548,  0.0690,  ...,  0.0172, -0.3713, -0.0262],
         [ 0.1461,  0.0414, -0.0877,  ..., -0.0577, -0.2219, -0.3685],
         [ 0.4876, -0.0714, -0.0020,  ..., -0.0628, -0.2701,  0.2210]]],
       grad_fn=<TransposeBackward0>)
```

Most values correspond, but there are some values different (e.g. 0.1681 vs 0.1461 or 0.2415 vs -0.3685)..

> Great work!
> 
> @stefan-it @julien-c @louismartin
> 
> Could you show how to get the embedding vector of a sentence please?
> 
> ```python
> from transformers import CamembertTokenizer
> import torch
> 
> camembert_tokenizer = CamembertTokenizer.from_pretrained("camembert-base")
> 
> camembert_tokenizer.encode("Salut, ça va ?") # How to get embedding of this sentence not just the ids of tokens ? 
> ```

> @stefan-it @julien-c @louismartin @hzitoun
> Question: the following weights I've extracted with the following code are the weights of the pre-trained CamemBERT model or the sentence embeddings?
> 
> ```
> from transformers import CamembertTokenizer
> from transformers import CamembertModel
> 
> text="J'aime le camembert !"
> tokenizer = CamembertTokenizer.from_pretrained('camembert-base')
> model = CamembertModel.from_pretrained('camembert-base', output_hidden_states=True) # add output_hidden_states in order to retrieve ALL hidden states of the CamemBERT model (so the embeddings layer too!)
> input_ids = torch.tensor(tokenizer.encode(text)).unsqueeze(0)
> output = model(input_ids)
> 
> embeddings = output[2][0]
> print('embeddings: \n{}'.format(embeddings))
> >>> tensor([[[ 0.0399, -0.0788,  0.0552,  ..., -0.0855, -0.0394,  0.0222],
>          [-0.1678,  0.1954, -0.2219,  ...,  0.1590, -0.2967, -0.1723],
>          [ 0.0699,  0.2307, -0.0723,  ...,  0.2294,  0.0463, -0.0167],
>          ...,
>          [ 0.0766,  0.2548,  0.0690,  ...,  0.0172, -0.3713, -0.0262],
>          [ 0.1681,  0.0253, -0.0386,  ...,  0.1626, -0.1203, -0.2415],
>          [ 0.4876, -0.0714, -0.0020,  ..., -0.0628, -0.2701,  0.2210]]],
>        grad_fn=<NativeLayerNormBackward>)
> ```
> 
> It has been tested by comparing with the official Camembert model found [here](https://camembert-model.fr/#contact).
> 
> ```
> import torch
> camembert = torch.hub.load('pytorch/fairseq', 'camembert.v0')
> camembert.eval()  # disable dropout (or leave in train mode to finetune)
> 
> line = "J'aime le camembert!"
> tokens = camembert.encode(line)
> 
> all_layers = camembert.extract_features(tokens, return_all_hiddens=True)
> embeddings = all_layers[0]
> print('embeddings: \n{}'.format(embeddings)
> >>> tensor([[[ 0.0399, -0.0788,  0.0552,  ..., -0.0855, -0.0394,  0.0222],
>          [-0.1678,  0.1954, -0.2219,  ...,  0.1590, -0.2967, -0.1723],
>          [ 0.0699,  0.2307, -0.0723,  ...,  0.2294,  0.0463, -0.0167],
>          ...,
>          [ 0.0766,  0.2548,  0.0690,  ...,  0.0172, -0.3713, -0.0262],
>          [ 0.1461,  0.0414, -0.0877,  ..., -0.0577, -0.2219, -0.3685],
>          [ 0.4876, -0.0714, -0.0020,  ..., -0.0628, -0.2701,  0.2210]]],
>        grad_fn=<TransposeBackward0>)
> ```
> 
> Most values correspond, but there are some values different (e.g. 0.1681 vs 0.1461 or 0.2415 vs -0.3685)..
> 
> > Great work!
> > @stefan-it @julien-c @louismartin
> > Could you show how to get the embedding vector of a sentence please?
> > ```python
> > from transformers import CamembertTokenizer
> > import torch
> > 
> > camembert_tokenizer = CamembertTokenizer.from_pretrained("camembert-base")
> > 
> > camembert_tokenizer.encode("Salut, ça va ?") # How to get embedding of this sentence not just the ids of tokens ? 
> > ```

@TheEdoardo93 
Thanks for showing how to do the same thing with native `torch` and for pointing out the difference in embedding for some values! 

Does any one have an idea how to get a fixed shape of embedding vector regardless the number of tokens in a sentence? (in order to do cosine distance for example between two sentences even if they have different tokens sizes) 

Calculating the `mean on axis=1` for example ?  I've asked the question on SOF too if you could answer https://stackoverflow.com/questions/59030907/nlp-transformers-how-to-get-a-fixed-embedding-vector-size :)