Accessing roberta embeddings
Author: aclifton314
Pull request: no

## Finetune Setup
* Model: roberta-base
* Language: english
* OS: Ubuntu 18.04.3
* Python version: 3.7.3
* PyTorch version: 1.3.1+cpu
* PyTorch Transformers version (or branch): 2.2.0
* Using GPU ? No
* Distributed of parallel setup ? No
* Script inputs:
```
python run_lm_finetuning.py \
--output_dir=$OUTPUT_DIR \
--model_type=roberta \
--model_name_or_path=roberta_base \
--do_train \
--train_data_file=$TRAIN_FILE \
--do_eval \
--eval_data_file=$TEST_FILE \
--mlm \
--no_cuda
```
## ❓ Questions & Help
I would like to compare the embeddings of a sentence produced by `roberta-base` and my finetuned model (which is based on roberta-base using my domain specific data), but I am not sure how to access them. Any pointers on how to do this? Thanks in advance.


Comments:
Hi, there are several ways to check out the embeddings.

1 - The easy way is to get the `embeddings` and use it as a `torch.nn.Module` (which it inherits from):

For example, this is the output of the embedding layer of the sentence "Alright, let's do this", of dimension (batch_size, sequence_length, hidden_size):

```py
from transformers import RobertaTokenizer, RobertaModel
import torch

tok = RobertaTokenizer.from_pretrained("roberta-base")
model = RobertaModel.from_pretrained("roberta-base")

sentence = torch.tensor([tok.encode("Alright, let's do this")])
embedding_output = model.embeddings(sentence)
```

2 - A different way you can access them is by accessing the hidden states. You have to create a configuration object in order to specify that you would like the model to output its hidden states. You can then initialize the model from that configuration.

Using the example described above:
```py
from transformers import RobertaTokenizer, RobertaModel, RobertaConfig
import torch

config = RobertaConfig.from_pretrained("roberta-base")
config.output_hidden_states = True

tok = RobertaTokenizer.from_pretrained("roberta-base")
model = RobertaModel.from_pretrained("roberta-base", config=config)

sentence = torch.tensor([tok.encode("Alright, let's do this")])

output = model(sentence)  # returns a tuple(sequence_output, pooled_output, hidden_states)
hidden_states = output[-1]

embedding_output = hidden_states[0]
```

Those are the embeddings using only the embeddings layer, which do not change much when fine-tuned. If you want to access the sentence representations of the two models, you can simply use the `sequence_outputs`:

```py
output = model(input)
sequence_output = output[0]

finetuned_output = finetuned_model(input)
finetuned_sequence_output = finetuned_output[0]
```

You can then compare those however you see fit!

which model from roberta i can use for RU lang?
or better using `xlm-mlm-17-1280` or `bert-base-multilingual-cased`?

@LysandreJik thank you very much for your response! 

To check my understanding, I can access the output of the embedding layer of roberta using the procedures you described (1&2). I can also access the embeddings learned at the last layer of roberta (the final layer) doing the following:
```python
from transformers import RobertaTokenizer, RobertaModel, RobertaConfig
import torch

config = RobertaConfig.from_pretrained("roberta-base")
config.output_hidden_states = True

tok = RobertaTokenizer.from_pretrained("roberta-base")
model = RobertaModel.from_pretrained("roberta-base", config=config)

sentence = torch.tensor([tok.encode("Alright, let's do this")])

output = model(sentence)
final_embeddings = output[0]
```
Is my understanding correct or have I missed something?

@aclifton314 i think u should take `output[-1]` not a `output[0]`
btw, the 1st & 2nd example return similar vectors
also, the 2nd example has been working longer  than 1st

@vtrokhymenko Do you know what the difference is between `output[-1]` and `output[0]`?

@aclifton314 the answer u can find here:
>You have to create a configuration object in order to specify that you would like the model to output its hidden states

@aclifton314 Referring to the output of the last layer as embeddings may be a bit ambiguous here, but yes, your `final_embeddings` variable holds the representation of your sequence at the uppermost layer (having gone through every model layer).

`output[-1]` returns the hidden states while `output[0]` returns the sequence output.

@LysandreJik @vtrokhymenko , thank you both for your replies! Closing this issue.