Bert encodings
Author: shubhamagarwal92
Pull request: no
Labels: name: wontfix

Hi,

Really interesting work! 

I want to use BERT embeddings for a downstream task. I have been following the [steps](https://github.com/huggingface/pytorch-transformers#quick-tour) here as:

```
import torch
from pytorch_transformers import BertModel, BertTokenizer

pretrained_weights = 'bert-base-uncased'
tokenizer = BertTokenizer.from_pretrained(pretrained_weights)
model = BertModel.from_pretrained(pretrained_weights)

raw_text = ["[CLS] This is first element [SEP] continuing statement", 
            "[CLS] second element of the list."]

encoding = tokenizer.encode(raw_text)
input_ids = torch.tensor(encoding)
last_hidden_states = model(input_ids)[0]  # Models outputs are now tuples
print(last_hidden_states.size())
```
getting the error as:
```
  File "/home/shubham/anaconda3/envs/test/lib/python3.6/site-packages/pytorch_transformers/tokenization_utils.py", line 356, in split_on_tokens
    split_text = text.split(tok)
AttributeError: 'list' object has no attribute 'split'
```

@thomwolf Is there an easy way to pass the list instead of strings or should I use lambda functions (which might be slow)? Can we pass the maximum sequence length as well? 

Also I need some advice related to the code structure. If I have pre-exisiting code with data loader, should I convert these embedding there (can't do finetuning then) or pass the raw strings and convert them before passing them to model (model run gets too slow)? 

Is there any way where we can pass the vocabulary (and tensors) directly instead of passing raw strings? 

Comments:
I have borrowed most of the ideas from [utils](https://github.com/huggingface/pytorch-transformers/blob/master/examples/utils_glue.py#L391) in one of the examples to create a [script to extract embeddings](https://gist.github.com/shubhamagarwal92/37ccb747f7130a35a8e76aa66d60e014). 

However, I am still curious if there is any way where we can pass the vocabulary (and tensors) directly instead of passing raw strings?



Hi, I think your example is nice.

I'm not sure to understand what you are referring to when you want to "pass the vocabulary (and tensors) directly instead of passing raw strings". 

Currently, I am trying to get the Bert embeddings in my encoder before I use `nn.Embedding` instead of pre-computing it. 

Thus, I have to convert the tensors to raw strings using `vocab` before passing it through the bert model and hence the gist. 

Sorry,I want to know why we pass the sentence(Hello, my dog is cute) directly instead of adding some tokens([CLS] Hello, my dog is cute [SEP]) in the  @thomwolf .

```
        >>> config = BertConfig.from_pretrained('bert-base-uncased')
        >>> tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
        >>> model = BertModel(config)
        >>> input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute")).unsqueeze(0)  # Batch size 1
        >>> outputs = model(input_ids)
        >>> last_hidden_states = outputs[0]  # The last hidden-state is the first element of the output tuple
```

If you use the function such as [this](https://github.com/huggingface/pytorch-transformers/blob/master/examples/utils_glue.py#L391), it appends the special tokens automatically. I guess the example you mentioned needs to append these tokens. 

@thomwolf could verify that for you! 

Yeah we'll add the option to automatically add control tokens. It can be useful.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


> If you use the function such as [this](https://github.com/huggingface/pytorch-transformers/blob/master/examples/utils_glue.py#L391), it appends the special tokens automatically. I guess the example you mentioned needs to append these tokens.
> 
> @thomwolf could verify that for you!

Yes, I understand one can set "add_special_tokens=True" for the same when encoding the document.