A sequence with no special tokens has been passed to the RoBERTa model. This model  requires special tokens in order to work. Please specify add_special_tokens=True in  your encoding.
Author: gr8Adakron
Pull request: no
Labels: name: wontfix

This is my code for Roberta:

```

# coding: utf-8

# In[4]:


import pandas as pd
import numpy as np
import json, re
from tqdm import tqdm_notebook
from uuid import uuid4

## Torch Modules
import torch
import torch.optim as optim
import torch.nn as nn
import torch.nn.functional as F
from torch.autograd import Variable
from torch.utils.data import Dataset, DataLoader

## PyTorch Transformer
from pytorch_transformers import RobertaModel, RobertaTokenizer
from pytorch_transformers import RobertaForSequenceClassification, RobertaConfig


# In[21]:


import pandas as pd
train_data = pd.read_csv("walmart_input/test/train.csv")
test_data  = pd.read_csv("walmart_input/test/test.csv")
dataset    = pd.concat([train_data, test_data])
test_data.head()


# In[46]:


total_length = len(dataset)

# In[47]:


label_to_ix = {}
for label in dataset.PT:
    if label not in label_to_ix:
        label_to_ix[label]=len(label_to_ix)

total_pt_count = len(list(set(list(train_data["PT"]))))



# In[48]:


config            = RobertaConfig.from_pretrained('roberta-base')
config.num_labels = len(list(set(list(train_data["PT"]))))

print(f"Total length of dataset: {total_length} \n Total PT count: {total_pt_count} \n {config}")
# In[36]:


tokenizer = RobertaTokenizer.from_pretrained('roberta-base')
model     = RobertaForSequenceClassification(config)

def prepare_features(seq_1, max_seq_length = 300, 
             zero_pad = False, include_CLS_token = True, include_SEP_token = True):
    ## Tokenzine Input
    tokens_a = tokenizer.tokenize(seq_1)

    ## Truncate
    if len(tokens_a) > max_seq_length - 2:
        tokens_a = tokens_a[0:(max_seq_length - 2)]
    ## Initialize Tokens
    tokens = []
    if include_CLS_token:
        tokens.append(tokenizer.cls_token)
    ## Add Tokens and separators
    for token in tokens_a:
        tokens.append(token)

    if include_SEP_token:
        tokens.append(tokenizer.sep_token)

    input_ids = tokenizer.convert_tokens_to_ids(tokens)
    ## Input Mask 
    input_mask = [1] * len(input_ids)
    ## Zero-pad sequence lenght
    if zero_pad:
        while len(input_ids) < max_seq_length:
            input_ids.append(0)
            input_mask.append(0)
    return torch.tensor(input_ids).unsqueeze(0), input_mask



# In[38]:


class Intents(Dataset):
    def __init__(self, dataframe):
        self.len = len(dataframe)
        self.data = dataframe
        
    def __getitem__(self, index):
        title = self.data.title[index]
        label = self.data.PT[index]
        X, _  = prepare_features(title)
        y = label_to_ix[self.data.PT[index]]
        return X, y
    
    def __len__(self):
        return self.len
    
    
print("FULL Dataset: {}".format(dataset.shape))
print("TRAIN Dataset: {}".format(train_data.shape))
print("TEST Dataset: {}".format(test_data.shape))
training_set = Intents(train_data)
testing_set = Intents(test_data)


training_set.__getitem__(0)[0].shape
model(training_set.__getitem__(0)[0])


# In[65]:


## Training Params
if torch.cuda.device_count() > 1:
    print("Let's use", torch.cuda.device_count(), "GPUs!")
    # dim = 0 [30, xxx] -> [10, ...], [10, ...], [10, ...] on 3 GPUs
    #model = nn.DataParallel(model)

## Training Params
device = torch.device("cuda:0,1" if torch.cuda.is_available() else "cpu")
model  = model.cuda()
model = nn.DataParallel(model,device_ids=[0,1],dim=1)
model.to(device)
# Parameters
params = {'batch_size': 1,
          'shuffle': True,
          'num_workers': 2}

training_loader = DataLoader(training_set, **params)
testing_loader  = DataLoader(testing_set, **params)


# In[66]:


loss_function = nn.CrossEntropyLoss()
learning_rate = 1e-02
optimizer = optim.Adam(params =  model.parameters(), lr=learning_rate)

## Test Forward Pass
inp = training_set.__getitem__(0)[0].cuda()

#print(inp)
output = model(inp)[0]
torch.max(output.data, 1)


# In[ ]:


import time
start_time    = time.time()
max_epochs    = 2
model         = model.train()
for epoch in tqdm_notebook(range(max_epochs)):
    print("EPOCH -- {}".format(epoch))
    for i, (sent, label) in enumerate(training_loader):
        optimizer.zero_grad()
        sent = sent.squeeze(0)
        if torch.cuda.is_available():
          sent = sent.cuda()
          label = label.cuda()
        print("CUDA detail:")
        print(sent)
        print(label)
        output = model.forward(sent)[0]
        _, predicted = torch.max(output, 1)
        print(f" - {i}.) {predicted}")
        loss = loss_function(output, label)
        loss.backward()
        optimizer.step()

        if i%100 == 0:
            correct = 0
            total = 0
            for sent, label in testing_loader:
                sent = sent.squeeze(0)
                if torch.cuda.is_available():
                  sent = sent.cuda()
                  label = label.cuda()
                output = model.forward(sent)[0]
                _, predicted = torch.max(output.data, 1)
                total += label.size(0)
                correct += (predicted.cpu() == label.cpu()).sum()
            accuracy = 100.00 * correct.numpy() / total
            print('Iteration: {}. Loss: {}. Accuracy: {}%'.format(i, loss.item(), accuracy))

timetaken    = format(float((time.time() - start_time)),'.3f')
print(timetaken)

torch.save(model.state_dict(), 'roberta_state_dict_on_new_data_MAY2019.pth')

```
Here I am trying to run the code on Multiple GPUs(2-P100). But I keep getting this warning:
```

A sequence with no special tokens has been passed to the RoBERTa model. This model 
requires special tokens in order to work. Please specify add_special_tokens=True in
 your encoding.


```

I am not sure, what causing this issue. But when I run it using single (i.e: after removing DataParallel wrapper), it doesn't give this warning.

Any help would be appreciated.

-thanks.


Comments:
@yaroslavvb @cynthia @myleott 

Hi, this error springs when you're passing an input to the model which doesn't have the special tokens it needs (cls token and sep token).

The `encode` method accepts the argument `add_special_tokens`, which will take care of adding the special tokens to your sequence.

I have exactly the same problem, when running on a single GPU it works well, but on the 2-GPUS it got this warning and an index error then caused cuDNN error: CUDNN_STATUS_NOT_INITIALIZED

I found the problem, when running on multi-gpus, all inputs in forward well divided into n-gpus, for an input tensor with shape (batch, x, y), it will divided into (n-gpus, batch/n-gpus, x, y), if tensor doesn't have the batch dim, then caused this error.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


I have the same issue on just loading the model

```
tokenizer = RobertaTokenizer.from_pretrained('roberta-base',add_special_tokens=True)
model = TFRobertaForSequenceClassification.from_pretrained('roberta-base')

```
Returns
```
A sequence with no special tokens has been passed to the RoBERTa model. This model requires special tokens in order to work. Please specify add_special_tokens=True in your encoding.
A sequence with no special tokens has been passed to the RoBERTa model. This model requires special tokens in order to work. Please specify add_special_tokens=True in your encoding.

```

Can you try specifying it in `encode()` not in `from_pretrained()`?

Yes but it surprises me that it throws the warning and i did not pass any data to the model. So far in the code there is nothing to encode.

TensorFlow models need to be "built" by first passing inputs through their layers. This warning occurs then.

This warning was removed in the recent versions of transformers.