Roberta semantic similarity
Author: subhamkhemka
Pull request: no

## ❓ Questions & Help

Hi 


I am trying to use Roberta for semantic similarity.
have 2 questions 
Can you validate my code to check if its able to correctly execute sentence-pair classification ?

I want to train  the roberta-large-mnli model on my own corpus, how do I do this?


Code : 
```python
from pytorch_transformers import RobertaModel, RobertaTokenizer
from pytorch_transformers import RobertaForSequenceClassification, RobertaConfig

config = RobertaConfig.from_pretrained('roberta-large')
config.num_labels = len(list(label_to_ix.values()))
tokenizer = RobertaTokenizer.from_pretrained('roberta-large-mnli')
model = RobertaForSequenceClassification(config)
def prepare_features(seq_1,seq_2):
    aa=tokenizer.encode(seq_1)
    bb=tokenizer.encode(seq_2)
    zz=tokenizer.add_special_tokens_sentences_pair(aa,bb)
    input_ids=torch.tensor(zz)
    input_mask = [1] * len(zz)
    return torch.tensor(input_ids).unsqueeze(0), input_mask
class Intents(Dataset):
    def __init__(self, dataframe):
        self.len = len(dataframe)
        self.data = dataframe
        
    def __getitem__(self, index):
        utterance = self.data.q1[index]
        sent2 = self.data.q2[index]
        label = self.data.label[index]
        X, _  = prepare_features(utterance,sent2)
        y = label_to_ix[self.data.label[index]]
        return X, y
    
    def __len__(self):
        return self.len
train_size = 0.95
train_dataset=dataset.sample(frac=train_size,random_state=200).reset_index(drop=True)
test_dataset=dataset.drop(train_dataset.index).reset_index(drop=True)

training_set = Intents(train_dataset)
testing_set = Intents(test_dataset)
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = model.cuda()
# Parameters
params = {'batch_size': 1,
          'shuffle': True,
          'num_workers': 1}
training_loader = DataLoader(training_set, **params)
testing_loader = DataLoader(testing_set, **params)
loss_function = nn.CrossEntropyLoss()
learning_rate = 1e-05
optimizer = optim.Adam(params =  model.parameters(), lr=learning_rate)

max_epochs = 3
model = model.train()
for epoch in tqdm_notebook(range(max_epochs)):
    print("EPOCH -- {}".format(epoch))
    for i, (sent, label) in enumerate(training_loader):
        optimizer.zero_grad()
        sent = sent.squeeze(0)
        if torch.cuda.is_available():
          sent = sent.cuda()
          label = label.cuda()
        output = model.forward(sent)[0]
        _, predicted = torch.max(output, 1)
        
        loss = loss_function(output, label)
        loss.backward()
        optimizer.step()
        
        if i%100 == 0:
            correct = 0
            total = 0
            for sent, label in testing_loader:
                sent = sent.squeeze(0)
                if torch.cuda.is_available():
                  sent = sent.cuda()
                  label = label.cuda()
                output = model.forward(sent)[0]
                _, predicted = torch.max(output.data, 1)
                total += label.size(0)
                correct += (predicted.cpu() == label.cpu()).sum()
            accuracy = 100.00 * correct.numpy() / total
            print('Iteration: {}. Loss: {}. Accuracy: {}%'.format(i, loss.item(), accuracy))

def get_reply(msg,msg1):
  model.eval()
  input_msg, _ = prepare_features(msg,msg1)
  if torch.cuda.is_available():
    input_msg = input_msg.cuda()
  output = model(input_msg)

```

Thanks 


Comments:
Hi, the provided `run_glue` example shows how to train/use `RoBERTa` for sentence pairs classification on the GLUE tasks (including MNLI).

Hi,
Thanks for your help

I have executed the run_glue.py file on my custom data set by using the following command
`
python run_glue.py --model_type roberta --model_name_or_path roberta-large-mnli --task_name=mnli --do_train --do_eval --do_lower_case --data_dir=input_roberta/ --max_seq_length 28 --per_gpu_eval_batch_size=8 --per_gpu_train_batch_size=8 --output_dir=output_roberta/ --save_steps=350 --overwrite_output_dir --overwrite_cache
`

input_roberta has my train.tsv (custom sentence pair corpus with labels)

Model gets trained, however it asks for dev_matched and dev_mismtached files, How do i provide this?

How do i predict test sentence-pair questions using the generated model weights ?

Thanks

However with above warning model has generated the weights and other files.
Using them I have loaded the model using the below code:
`output_model_file = './output_roberta/pytorch_model.bin'
output_config_file = "./output_roberta/config.json"
output_vocab_file = "./output_roberta/vocab.json"


config = RobertaConfig.from_json_file(output_config_file)
model = RobertaForSequenceClassification(config)
state_dict = torch.load(output_model_file)
model.load_state_dict(torch.load(model_path))
tokenizer =RobertaTokenizer(output_vocab_file,merges_file="./output_roberta/merges.txt")

aa=tokenizer.encode("what is my sales")
bb=tokenizer.encode("top store by net sales")
zz=tokenizer.add_special_tokens_sentences_pair(aa,bb)
input_ids=torch.tensor(zz).unsqueeze(0)

model.eval()
output = model(input_ids)`

output : (tensor([[-5.2188,  2.2234,  2.4296]], grad_fn=<AddmmBackward>),)
​For any sentence pair it gives the same output as above.
Can you please help?
Thanks



Hi

Is there any update on this ?

Have figured out the solution.

Hi @subhamkhemka, what was the solution you found?

Hey @julien-c 

I switched to the fairseq implementation of roberta.
Using train.py to fine tune using roberta mnli weights

> Hi, the provided `run_glue` example shows how to train/use `RoBERTa` for sentence pairs classification on the GLUE tasks (including MNLI).

hi @thomwolf, I have run  run_glue.py to finetuned model on sts ben mark but I don't know how to inference trained model for sentence similarity. Please help me, thank you.