Roberta embeddings comparison
Author: aclifton314
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help
**SYSTEM**
OS: Linux pop-os 5.0.0
Python version: 3.6.8
Torch version: 1.3.0
Transformers version: 2.1.1
I am running this linux VM with the above software versions on a Windows 10 laptop.

I wanted to compare the cosine similarity between elmo and roberta embeddings for two sequences:
```python
from allennlp.commands.elmo import ElmoEmbedder
from allennlp.data.tokenizers.spacy_tokenizer import SpacyTokenizer
from scipy.spatial.distance import cdist
import numpy as np
import pandas as pd
from transformers import RobertaTokenizer, RobertaModel
import torch

seqA = 'How many cars are in the target parking lot'
seqB = 'How many countries are in the continent of Africa'
comparison_df = pd.DataFrame()
sim_list = []

model_types = ['elmo', 'roberta']
for model_type in model_types:
    sim_list = []
    if model_type == 'elmo':
        tokenizer = SpacyTokenizer()
        model = ElmoEmbedder()
        seqA_tokens = np.array([s.text for s in tokenizer.tokenize(seqA)])
        seqB_tokens = np.array([s.text for s in tokenizer.tokenize(seqB)])

        seqA_embeddings = model.embed_sentence(seqA_tokens)
        seqB_embeddings = model.embed_sentence(seqB_tokens)

        seqA_embeds = np.sum(seqA_embeddings[-1], axis=0).reshape((1, seqA_embeddings.shape[2]))
        seqB_embeds = np.sum(seqB_embeddings[-1], axis=0).reshape((1, seqB_embeddings.shape[2]))
        Y = cdist(seqA_embeds, seqB_embeds, 'cosine')[0][0]
        cos_sim = 1.0 - Y
        sim_list.append(cos_sim)


    elif model_type == 'roberta':
        tokenizer = RobertaTokenizer.from_pretrained('roberta-large')
        model = RobertaModel.from_pretrained('roberta-large')
        seqA_tokens = torch.tensor(tokenizer.encode(seqA, add_special_tokens=True)).unsqueeze(0)
        seqB_tokens = torch.tensor(tokenizer.encode(seqB, add_special_tokens=True)).unsqueeze(0)

        with torch.no_grad():
            seqA_embeddings = model(seqA_tokens)[0]
            seqB_embeddings = model(seqB_tokens)[0]

        seqA_embeddings = seqA_embeddings.detach().numpy()
        seqA_embeddings = np.sum(seqA_embeddings[-1], axis=0).reshape((1, seqA_embeddings.shape[2]))

        seqB_embeddings = seqB_embeddings.detach().numpy()
        seqB_embeddings = np.sum(seqB_embeddings[-1], axis=0).reshape((1, seqB_embeddings.shape[2]))

        Y = cdist(seqA_embeddings, seqB_embeddings, 'cosine')[0][0]
        cos_sim = 1.0 - Y
        sim_list.append(cos_sim)

    comparison_df[model_type] = sim_list
print(comparison_df)
```

The result of the print statement is:
```
      elmo   roberta
0  0.55092  0.996526
```

I find this strange because, despite having the same number of tokens, I would expect the similarity between the two sequences to not be so close to 1 as roberta is showing. My concern is that I am not accessing the embeddings correctly or not pooling correctly (or both), and was wondering if people here wouldn't mind letting me know what I am doing wrong? Thanks in advance!


Comments:
Any update on this issue?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


> ## ❓ Questions & Help
> **SYSTEM**
> OS: Linux pop-os 5.0.0
> Python version: 3.6.8
> Torch version: 1.3.0
> Transformers version: 2.1.1
> I am running this linux VM with the above software versions on a Windows 10 laptop.
> 
> I wanted to compare the cosine similarity between elmo and roberta embeddings for two sequences:
> 
> ```python
> from allennlp.commands.elmo import ElmoEmbedder
> from allennlp.data.tokenizers.spacy_tokenizer import SpacyTokenizer
> from scipy.spatial.distance import cdist
> import numpy as np
> import pandas as pd
> from transformers import RobertaTokenizer, RobertaModel
> import torch
> 
> seqA = 'How many cars are in the target parking lot'
> seqB = 'How many countries are in the continent of Africa'
> comparison_df = pd.DataFrame()
> sim_list = []
> 
> model_types = ['elmo', 'roberta']
> for model_type in model_types:
>     sim_list = []
>     if model_type == 'elmo':
>         tokenizer = SpacyTokenizer()
>         model = ElmoEmbedder()
>         seqA_tokens = np.array([s.text for s in tokenizer.tokenize(seqA)])
>         seqB_tokens = np.array([s.text for s in tokenizer.tokenize(seqB)])
> 
>         seqA_embeddings = model.embed_sentence(seqA_tokens)
>         seqB_embeddings = model.embed_sentence(seqB_tokens)
> 
>         seqA_embeds = np.sum(seqA_embeddings[-1], axis=0).reshape((1, seqA_embeddings.shape[2]))
>         seqB_embeds = np.sum(seqB_embeddings[-1], axis=0).reshape((1, seqB_embeddings.shape[2]))
>         Y = cdist(seqA_embeds, seqB_embeds, 'cosine')[0][0]
>         cos_sim = 1.0 - Y
>         sim_list.append(cos_sim)
> 
> 
>     elif model_type == 'roberta':
>         tokenizer = RobertaTokenizer.from_pretrained('roberta-large')
>         model = RobertaModel.from_pretrained('roberta-large')
>         seqA_tokens = torch.tensor(tokenizer.encode(seqA, add_special_tokens=True)).unsqueeze(0)
>         seqB_tokens = torch.tensor(tokenizer.encode(seqB, add_special_tokens=True)).unsqueeze(0)
> 
>         with torch.no_grad():
>             seqA_embeddings = model(seqA_tokens)[0]
>             seqB_embeddings = model(seqB_tokens)[0]
> 
>         seqA_embeddings = seqA_embeddings.detach().numpy()
>         seqA_embeddings = np.sum(seqA_embeddings[-1], axis=0).reshape((1, seqA_embeddings.shape[2]))
> 
>         seqB_embeddings = seqB_embeddings.detach().numpy()
>         seqB_embeddings = np.sum(seqB_embeddings[-1], axis=0).reshape((1, seqB_embeddings.shape[2]))
> 
>         Y = cdist(seqA_embeddings, seqB_embeddings, 'cosine')[0][0]
>         cos_sim = 1.0 - Y
>         sim_list.append(cos_sim)
> 
>     comparison_df[model_type] = sim_list
> print(comparison_df)
> ```
> 
> The result of the print statement is:
> 
> ```
>       elmo   roberta
> 0  0.55092  0.996526
> ```
> 
> I find this strange because, despite having the same number of tokens, I would expect the similarity between the two sequences to not be so close to 1 as roberta is showing. My concern is that I am not accessing the embeddings correctly or not pooling correctly (or both), and was wondering if people here wouldn't mind letting me know what I am doing wrong? Thanks in advance!

@aclifton314 have you deduce any meaningful intuition on this thus far ?? I found another similar issue, hopefully this can help: https://github.com/huggingface/transformers/issues/2298 ...

@aclifton314 

In addition to the above link, I also ran a quick test with most popular models not just Roberta, it seems there are some models that can differentiate better, though I am not sure if you can generalize that statement without the test. Code and test results below:

```
import torch
from torch import nn
from transformers import *

import numpy as np
from scipy.spatial import distance
from sklearn.metrics.pairwise import cosine_similarity

# for 10 transformer architectures and 30 pretrained weights.
#          Model          | Tokenizer          | Pretrained weights shortcut
MODELS = [(BertModel,       BertTokenizer,       'bert-base-uncased'),
          (OpenAIGPTModel,  OpenAIGPTTokenizer,  'openai-gpt'),
          (GPT2Model,       GPT2Tokenizer,       'gpt2'),
          (CTRLModel,       CTRLTokenizer,       'ctrl'),
          (TransfoXLModel,  TransfoXLTokenizer,  'transfo-xl-wt103'),
          (XLNetModel,      XLNetTokenizer,      'xlnet-base-cased'),
          (XLMModel,        XLMTokenizer,        'xlm-mlm-enfr-1024'),
          (DistilBertModel, DistilBertTokenizer, 'distilbert-base-uncased'),
          (RobertaModel,    RobertaTokenizer,    'roberta-base'),
          (XLMRobertaModel, XLMRobertaTokenizer, 'xlm-roberta-base'),
         ]

for model_class, tokenizer_class, pretrained_weights in MODELS:
    tokenizer = tokenizer_class.from_pretrained(pretrained_weights)
    model = model_class.from_pretrained(pretrained_weights)

    print ("")
    print ("Model", ''.join(e for e in str(type(model)).split(".")[-1] if e.isalnum()))

    input_ids1 = torch.tensor([tokenizer.encode("How many cars are in the target parking lot", add_special_tokens=True)])                                            
    input_ids2 = torch.tensor([tokenizer.encode("How many countries are in the continent of Africa", add_special_tokens=True)])  

    with torch.no_grad():
        last_hidden_states1 = model(input_ids1)[0].detach().numpy()
        last_hidden_states2 = model(input_ids2)[0].detach().numpy()
        
        last_hidden_states1_sq = np.sum(last_hidden_states1[-1], axis=0).reshape((1, last_hidden_states1.shape[2]))
        last_hidden_states2_sq = np.sum(last_hidden_states2[-1], axis=0).reshape((1, last_hidden_states2.shape[2]))
        
        print ("Similarity between Sent1 & Sent 2", cosine_similarity(last_hidden_states1_sq, last_hidden_states2_sq) )

```
**Results**:

Model BertModel
Similarity between Sent1 & Sent 2 [[0.7234119]]

Model OpenAIGPTModel
Similarity between Sent1 & Sent 2 [[0.6779459]]

Model GPT2Model
Similarity between Sent1 & Sent 2 [[0.9986031]]

Model CTRLModel
Similarity between Sent1 & Sent 2 [[0.9086089]]

Model TransfoXLModel
Similarity between Sent1 & Sent 2 [[0.77547705]]

Model XLNetModel
Similarity between Sent1 & Sent 2 [[0.98956037]]

Model XLMModel
Similarity between Sent1 & Sent 2 [[0.623127]]

Model DistilBertModel
Similarity between Sent1 & Sent 2 [[0.7640958]]

Model RobertaModel
Similarity between Sent1 & Sent 2 [[0.96598077]]

Model XLMRobertaModel
Similarity between Sent1 & Sent 2 [[0.99813896]]

**I am interested in Passage (Few Paragraphs) Level Embeddings based Similarity Comparison, if anyone has any viable solution, suggestions please do share.** 

Thanks!!