Alignment of tokens - 'extract_features_aligned_to_words' from fairseq roberta?
Author: sidsvash26
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

I'm using RoBERTa pretrained model to get embeddings for a dataset. But I want to get the embeddings as per the tokenization which is already present in my dataset. So basically I would want to average the embeddings of a token's BPE if that token in my dataset is getting split into different BPEs. fairseq roberta has a method for this as follows: 

```
import torch
from fairseq.models.roberta import alignment_utils
roberta = torch.hub.load('pytorch/fairseq', 'roberta.large')
roberta.eval()

example_string_tokens = ['Dr', 'Greenwalt', 'fixed', 'my', 'neck', 'from', 'a', 'snowboard', 'injury', 'and', 'was', 'way', 'more', 'effective', 'that', 'a', 'regular', 'doctor', '.']
doc = roberta.extract_features_aligned_to_words(" ".join(example_string_tokens))

for tok in doc:
    print('{:10}{} (...)'.format(str(tok), tok.vector[:5]))
```  

The output is:

```

       tensor([-0.0656,  0.0189, -0.0003, -0.0907,  0.0550], grad_fn=<SliceBackward>) (...)
Dr        tensor([ 0.2180, -0.0530, -0.3689, -0.0619, -0.6243], grad_fn=<SliceBackward>) (...)
Greenwalt tensor([ 0.3744,  0.0741, -0.7149,  0.0654, -0.1234], grad_fn=<SliceBackward>) (...)
fixed     tensor([ 0.2132,  0.0841, -0.2535, -0.1404, -0.0060], grad_fn=<SliceBackward>) (...)
my        tensor([ 0.1313, -0.0466, -0.1373,  0.1730,  0.1771], grad_fn=<SliceBackward>) (...)
neck      tensor([ 0.0674, -0.3413, -0.0192,  0.0290, -0.3497], grad_fn=<SliceBackward>) (...)
from      tensor([-0.0301, -0.3562, -0.3798,  0.0687,  0.0290], grad_fn=<SliceBackward>) (...)
a         tensor([-0.2496, -0.1036,  0.0270, -0.0819, -0.2146], grad_fn=<SliceBackward>) (...)
snowboard tensor([ 0.4018,  0.1432, -0.0499,  0.2095, -0.0520], grad_fn=<SliceBackward>) (...)
injury    tensor([ 0.0010, -0.6273, -0.0312, -0.1957, -0.4832], grad_fn=<SliceBackward>) (...)
and       tensor([ 0.0747, -0.3335, -0.0593, -0.3805,  0.0930], grad_fn=<SliceBackward>) (...)
was       tensor([ 0.1501, -0.1334, -0.4789, -0.1974, -0.3096], grad_fn=<SliceBackward>) (...)
way       tensor([-0.2803,  0.3204, -0.1663, -0.4420, -0.2641], grad_fn=<SliceBackward>) (...)
more      tensor([-0.1037,  0.1878, -0.5839, -0.4437, -0.1683], grad_fn=<SliceBackward>) (...)
effective tensor([-0.1794,  0.2419, -0.3182, -0.2252, -0.1534], grad_fn=<SliceBackward>) (...)
that      tensor([-0.1146, -0.1935, -0.3615, -0.4998, -0.1000], grad_fn=<SliceBackward>) (...)
a         tensor([-0.2107, -0.2103, -0.1996,  0.0046, -0.1112], grad_fn=<SliceBackward>) (...)
regular   tensor([ 0.2236, -0.0613, -0.5496, -0.3562,  0.1022], grad_fn=<SliceBackward>) (...)
doctor    tensor([ 0.1275, -0.0589, -0.0283, -0.1557, -0.9282], grad_fn=<SliceBackward>) (...)
.         tensor([ 0.1765,  0.0812, -0.1684, -0.2818,  0.0134], grad_fn=<SliceBackward>) (...)

      tensor([-0.0409, -0.0024,  0.0107, -0.0183, -0.0479], grad_fn=<SliceBackward>) (...)


```

I want to be able to do something similar with the hugging transformers library, but I can't find any alignment methods. 

Any suggestions?

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


It seems this is still unsupported by Huggingface?