Best practices for Bert passage similarity. Perhaps further processing Bert vectors; Bert model inside another model 
Author: Santosh-Gupta
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->

I'm trying develop Bert passage similarity, specifically question/ answer-retrieval. The architecture is pooling bert contextualized embeddings for passages of text, and then doing cos similarity. Basically the architecture is as it's described here

https://github.com/re-search/DocProduct#architecture

Unless I skipped over something, as far as I can tell, there isn't an established way to do this in the Transformer model. It looks like I have to take the Bert output, and do further processing on it. So in this case, Bert would be a component of a larger model. 

This is the best I could come up with up some prototype code. 

```
!pip install transformers

import torch
from transformers import BertModel, BertTokenizer
from transformers import AdamW
import torch.nn as nn
import torch.nn.functional as F

# Overall model, with Bert model inside it
class OverallModel(nn.Module):
    def __init__(self):
        super(OverallModel, self).__init__()
        self.tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
        self.bertModel = BertModel.from_pretrained('bert-base-uncased')
        self.lossFunction = nn.BCEWithLogitsLoss( reduction = 'none' )

    def forward(self, queries, docs, targets ):
        #Set bert model to train mode
        self.bertModel.train()

        #tokenize queries and pad to maximum length 
        queriesT = [self.tokenizer.encode(piece, add_special_tokens=True) for piece in queries] 
        widthQ = max(len(d) for d in queriesT)
        queriesP = [d + [0] * (widthQ - len(d)) for d in queriesT]
        queriesTen= torch.tensor(queriesP)

        #tokenize answers and pad to maximum length 
        docsT = [ self.tokenizer.encode(piece, add_special_tokens=True) for piece in docs] 
        widthD = max(len(d) for d in docsT)
        docsP = [d + [0] * (widthD - len(d)) for d in docsT]
        docsTen= torch.tensor(docsP)

        #Get contextualized Bert embeddings 
        bertEmbedsQ = self.bertModel(queriesTen)[0]
        bertEmbedsA = self.bertModel(docsTen)[0]

        #Mean pool embeddings for each text, and normalize to unit vectors
        bertEmbedsQPooled = torch.mean(bertEmbedsQ, 1)
        bertEmbedsAPooled = torch.mean(bertEmbedsA, 1)
        bertEmbedsQPN = F.normalize(bertEmbedsQPooled, p=2, dim=1)
        bertEmbedsAPN = F.normalize(bertEmbedsAPooled, p=2, dim=1)

        #Take dot products of query and answer embeddings, calculate loss with target. 
        dotProds = torch.bmm(bertEmbedsQPN.unsqueeze(1), bertEmbedsAPN.unsqueeze(2) ).squeeze()
        indLoss = self.lossFunction( dotProds, targets )
        finalLoss = torch.mean(indLoss)

        return finalLoss

QAmodel = OverallModel()
optimizer = AdamW(QAmodel.parameters())
optimizer.zero_grad()

#Prepare inputs and target
queriesI = ['What do cats eat', 'where do apples come from', 'Who is Mr Rogers', 'What are kites']
docsI = ['they eat catfood', 'from trees' , 'President of Iceland', 'used for diving']
targets = torch.tensor( [1, 1, 0 ,0] , dtype = torch.float32 )

#Caculate loss, gradients, and update weights
loss = QAmodel.forward(queriesI, docsI, targets)
loss.backward()
optimizer.step()
```

This approach seems to be working well. The gradients are flowing all the way through, the weights are updating, etc. 

Is this code the best way to train Bert passage similarity. 

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.