BERT to implement

XiMing_Dong: <p>Hi everyone<br>
I have a question about BERT<br>
I want to bert to encode short desc and long desc instead of lstm and cnn. on the web, I just find something about <strong>BertForSequenceClassification</strong> this model and in this model, it needs to have a label, but in my dataset, there hasn’t label and only have data pairs such as A and B are similar. in my ideas, I just want to use bert mode and fine-tuning and use the same loss function. here is the structure of the model and here is the loss function! here is my code!<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/0/4043d8bd0909aeeecd0f4af7634faffd7c5347b9.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/4043d8bd0909aeeecd0f4af7634faffd7c5347b9" title="Screen Shot 2020-12-18 at 1.00.19 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/0/4043d8bd0909aeeecd0f4af7634faffd7c5347b9_2_690x363.jpeg" alt="Screen Shot 2020-12-18 at 1.00.19 PM" data-base62-sha1="9avUY4jvT3bv4Imjqj7H5t5e2I9" width="690" height="363" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/0/4043d8bd0909aeeecd0f4af7634faffd7c5347b9_2_690x363.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/4/0/4043d8bd0909aeeecd0f4af7634faffd7c5347b9_2_1035x544.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/0/4043d8bd0909aeeecd0f4af7634faffd7c5347b9_2_1380x726.jpeg 2x" data-dominant-color="C5C1C8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2020-12-18 at 1.00.19 PM</span><span class="informations">1764×929 287 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I do not sure what I read is right or not, but the result is not improved.<br>
anyone can help me?<br>
thanks a lot</p>

Abhilash_Srivastava: <p>Sentence pair similarity can be formulated like a binary classification problem. Hence, using BinaryCrossEntropy loss is a good idea.<br>
Can you share the full code (not screenshots) here for better understanding?</p>

XiMing_Dong: <p>Hi bro thanks for your kindly reply</p>

XiMing_Dong: <p>import torch.nn as nn<br>
from transformers import BertModel</p>
<p>from data_generator import *<br>
import torch<br>
import math<br>
import torch.nn.functional as F</p>
<p>SEED=0<br>
torch.manual_seed(SEED)<br>
torch.cuda.manual_seed_all(SEED)<br>
torch.backends.cudnn.deterministic=True<br>
torch.backends.cudnn.benchmark = False<br>
class Residual(nn.Module):<br>
def <strong>init</strong>(self, d, fn):<br>
super(Residual, self).<strong>init</strong>()<br>
self.fn = fn<br>
self.projection = nn.Sequential(nn.Linear(d, d), fn, nn.Linear(d, d))</p>
<p>def forward(self, x):<br>
return self.fn(x + self.projection(x))</p>
<p>class Net(nn.Module):<br>
def <strong>init</strong>(self, args):<br>
super(Net, self).<strong>init</strong>()<br>
self.bert = BertModel.from_pretrained(‘bert-base-uncased’)<br>
for param in self.bert.parameters():<br>
param.requires_grad = False</p>
<pre><code>self.info_proj = nn.Sequential(nn.Linear(args.n_prop, 100), nn.Tanh())
self.residual = Residual(300, nn.Tanh())
self.projection = nn.Linear(300, 100)

self.bert_proj = nn.Sequential(nn.Linear(768, 100), nn.Tanh())
</code></pre>
<p>def forward_cnn(self, x):<br>
context = x[0]<br>
mask = x[1]<br>
pooled = self.bert(context, attention_mask=mask)<br>
return pooled.last_hidden_state</p>
<p>def forward_rnn(self, x):<br>
context = x[0]<br>
mask = x[1]<br>
pooled = self.bert(context, attention_mask=mask)<br>
return pooled.last_hidden_state</p>
<p>def forward(self, x):<br>
info = x[‘info’]<br>
info_feature = self.info_proj(info.float())<br>
bert_long= self.forward_cnn(x[‘desc’])<br>
bert_long = torch.mean( bert_long, dim=1)<br>
bert_long = self.bert_proj(bert_long)<br>
bert_short = self.forward_rnn(x[‘short_desc’])<br>
bert_short = torch.mean( bert_short, dim=1)<br>
bert_short= self.bert_proj(bert_short)</p>
<pre><code>feature = torch.cat([info_feature,bert_long, bert_short], -1)
# feature_res = self.residual(feature)
return self.projection(feature)
</code></pre>
<p>The next part is to generate a mask and ID</p>
<p>for bug_id in batch_bugs:<br>
bug = pickle.load(open(os.path.join(’/content/drive/My Drive/DuplicateBugFinder/openOffice/bugs’,  ‘{}.pkl’.format(bug_id)), ‘rb’))<br>
desc_word.append(bug[‘description_long’][0:500])<br>
short_desc_word.append(bug[‘description_short’])<br>
info_ = np.concatenate((<br>
to_one_hot(bug[‘bug_severity’], info_dict[‘bug_severity’]),<br>
to_one_hot(bug[‘bug_status’], info_dict[‘bug_status’]),<br>
to_one_hot(bug[‘component’], info_dict[‘component’]),<br>
to_one_hot(bug[‘priority’], info_dict[‘priority’]),<br>
to_one_hot(bug[‘product’], info_dict[‘product’]),<br>
to_one_hot(bug[‘version’], info_dict[‘version’])))<br>
info.append(info_)</p>
<p>encoded_long = []<br>
encoded_short = []<br>
for input_long in desc_word:<br>
encoded_input_long = tokenizer.encode(input_long,add_special_tokens = True,)<br>
encoded_long.append(encoded_input_long)</p>
<p>input_ids_long = pad_sequences(encoded_long, maxlen=500, dtype=“long”,value=0, truncating=“post”, padding=“post”)<br>
attention_masks_long = []</p>
<p>for sent in input_ids_long :<br>
att_mask = [int(token_id &gt; 0) for token_id in sent]<br>
attention_masks_long.append(att_mask)<br>
attention_masks_long = torch.tensor( attention_masks_long).cuda()<br>
encoded_long = torch.tensor( input_ids_long ).cuda()</p>
<p>for input_short in short_desc_word:<br>
encoded_input_short = tokenizer.encode(input_short,add_special_tokens = True,)<br>
encoded_short.append(encoded_input_short)<br>
input_ids_short = pad_sequences(encoded_short, maxlen=30, dtype=“long”,value=0, truncating=“post”, padding=“post”)<br>
attention_masks_short = []<br>
for sent in input_ids_short :<br>
att_mask = [int(token_id &gt; 0) for token_id in sent]<br>
attention_masks_short.append(att_mask)<br>
attention_masks_short = torch.tensor(attention_masks_short).cuda()<br>
encoded_short = torch.tensor(input_ids_short).cuda()<br>
info = torch.from_numpy(np.array(info)).type(dtype)cuda()<br>
batch_bugs = dict()<br>
batch_bugs[‘info’] = info<br>
batch_bugs[‘desc’] = (encoded_long,attention_masks_long)<br>
batch_bugs[‘short_desc’] = (encoded_short,attention_masks_short)</p>
<p>return batch_bugs</p>