Concatenate the output of Bert and transformer

Omama: <p>How I can feed two pytorch models with different data, then concatenate the output of the two models before prediction.</p>

ptrblck: <p>You could basically write the model in the same way you’ve described it:</p>
<pre><code class="lang-python">out1 = model1(in1)
out2 = model2(in2)
out = torch.cat((out1, out2), dim=1)

pred = classifier(out)
</code></pre>

Omama: <p>Thanks for your response. I’m still confused. This is what I did, But I’m stuck in point number 3:<br>
1- I created two models (BERTClassA, BERTClassB), and detatch the classifier layers from them.<br>
2- Then I concatenated the output from the two models in train function.<br>
3- Then How I can loop through each data loader and pass each one to the models, then taking the output back from two models and concatenate it, after that pass it to the ensamble class.</p>
<h1>
<a name="data-1" class="anchor" href="#data-1"></a>Data</h1>
<p>training_loader1 = DataLoader(training_set2, **train_params)<br>
training_loader2 = DataLoader(training_set2, **train_params)</p>
<h1>
<a name="model-a-2" class="anchor" href="#model-a-2"></a>Model A</h1>
<p>class BERTClassA(torch.nn.Module):<br>
def <strong>init</strong>(self):<br>
super(BERTClassB, self).<strong>init</strong>()<br>
self.l1 = BertModel.from_pretrained(“bert-base-uncased”)<br>
self.pre_classifier = torch.nn.Linear(768, 768)<br>
self.dropout = torch.nn.Dropout(.3)</p>
<pre><code>def forward(self, input_ids, attention_mask):
    output_1 = self.l1(input_ids=input_ids, attention_mask=attention_mask)
    hidden_state = output_1[0]
    pooler = hidden_state[:, 0]
    pooler = self.pre_classifier(pooler)
    pooler = torch.nn.ReLU()(pooler)
    pooler = self.dropout(pooler)
    return output
</code></pre>
<h1>
<a name="model-b-3" class="anchor" href="#model-b-3"></a>Model B</h1>
<p>class BERTClassB(torch.nn.Module):<br>
def <strong>init</strong>(self):<br>
super(BERTClassB, self).<strong>init</strong>()<br>
self.l1 = BertModel.from_pretrained(“bert-base-uncased”)<br>
self.pre_classifier = torch.nn.Linear(768, 768)<br>
self.dropout = torch.nn.Dropout(.3)</p>
<pre><code>def forward(self, input_ids, attention_mask):
    output_1 = self.l1(input_ids=input_ids, attention_mask=attention_mask)
    hidden_state = output_1[0]
    pooler = hidden_state[:, 0]
    pooler = self.pre_classifier(pooler)
    pooler = torch.nn.ReLU()(pooler)
    pooler = self.dropout(pooler)
    return output
</code></pre>
<h1>
<a name="train-function-4" class="anchor" href="#train-function-4"></a>Train function</h1>
<p>def train(epoch):<br>
tr_loss = 0<br>
n_correct = 0<br>
nb_tr_steps = 0<br>
nb_tr_examples = 0<br>
model.train()</p>
<pre><code>for _,data1  in enumerate(training_loader1 , 0):
    for _,data2  in enumerate(training_loader2 , 0):
        ids = data1['ids'].to(device, dtype = torch.long)
        mask = data1['mask'].to(device, dtype = torch.long)
        targets1 = data1['targets'].to(device, dtype = torch.long)

        ids2 = data2['ids'].to(device, dtype = torch.long)
        mask2 = data2['mask'].to(device, dtype = torch.long)
        targets2 = data2['targets'].to(device, dtype = torch.long)



        outputs1 = modelBERT(ids, mask)
        outputs2 = modelBERT2(ids2, mask2)

        outputs = torch.cat((outputs1, outputs2), dim=1)

        

        
        ensamble_output = model(outputs)


        loss = loss_function(ensamble_output, targets1)
        tr_loss += loss.item()
        big_val, big_idx = torch.max(ensamble_output.data, dim=1)
        n_correct += calcuate_accu(big_idx, targets1)

        nb_tr_steps += 1
        nb_tr_examples+=targets1.size(0)
    
        if _%100==0:
            loss_step = tr_loss/nb_tr_steps
            accu_step = (n_correct*100)/nb_tr_examples 
            print(f"Training Loss per 100 steps: {loss_step}")
            print(f"Training Accuracy per 100 steps: {accu_step}")

        optimizer.zero_grad()
        loss.backward()

        optimizer.step()

    print(f'The Total Accuracy for Epoch {epoch}: {(n_correct*100)/nb_tr_examples}')
    epoch_loss = tr_loss/nb_tr_steps
    epoch_accu = (n_correct*100)/nb_tr_examples
    print(f"Training Loss Epoch: {epoch_loss}")
    print(f"Training Accuracy Epoch: {epoch_accu}")

    return 
</code></pre>
<p>for epoch in range(EPOCHS):<br>
train(epoch)</p>
<h1>
<a name="ensamble-function-5" class="anchor" href="#ensamble-function-5"></a>Ensamble function</h1>
<p>class MyEnsemble(torch.nn.Module):<br>
def <strong>init</strong>(self, modelA, modelB):<br>
super(MyEnsemble, self).<strong>init</strong>()<br>
self.modelA = modelA<br>
self.modelB = modelB<br>
self.classifier = torch.nn.Linear(768, 3)</p>
<pre><code>def forward(self, x1, x2):
    x1 = self.modelA(x1)
    x2 = self.modelB(x2)
    x = torch.cat((x1, x2), dim=1)
    x = self.classifier(F.relu(x))
    return x
</code></pre>
<p>modelA = BERTClass()<br>
modelB = BERTClassB()</p>
<p>model = MyEnsemble(modelA, modelB)<br>
model.to(device)</p>

ptrblck: <p>I assume you would like to pass batches from <code>training_loader1</code> as <code>x1</code> and batches from <code>training_loader2</code> as <code>x2</code> to the model. If so, you could either iterate both loaders via <code>zip</code> or load the batches manually via:</p>
<pre><code class="lang-python">iter_loader1 = iter(training_loader1)
batch1 = next(iter_loader1)
...
</code></pre>

Omama: <p>Thanks a lot Ptrblck. Your answer worked with me.<br>
Please help me with this error<br>
TypeError: not all arguments converted during string formatting</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/2/326df0eb1b4ae5ba630bb07a7eb2a299432940b1.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/326df0eb1b4ae5ba630bb07a7eb2a299432940b1" title="Screen Shot 2021-10-19 at 5.29.34 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/2/326df0eb1b4ae5ba630bb07a7eb2a299432940b1_2_690x488.jpeg" alt="Screen Shot 2021-10-19 at 5.29.34 PM" data-base62-sha1="7c7q79yBhGkvUwb2NiSkgVUyKzL" width="690" height="488" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/2/326df0eb1b4ae5ba630bb07a7eb2a299432940b1_2_690x488.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/3/2/326df0eb1b4ae5ba630bb07a7eb2a299432940b1_2_1035x732.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/3/2/326df0eb1b4ae5ba630bb07a7eb2a299432940b1_2_1380x976.jpeg 2x" data-dominant-color="F4F7F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2021-10-19 at 5.29.34 PM</span><span class="informations">1920×1359 149 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/2/e2d9233c6e5306db482b49d199f16847be011f46.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/e2d9233c6e5306db482b49d199f16847be011f46" title="Screen Shot 2021-10-19 at 5.29.19 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/2/e2d9233c6e5306db482b49d199f16847be011f46_2_690x471.jpeg" alt="Screen Shot 2021-10-19 at 5.29.19 PM" data-base62-sha1="wmN7F0xtrr34eCrTeio2fBBI7cO" width="690" height="471" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/2/e2d9233c6e5306db482b49d199f16847be011f46_2_690x471.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/e/2/e2d9233c6e5306db482b49d199f16847be011f46_2_1035x706.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/e/2/e2d9233c6e5306db482b49d199f16847be011f46_2_1380x942.jpeg 2x" data-dominant-color="F3F6F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2021-10-19 at 5.29.19 PM</span><span class="informations">1916×1310 169 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<h1><a name="training-function-1" class="anchor" href="#training-function-1"></a>Training function</h1>
<p>for i in range(TRAIN_BATCH_SIZE):</p>
<pre><code>    iter_loader1 = iter(training_loader1)
    batch1 = next(iter_loader1)
    iter_loader2 = iter(training_loader2)
    batch2 = next(iter_loader2)
    
    
    ids1 = batch1['ids'].to(device, dtype = torch.long)
    mask1 = batch1['mask'].to(device, dtype = torch.long)
    targets1 = batch1['targets'].to(device, dtype = torch.long)
    
    
    ids2 = batch2['ids'].to(device, dtype = torch.long)
    mask2 = batch2['mask'].to(device, dtype = torch.long)
    targets2 = batch2['targets'].to(device, dtype = torch.long)
    
    outputs = model(ids1, mask1, ids2, mask2)

    print(outputs)
    loss = loss_function(outputs, targets2)
    print(loss)
    tr_loss += loss.item()
    print(tr_loss)
    big_val, big_idx = torch.max(outputs.data, dim=1)
    print(big_val, big_idx)
    n_correct += calcuate_accu(big_idx, targets2)

    nb_tr_steps += 1
    print(nb_tr_steps)
    nb_tr_examples+=targets2.size(0)
    print(nb_tr_examples)
    #print (_)
    
    if _%100==0:
        loss_step = tr_loss/nb_tr_steps
        accu_step = (n_correct*100)/nb_tr_examples 
        print(f"Training Loss per 100 steps: {loss_step}")
        print(f"Training Accuracy per 100 steps: {accu_step}")

    optimizer.zero_grad()
    loss.backward()
    # # When using GPU
    optimizer.step()

print(f'The Total Accuracy for Epoch {epoch}: {(n_correct*100)/nb_tr_examples}')
epoch_loss = tr_loss/nb_tr_steps
epoch_accu = (n_correct*100)/nb_tr_examples
print(f"Training Loss Epoch: {epoch_loss}")
print(f"Training Accuracy Epoch: {epoch_accu}")

return
</code></pre>

ptrblck: <p>The new error is raised by pandas and I don’t know which line of code is causing the issue.<br>
My guess would be <code>pandas</code> might be used for some preprocessing, which might fail?</p>

Omama: <p>This line caused the error:<br>
if _%100==0:</p>
<p>When I print the result for _ using one model without ensambling, it is just an integer number, but after I made my update with ensambling, this variable _ is a dataframe. But I don’t know how to fix it.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/8/38e5ce962c10486a8a311511d08896f8937a9247.png" data-download-href="https://discuss.pytorch.org/uploads/default/38e5ce962c10486a8a311511d08896f8937a9247" title="Screen Shot 2021-10-19 at 11.46.46 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/8/38e5ce962c10486a8a311511d08896f8937a9247_2_547x500.png" alt="Screen Shot 2021-10-19 at 11.46.46 PM" data-base62-sha1="87l62TmLpfJE8tDCxfxcIoMDFIP" width="547" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/8/38e5ce962c10486a8a311511d08896f8937a9247_2_547x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/3/8/38e5ce962c10486a8a311511d08896f8937a9247_2_820x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/3/8/38e5ce962c10486a8a311511d08896f8937a9247_2_1094x1000.png 2x" data-dominant-color="F4F4F9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2021-10-19 at 11.46.46 PM</span><span class="informations">1690×1544 437 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>The <code>_</code> might refer to the last return value in this case, wouldn’t it?<br>
Change the <code>_</code> to an explicit variable which tracks the iterations and it should work.</p>

Omama: <p>Thanks a lot Ptrblck… your answers very helpful</p>