Concatenating data with the output of bert

Omama: <p>I have model A, which is BERT, and model B, which is also BERT. Each model has different input features.</p>
<p>I removed the classifier layer from each model before constructing a parent model in which I will perform concatenation for three distinct features:</p>
<p>-The output of model A (text)</p>
<p>-The output of model B (POS)</p>
<p>-And other features (subjectivity)</p>
<p>Then I will make it input to the fully connected layer and make the classification</p>
<p>The architecture will be like this<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/1/216895a5d5a4c6bfd25bd56f744268b8733fba42.png" data-download-href="https://discuss.pytorch.org/uploads/default/216895a5d5a4c6bfd25bd56f744268b8733fba42" title="Screen Shot 2021-11-02 at 2.50.31 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/1/216895a5d5a4c6bfd25bd56f744268b8733fba42_2_400x250.png" alt="Screen Shot 2021-11-02 at 2.50.31 PM" data-base62-sha1="4LxPkvhzTuYrNS9Z9pYDwejoV4S" width="400" height="250" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/1/216895a5d5a4c6bfd25bd56f744268b8733fba42_2_400x250.png, https://discuss.pytorch.org/uploads/default/optimized/3X/2/1/216895a5d5a4c6bfd25bd56f744268b8733fba42_2_600x375.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/2/1/216895a5d5a4c6bfd25bd56f744268b8733fba42_2_800x500.png 2x" data-dominant-color="D0D9EC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2021-11-02 at 2.50.31 PM</span><span class="informations">1592×1080 56.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>the problem is that the loss too high 218 and the accuracy not improved compared to the results before adding the new features which is not input to any model…</p>
<p>I think I should do normalization but I don’t know how…</p>
<p>This is the parent model:</p>
<p>class ParentModel(torch.nn.Module):</p>
<pre><code>def __init__(self, modelA, modelB):
    super(ParentModel, self).__init__()
    self.modelA = modelA
    self.modelB = modelB
    self.fc1 = torch.nn.Linear(2560, 1536)
    self.classifier = torch.nn.Linear(1536, 3)
    
def forward(self, x1, x2, x3, x4, x5, x6):
    x11 = self.modelA(x1, x2)
    x22 = self.modelB(x3, x4)
    x = torch.cat((x11, x22, x5, x6), dim=1)
    x = torch.nn.ReLU()(self.fc1(x))

    x = self.classifier(x)

    return x
</code></pre>
<p>ids1 = batch1[‘ids’].to(device, dtype = torch.long)<br>
mask1 = batch1[‘mask’].to(device, dtype = torch.long)<br>
targets1 = batch1[‘targets’].to(device, dtype = torch.long)</p>
<p>ids2 = batch2[‘ids’].to(device, dtype = torch.long)<br>
mask2 = batch2[‘mask’].to(device, dtype = torch.long)</p>
<p>ids3 = batch3[‘ids’].to(device, dtype = torch.long)<br>
mask3 = batch3[‘mask’].to(device, dtype = torch.long)</p>
<p>outputs = model(ids1, mask1, ids2, mask2, ids3, mask3)</p>
<p>Please advise if the problem is not related to the normalization issue.</p>
<p>this is the results:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/9/99c3379878d1aa111aa3767d69d6b140cbe8a514.png" data-download-href="https://discuss.pytorch.org/uploads/default/99c3379878d1aa111aa3767d69d6b140cbe8a514" title="Screen Shot 2021-11-02 at 6.53.48 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/9/99c3379878d1aa111aa3767d69d6b140cbe8a514_2_400x200.png" alt="Screen Shot 2021-11-02 at 6.53.48 PM" data-base62-sha1="lWfiPWSYPVQr6Tfw1vGSzzh19PK" width="400" height="200" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/9/99c3379878d1aa111aa3767d69d6b140cbe8a514_2_400x200.png, https://discuss.pytorch.org/uploads/default/optimized/3X/9/9/99c3379878d1aa111aa3767d69d6b140cbe8a514_2_600x300.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/9/9/99c3379878d1aa111aa3767d69d6b140cbe8a514_2_800x400.png 2x" data-dominant-color="DEDEDF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2021-11-02 at 6.53.48 PM</span><span class="informations">942×682 147 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The results for concatenating just the output of model A and model B is:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/c/2ce425aeb2cd6a5975b54458e00c3dd16365dbd8.png" data-download-href="https://discuss.pytorch.org/uploads/default/2ce425aeb2cd6a5975b54458e00c3dd16365dbd8" title="Screen Shot 2021-11-02 at 7.57.07 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/c/2ce425aeb2cd6a5975b54458e00c3dd16365dbd8_2_400x250.png" alt="Screen Shot 2021-11-02 at 7.57.07 PM" data-base62-sha1="6p7OkCd8ktMwP3AOZSENvgWvvTy" width="400" height="250" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/c/2ce425aeb2cd6a5975b54458e00c3dd16365dbd8_2_400x250.png, https://discuss.pytorch.org/uploads/default/optimized/3X/2/c/2ce425aeb2cd6a5975b54458e00c3dd16365dbd8_2_600x375.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/2/c/2ce425aeb2cd6a5975b54458e00c3dd16365dbd8_2_800x500.png 2x" data-dominant-color="E0E0E0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2021-11-02 at 7.57.07 PM</span><span class="informations">950×498 45.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

valgi0: <p>Hi:)</p>
<p>Maybe you should share some information more as training parameters such as lr, epochs, optimizer, dropout.</p>
<p>Are you using model BERT from huggingface?</p>
<p>Which are the input features?</p>
<p>Try to expand a bit the question in order to let us better understand <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=10" title=":wink:" class="emoji" alt=":wink:"></p>
<p>Anyway I will investigate the learning rate. If it is to high the model do not converge, try to lower it and add some dropout and weights normalization.</p>