Ensemble of five Transformers for text classification

the_coder: <p>I am trying to ensemble 5 transformers inspired by<br>
<a href="https://discuss.pytorch.org/t/concatenate-the-output-of-bert-and-transformer/134446" class="inline-onebox">Concatenate the output of Bert and transformer</a>.</p>
<p>My code for the model is following:</p>
<pre><code class="lang-auto">
class BERTClassA(torch.nn.Module):
  def init(self):
    super(BERTClassA, self).init()
    self.l1 = BertModel.from_pretrained('bert-base-uncased')
    self.pre_classifier = torch.nn.Linear(768, 768)
    self.dropout = torch.nn.Dropout(.3)
  
  def forward(self, input_ids, attention_mask):
    output_1 = self.l1(input_ids=input_ids, attention_mask=attention_mask)
    hidden_state = output_1[0]
    pooler = hidden_state[:, 0]
    pooler = self.pre_classifier(pooler)
    pooler = torch.nn.ReLU()(pooler)
    output = self.dropout(pooler)
    return output

class BERTClassB(torch.nn.Module):
  def init(self):
    super(BERTClassB, self).init()
    self.l2 = TFRobertaModel.from_pretrained('roberta-base')
    self.pre_classifier = torch.nn.Linear(768, 768)
    self.dropout = torch.nn.Dropout(.3)

  def forward(self, input_ids, attention_mask):
    output_2 = self.l2(input_ids=input_ids, attention_mask=attention_mask)
    hidden_state = output_2[0]
    pooler = hidden_state[:, 0]
    pooler = self.pre_classifier(pooler)
    pooler = torch.nn.ReLU()(pooler)
    output = self.dropout(pooler)
    return output

class BERTClassC(torch.nn.Module):
  def init(self):
    super(BERTClassC, self).init()
    self.l3 = XLNetForSequenceClassification.from_pretrained('xlnet-base-cased', num_labels = 2)
    self.pre_classifier = torch.nn.Linear(768, 768)
    self.dropout = torch.nn.Dropout(.3)

  def forward(self, input_ids, attention_mask):
    output_3 = self.l3(input_ids=input_ids, attention_mask=attention_mask)
    hidden_state = output_3[0]
    pooler = hidden_state[:, 0]
    pooler = self.pre_classifier(pooler)
    pooler = torch.nn.ReLU()(pooler)
    output = self.dropout(pooler)
    return output


class BERTClassD(torch.nn.Module):
  def init(self):
    super(BERTClassD, self).init()
    self.l4 = DistilBertModel.from_pretrained('distilbert-base-uncased',output_hidden_states=True)
    self.pre_classifier = torch.nn.Linear(768, 768)
    self.dropout = torch.nn.Dropout(.3)

  def forward(self, input_ids, attention_mask):
    output_4 = self.l4(input_ids=input_ids, attention_mask=attention_mask)
    hidden_state = output_4[0]
    pooler = hidden_state[:, 0]
    pooler = self.pre_classifier(pooler)
    pooler = torch.nn.ReLU()(pooler)
    output = self.dropout(pooler)
    return output



class BERTClassE(torch.nn.Module):
  def init(self):
    super(BERTClassE, self).init()
    self.l5 = ElectraForSequenceClassification.from_pretrained('google/electra-base-discriminator',num_labels=2,return_dict= True)
    self.pre_classifier = torch.nn.Linear(768, 768)
    self.dropout = torch.nn.Dropout(.3)

  def forward(self, input_ids, attention_mask):
    output_5 = self.l5(input_ids=input_ids, attention_mask=attention_mask)
    hidden_state = output_5[0]
    pooler = hidden_state[:, 0]
    pooler = self.pre_classifier(pooler)
    pooler = torch.nn.ReLU()(pooler)
    output = self.dropout(pooler)
    return output

class MyEnsemble(torch.nn.Module):
  def init(self, modelA, modelB,modelC,modelD,modelE):
    super(MyEnsemble, self).init()
    self.modelA = modelA
    self.modelB = modelB
    self.modelC=modelC
    self.modelD=modelD
    self.modelE=modelE
    self.classifier = torch.nn.Linear(768, 6)
  def forward(self, x1, x2,x3,x4,x5):
    x1 = self.modelA(x1)
    x2 = self.modelB(x2)
    x3 = self.modelC(x3)
    x4 = self.modelD(x4)
    x5 = self.modelE(x5)
    x = torch.cat((x1, x2, x3, x4, x5), dim=1)
    x = self.classifier(F.relu(x))
    return x
</code></pre>
<p>However while running the training epoch i am getting the following error,</p>
<pre><code class="lang-auto">
Epoch 1/4
----------

Truncation was not explicitly activated but `max_length` is provided a specific value, please use `truncation=True` to explicitly truncate examples to max length. Defaulting to 'longest_first' truncation strategy. If you encode pairs of sequences (GLUE-style) with the tokenizer you can select this strategy more precisely by providing a specific strategy to `truncation`.
Truncation was not explicitly activated but `max_length` is provided a specific value, please use `truncation=True` to explicitly truncate examples to max length. Defaulting to 'longest_first' truncation strategy. If you encode pairs of sequences (GLUE-style) with the tokenizer you can select this strategy more precisely by providing a specific strategy to `truncation`.
/usr/local/lib/python3.7/dist-packages/transformers/tokenization_utils_base.py:2257: FutureWarning: The `pad_to_max_length` argument is deprecated and will be removed in a future version, use `padding=True` or `padding='longest'` to pad to the longest sequence in the batch, or use `padding='max_length'` to pad to a max length. In this case, you can give a specific length with `max_length` (e.g. `max_length=45`) or leave max_length to None to pad to the maximal input size of the model (e.g. 512 for Bert).
  FutureWarning,
/usr/local/lib/python3.7/dist-packages/transformers/tokenization_utils_base.py:2257: FutureWarning: The `pad_to_max_length` argument is deprecated and will be removed in a future version, use `padding=True` or `padding='longest'` to pad to the longest sequence in the batch, or use `padding='max_length'` to pad to a max length. In this case, you can give a specific length with `max_length` (e.g. `max_length=45`) or leave max_length to None to pad to the maximal input size of the model (e.g. 512 for Bert).
  FutureWarning,

---------------------------------------------------------------------------

TypeError                                 Traceback (most recent call last)

&lt;ipython-input-55-672eca036a61&gt; in &lt;module&gt;()
----&gt; 1 get_ipython().run_cell_magic('time', '', "\nhistory = defaultdict(list)\nbest_accuracy = 0\n\nfor epoch in range(EPOCHS):\n\n  print(f'Epoch {epoch + 1}/{EPOCHS}')\n  print('-' * 10)\n\n  train_acc, train_loss = train_epoch(\n    model,\n    train_data_loader,    \n    loss_fn, \n    optimizer, \n    device, \n    scheduler, \n    len(df_train)\n  )\n\n  print(f'Train loss {train_loss} accuracy {train_acc}')\n\n  val_acc, val_loss = eval_model(\n    model,\n    val_data_loader,\n    loss_fn, \n    device, \n    len(df_val)\n  )\n\n  print(f'Val   loss {val_loss} accuracy {val_acc}')\n  print()\n\n  history['train_acc'].append(train_acc)\n  history['train_loss'].append(train_loss)\n  history['val_acc'].append(val_acc)\n  history['val_loss'].append(val_loss)\n\n  if val_acc &gt; best_accuracy:\n    torch.save(model.state_dict(), 'best_model_state_ensemble_1.bin')\n    best_accuracy = val_acc")

4 frames

/usr/local/lib/python3.7/dist-packages/IPython/core/interactiveshell.py in run_cell_magic(self, magic_name, line, cell)
   2115             magic_arg_s = self.var_expand(line, stack_depth)
   2116             with self.builtin_trap:
-&gt; 2117                 result = fn(magic_arg_s, cell)
   2118             return result
   2119 

&lt;decorator-gen-53&gt; in time(self, line, cell, local_ns)

/usr/local/lib/python3.7/dist-packages/IPython/core/magic.py in &lt;lambda&gt;(f, *a, **k)
    186     # but it's overkill for just that one bit of state.
    187     def magic_deco(arg):
--&gt; 188         call = lambda f, *a, **k: f(*a, **k)
    189 
    190         if callable(arg):

/usr/local/lib/python3.7/dist-packages/IPython/core/magics/execution.py in time(self, line, cell, local_ns)
   1191         else:
   1192             st = clock2()
-&gt; 1193             exec(code, glob, local_ns)
   1194             end = clock2()
   1195             out = None

&lt;timed exec&gt; in &lt;module&gt;()

&lt;ipython-input-51-0f1c9c9d09ab&gt; in train_epoch(model, data_loader, loss_fn, optimizer, device, scheduler, n_examples)
     21     outputs = model(
     22       input_ids=input_ids,
---&gt; 23       attention_mask=attention_mask
     24     )
     25 

/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
   1100         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1101                 or _global_forward_hooks or _global_forward_pre_hooks):
-&gt; 1102             return forward_call(*input, **kwargs)
   1103         # Do not call functions when jit is used
   1104         full_backward_hooks, non_full_backward_hooks = [], []

TypeError: forward() got an unexpected keyword argument 'input_ids
</code></pre>
<p>Although i am passing the input_ids and attention masks in the forward method for each of the transformers the final ensemble model is unable to incorporate that.<br>
I called the models classes with the following code:</p>
<pre><code class="lang-auto">modelA = BERTClassA()

modelB = BERTClassB()

modelC = BERTClassC()

modelD = BERTClassD()

modelE = BERTClassE()
</code></pre>
<p>and finally the ensemble class is as follows:</p>
<pre><code class="lang-auto">model = MyEnsemble()

model.to(device)
</code></pre>
<p>Clearly i am doing some mistake somewhere.</p>
<p>Your help would be much appreciated.</p>

mrich: <p>This seems like a python error; if you can you should post you train_epoch code.</p>
<p>I think the problem is that you’ve defined input_ids to be a non keyword argument.</p>
<p>For reference:</p>
<pre><code class="lang-auto">def keyword_arg_fn(first_param=1): 
    &lt;some code&gt; 

def non_keyword_arg_fn(first_param): 
   &lt;some code&gt; 
</code></pre>
<p>keyword arguments are params to a function with a default value. See <a href="https://www.w3schools.com/python/gloss_python_function_keyword_arguments.asp" rel="noopener nofollow ugc">here</a> for more details.</p>
<p>The fix is just to remove forward input_ids in the function call.</p>
<pre><code class="lang-auto">outputs = model(input_ids,attention_mask)
</code></pre>
<p>rather than</p>
<pre><code class="lang-auto">outputs = model(input_ids=input_ids, attention_mask=attention_mask)
</code></pre>

the_coder: <p>Thank you so much for the time you have taken to debug.</p>
<p>Your correction removed the error but it gave another error. My training epoch function is as follows:</p>
<pre><code class="lang-auto">def train_epoch(
  model, 
  data_loader, 
  loss_fn, 
  optimizer, 
  device, 
  scheduler, 
  n_examples
):
  model = model.train()

  losses = []
  correct_predictions = 0


  for d in data_loader:
    input_ids = d["input_ids"].to(device)
    attention_mask = d["attention_mask"].to(device)
    targets = d["targets"].to(device)

    outputs = model(
      input_ids,
      attention_mask
    )

    _, preds = torch.max(outputs, dim=1)
    loss = loss_fn(outputs, targets)

    correct_predictions += torch.sum(preds == targets)
    losses.append(loss.item())

    loss.backward()
    nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
    optimizer.step()
    scheduler.step()
    optimizer.zero_grad()

  return correct_predictions.double() / n_examples, np.mean(losses)
</code></pre>
<p>The new error is :</p>
<pre><code class="lang-auto">
Epoch 1/4
----------

Truncation was not explicitly activated but `max_length` is provided a specific value, please use `truncation=True` to explicitly truncate examples to max length. Defaulting to 'longest_first' truncation strategy. If you encode pairs of sequences (GLUE-style) with the tokenizer you can select this strategy more precisely by providing a specific strategy to `truncation`.
/usr/local/lib/python3.7/dist-packages/transformers/tokenization_utils_base.py:2257: FutureWarning: The `pad_to_max_length` argument is deprecated and will be removed in a future version, use `padding=True` or `padding='longest'` to pad to the longest sequence in the batch, or use `padding='max_length'` to pad to a max length. In this case, you can give a specific length with `max_length` (e.g. `max_length=45`) or leave max_length to None to pad to the maximal input size of the model (e.g. 512 for Bert).
  FutureWarning,
Truncation was not explicitly activated but `max_length` is provided a specific value, please use `truncation=True` to explicitly truncate examples to max length. Defaulting to 'longest_first' truncation strategy. If you encode pairs of sequences (GLUE-style) with the tokenizer you can select this strategy more precisely by providing a specific strategy to `truncation`.
/usr/local/lib/python3.7/dist-packages/transformers/tokenization_utils_base.py:2257: FutureWarning: The `pad_to_max_length` argument is deprecated and will be removed in a future version, use `padding=True` or `padding='longest'` to pad to the longest sequence in the batch, or use `padding='max_length'` to pad to a max length. In this case, you can give a specific length with `max_length` (e.g. `max_length=45`) or leave max_length to None to pad to the maximal input size of the model (e.g. 512 for Bert).
  FutureWarning,

---------------------------------------------------------------------------

TypeError                                 Traceback (most recent call last)

&lt;ipython-input-39-672eca036a61&gt; in &lt;module&gt;()
----&gt; 1 get_ipython().run_cell_magic('time', '', "\nhistory = defaultdict(list)\nbest_accuracy = 0\n\nfor epoch in range(EPOCHS):\n\n  print(f'Epoch {epoch + 1}/{EPOCHS}')\n  print('-' * 10)\n\n  train_acc, train_loss = train_epoch(\n    model,\n    train_data_loader,    \n    loss_fn, \n    optimizer, \n    device, \n    scheduler, \n    len(df_train)\n  )\n\n  print(f'Train loss {train_loss} accuracy {train_acc}')\n\n  val_acc, val_loss = eval_model(\n    model,\n    val_data_loader,\n    loss_fn, \n    device, \n    len(df_val)\n  )\n\n  print(f'Val   loss {val_loss} accuracy {val_acc}')\n  print()\n\n  history['train_acc'].append(train_acc)\n  history['train_loss'].append(train_loss)\n  history['val_acc'].append(val_acc)\n  history['val_loss'].append(val_loss)\n\n  if val_acc &gt; best_accuracy:\n    torch.save(model.state_dict(), 'best_model_state_ensemble_1.bin')\n    best_accuracy = val_acc")

4 frames

/usr/local/lib/python3.7/dist-packages/IPython/core/interactiveshell.py in run_cell_magic(self, magic_name, line, cell)
   2115             magic_arg_s = self.var_expand(line, stack_depth)
   2116             with self.builtin_trap:
-&gt; 2117                 result = fn(magic_arg_s, cell)
   2118             return result
   2119 

&lt;decorator-gen-53&gt; in time(self, line, cell, local_ns)

/usr/local/lib/python3.7/dist-packages/IPython/core/magic.py in &lt;lambda&gt;(f, *a, **k)
    186     # but it's overkill for just that one bit of state.
    187     def magic_deco(arg):
--&gt; 188         call = lambda f, *a, **k: f(*a, **k)
    189 
    190         if callable(arg):

/usr/local/lib/python3.7/dist-packages/IPython/core/magics/execution.py in time(self, line, cell, local_ns)
   1191         else:
   1192             st = clock2()
-&gt; 1193             exec(code, glob, local_ns)
   1194             end = clock2()
   1195             out = None

&lt;timed exec&gt; in &lt;module&gt;()

&lt;ipython-input-35-af29543d4219&gt; in train_epoch(model, data_loader, loss_fn, optimizer, device, scheduler, n_examples)
     21     outputs = model(
     22       input_ids,
---&gt; 23       attention_mask
     24     )
     25 

/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
   1100         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1101                 or _global_forward_hooks or _global_forward_pre_hooks):
-&gt; 1102             return forward_call(*input, **kwargs)
   1103         # Do not call functions when jit is used
   1104         full_backward_hooks, non_full_backward_hooks = [], []

TypeError: forward() missing 3 required positional arguments: 'x3', 'x4', and 'x5'
</code></pre>
<p>It does not recognise x3, x4 and x5 from the ensemble class. My question is with the following code:</p>
<p><code>outputs = model(input_ids, attention_mask)</code><br>
is it considering input_ids as x1 and attention_mask as x2 and cannot recognise x3,x4 and x5 ?</p>
<p>I could not fully understand the following code in the MyEnsemble() class. Could you please explain if i am correct on this one?</p>
<pre><code class="lang-auto">self.classifier = torch.nn.Linear(768, 6)
</code></pre>

mrich: <aside class="quote no-group" data-username="the_coder" data-post="3" data-topic="142719">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/the_coder/48/19504_2.png" class="avatar"> the_coder:</div>
<blockquote>
<p>It does not recognise x3, x4 and x5 from the ensemble class. My question is with the following code:</p>
<p><code>outputs = model(input_ids, attention_mask)</code><br>
is it considering input_ids as x1 and attention_mask as x2 and cannot recognise x3,x4 and x5 ?</p>
</blockquote>
</aside>
<p>I believe that’s correct. According to your forward function for your ensemble, it requires 5 inputs:</p>
<aside class="quote no-group" data-username="the_coder" data-post="1" data-topic="142719">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/the_coder/48/19504_2.png" class="avatar"> the_coder:</div>
<blockquote>
<p><code>def forward(self, x1, x2,x3,x4,x5):</code></p>
</blockquote>
</aside>
<p>and you are only passing in 2.</p>
<aside class="quote no-group" data-username="the_coder" data-post="3" data-topic="142719">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/the_coder/48/19504_2.png" class="avatar"> the_coder:</div>
<blockquote>
<p>I could not fully understand the following code in the MyEnsemble() class. Could you please explain if i am correct on this one?</p>
<pre><code class="lang-auto">self.classifier = torch.nn.Linear(768, 6)
</code></pre>
</blockquote>
</aside>
<p>It is a linear layer with 768 inputs and 6 outputs. I’m assuming you are doing some text classification with 6 possible categories. It’s mapping the BERT output representation (which look like it is 768 dimensional) to 1 of 6 categories.</p>

the_coder: <p>I thought the x1, x2, x3 ,x4 and x5 s are the outputs from the BERTClassA to BERTClassE. As defined in BERTClassA:</p>
<pre><code class="lang-auto">class BERTClassA(torch.nn.Module):
  def init(self):
    super(BERTClassA, self).init()
    self.l1 = BertModel.from_pretrained('bert-base-uncased')
    self.pre_classifier = torch.nn.Linear(768, 768)
    self.dropout = torch.nn.Dropout(.3)
  
  def forward(self, input_ids, attention_mask):
    output_1 = self.l1(input_ids=input_ids, attention_mask=attention_mask)
    hidden_state = output_1[0]
    pooler = hidden_state[:, 0]
    pooler = self.pre_classifier(pooler)
    pooler = torch.nn.ReLU()(pooler)
    output = self.dropout(pooler)
    return output
</code></pre>
<p>I am asking to inherit the output from BERTClassA and give me output as x1, same goes for the next 4 BERT classes. Am i correct?</p>
<p>The model should conisder the input_ids and attention_masks as i defined them in the BERTClassA- BERTClassE as below:</p>
<pre><code class="lang-auto">def forward(self, input_ids, attention_mask):
    output_1 = self.l1(input_ids=input_ids, attention_mask=attention_mask)
</code></pre>
<p>Somehow i need to connect this to the my ensemble model.</p>
<p>I am a bit confused at this stage…</p>

samm: <p>Did you solve it please</p>