RuntimeError: The expanded size of the tensor (449) must match the existing size (2) at non-singleton dimension 2.  Target sizes: [4, 2, 449].  Tensor sizes: [1, 2] while using ALBERT
Author: hasnain2808
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
I wanted to use ALBERT with a double head 
as we have one  for openaigpt with the name OpenAIGPTDoubleHeadsModel
I am doing it with taking inspiration from OpenAIGPTDoubleHeadsModel
but I am getting this error

` File "train.py", line 266, in <module>
    train()
  File "train.py", line 258, in train
    trainer.run(train_loader, max_epochs=args.n_epochs)
  File "/home/moha/venv/huggtrans/lib/python3.6/site-packages/ignite/engine/engine.py", line 446, in run
    self._handle_exception(e)
  File "/home/moha/venv/huggtrans/lib/python3.6/site-packages/ignite/engine/engine.py", line 410, in _handle_exception
    raise e
  File "/home/moha/venv/huggtrans/lib/python3.6/site-packages/ignite/engine/engine.py", line 433, in run
    hours, mins, secs = self._run_once_on_dataset()
  File "/home/moha/venv/huggtrans/lib/python3.6/site-packages/ignite/engine/engine.py", line 399, in _run_once_on_dataset
    self._handle_exception(e)
  File "/home/moha/venv/huggtrans/lib/python3.6/site-packages/ignite/engine/engine.py", line 410, in _handle_exception
    raise e
  File "/home/moha/venv/huggtrans/lib/python3.6/site-packages/ignite/engine/engine.py", line 391, in _run_once_on_dataset
    self.state.output = self._process_function(self, batch)
  File "train.py", line 180, in update
    mc_labels=mc_labels, lm_labels=lm_labels
  File "/home/moha/venv/huggtrans/lib/python3.6/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/moha/venv/huggtrans/lib/python3.6/site-packages/transformers/modeling_albert.py", line 956, in forward
    inputs_embeds=inputs_embeds)
  File "/home/moha/venv/huggtrans/lib/python3.6/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/moha/venv/huggtrans/lib/python3.6/site-packages/transformers/modeling_albert.py", line 499, in forward
    inputs_embeds=inputs_embeds)
  File "/home/moha/venv/huggtrans/lib/python3.6/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/moha/venv/huggtrans/lib/python3.6/site-packages/transformers/modeling_bert.py", line 171, in forward
    position_ids = position_ids.unsqueeze(0).expand(input_shape)
RuntimeError: The expanded size of the tensor (449) must match the existing size (2) at non-singleton dimension 2.  Target sizes: [4, 2, 449].  Tensor sizes: [1, 2]`


the code block for AlbertDoubleHeadsModel 


`class AlbertDoubleHeadsModel(AlbertPreTrainedModel):`
`    def __init__(self, config):`
`        super(AlbertDoubleHeadsModel, self).__init__(config)`

        self.albert = AlbertModel(config)
        self.lm_head = nn.Linear(config.embedding_size, config.vocab_size, bias=False)
        self.multiple_choice_head = SequenceSummary(config)

        self.init_weights()

    def get_output_embeddings(self):
        return self.lm_head

    def forward(self, input_ids=None, attention_mask=None, token_type_ids=None, position_ids=None, head_mask=None, inputs_embeds=None,
                mc_token_ids=None, lm_labels=None, mc_labels=None):
        transformer_outputs = self.albert(input_ids,
                                               attention_mask=attention_mask,
                                               token_type_ids=token_type_ids,
                                               position_ids=position_ids,
                                               head_mask=head_mask,
                                               inputs_embeds=inputs_embeds)
        hidden_states = transformer_outputs[0]

        lm_logits = self.lm_head(hidden_states)
        mc_logits = self.multiple_choice_head(hidden_states, mc_token_ids).squeeze(-1)

        outputs = (lm_logits, mc_logits) + transformer_outputs[1:]
        if mc_labels is not None:
            loss_fct = CrossEntropyLoss()
            loss = loss_fct(mc_logits.view(-1, mc_logits.size(-1)),
                            mc_labels.view(-1))
            outputs = (loss,) + outputs
        if lm_labels is not None:
            shift_logits = lm_logits[..., :-1, :].contiguous()
            shift_labels = lm_labels[..., 1:].contiguous()
            loss_fct = CrossEntropyLoss(ignore_index=-1)
            loss = loss_fct(shift_logits.view(-1, shift_logits.size(-1)),
                            shift_labels.view(-1))
            outputs = (loss,) + outputs

        return outputs  # (lm loss), (mc loss), lm logits, mc logits, (all hidden_states), (attentions)`



Is there anything that I am missing?
Please do tell me.


Comments:
What is the code that you are executing that leads to this error? 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


How did you fix the problem?