Camembert fine tuning

mima_mrb: <p>I am trying to fine tune Camembert on my data for classification. I have two classes Accepted and Rejected. I installed manually Camembert Base in my work laptop. I am having this issue. I tried to change the embedding dimensions as well as max length seq.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/8/58779d211fed94053db3558c3429d8b3f71c0eed.png" data-download-href="https://discuss.pytorch.org/uploads/default/58779d211fed94053db3558c3429d8b3f71c0eed" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/8/58779d211fed94053db3558c3429d8b3f71c0eed.png" alt="image" data-base62-sha1="cCChTYxHJVpuf7v4tZSfN6XRbjv" width="557" height="500" data-dominant-color="F1F0F0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">819×735 33.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<hr>
<p>IndexError                                Traceback (most recent call last)<br>
Input In [66], in &lt;cell line: 1&gt;()<br>
----&gt; 1 lm_learner.fit(epochs=5, <span class="hashtag-raw">#was</span> 30<br>
2             lr=6e-5,<br>
3             validate=True,<br>
4             schedule_type=“warmup_cosine”,<br>
5             optimizer_type=“adamw”)</p>
<p>File ~\Miniconda3\envs\env2clone\lib\site-packages\fast_bert-2.0.6-py3.8.egg\fast_bert\learner_lm.py:211, in BertLMLearner.fit(self, epochs, lr, validate, schedule_type, optimizer_type)<br>
209 inputs, labels = self.data.mask_tokens(batch)<br>
210 cpu_device = torch.device(“cpu”)<br>
 → 211 loss = self.training_step(batch)<br>
213 tr_loss += loss.item()<br>
214 epoch_loss += loss.item()</p>
<p>File ~\Miniconda3\envs\env2clone\lib\site-packages\fast_bert-2.0.6-py3.8.egg\fast_bert\learner_lm.py:312, in BertLMLearner.training_step(self, batch)<br>
310         outputs = self.model(inputs, labels=labels)<br>
311 else:<br>
 → 312     outputs = self.model(inputs, labels=labels)<br>
314 loss = outputs[0]<br>
316 if self.n_gpu &gt; 1:</p>
<p>File ~\Miniconda3\envs\env2clone\lib\site-packages\torch\nn\modules\module.py:1130, in Module._call_impl(self, *input, **kwargs)<br>
1126 # If we don’t have any hooks, we want to skip the rest of the logic in<br>
1127 # this function, and just call forward.<br>
1128 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks<br>
1129         or _global_forward_hooks or _global_forward_pre_hooks):<br>
 → 1130     return forward_call(*input, **kwargs)<br>
1131 # Do not call functions when jit is used<br>
1132 full_backward_hooks, non_full_backward_hooks = <span class="chcklst-box fa fa-square-o fa-fw"></span>, <span class="chcklst-box fa fa-square-o fa-fw"></span></p>
<p>File ~\Miniconda3\envs\env2clone\lib\site-packages\transformers\models\roberta\modeling_roberta.py:1098, in RobertaForMaskedLM.forward(self, input_ids, attention_mask, token_type_ids, position_ids, head_mask, inputs_embeds, encoder_hidden_states, encoder_attention_mask, labels, output_attentions, output_hidden_states, return_dict)<br>
1088 r"“”<br>
1089 labels (<code>torch.LongTensor</code> of shape <code>(batch_size, sequence_length)</code>, <em>optional</em>):<br>
1090     Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, …,<br>
(…)<br>
1094     Used to hide legacy arguments that have been deprecated.<br>
1095 “”"<br>
1096 return_dict = return_dict if return_dict is not None else self.config.use_return_dict<br>
 → 1098 outputs = self.roberta(<br>
1099     input_ids,<br>
1100     attention_mask=attention_mask,<br>
1101     token_type_ids=token_type_ids,<br>
1102     position_ids=position_ids,<br>
1103     head_mask=head_mask,<br>
1104     inputs_embeds=inputs_embeds,<br>
1105     encoder_hidden_states=encoder_hidden_states,<br>
1106     encoder_attention_mask=encoder_attention_mask,<br>
1107     output_attentions=output_attentions,<br>
1108     output_hidden_states=output_hidden_states,<br>
1109     return_dict=return_dict,<br>
1110 )<br>
1111 sequence_output = outputs[0]<br>
1112 prediction_scores = self.lm_head(sequence_output)</p>
<p>File ~\Miniconda3\envs\env2clone\lib\site-packages\torch\nn\modules\module.py:1130, in Module._call_impl(self, *input, **kwargs)<br>
1126 # If we don’t have any hooks, we want to skip the rest of the logic in<br>
1127 # this function, and just call forward.<br>
1128 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks<br>
1129         or _global_forward_hooks or _global_forward_pre_hooks):<br>
 → 1130     return forward_call(*input, **kwargs)<br>
1131 # Do not call functions when jit is used<br>
1132 full_backward_hooks, non_full_backward_hooks = <span class="chcklst-box fa fa-square-o fa-fw"></span>, <span class="chcklst-box fa fa-square-o fa-fw"></span></p>
<p>File ~\Miniconda3\envs\env2clone\lib\site-packages\transformers\models\roberta\modeling_roberta.py:844, in RobertaModel.forward(self, input_ids, attention_mask, token_type_ids, position_ids, head_mask, inputs_embeds, encoder_hidden_states, encoder_attention_mask, past_key_values, use_cache, output_attentions, output_hidden_states, return_dict)<br>
837 # Prepare head mask if needed<br>
838 # 1.0 in head_mask indicate we keep the head<br>
839 # attention_probs has shape bsz x n_heads x N x N<br>
840 # input head_mask has shape [num_heads] or [num_hidden_layers x num_heads]<br>
841 # and head_mask is converted to shape [num_hidden_layers x batch x num_heads x seq_length x seq_length]<br>
842 head_mask = self.get_head_mask(head_mask, self.config.num_hidden_layers)<br>
 → 844 embedding_output = self.embeddings(<br>
845     input_ids=input_ids,<br>
846     position_ids=position_ids,<br>
847     token_type_ids=token_type_ids,<br>
848     inputs_embeds=inputs_embeds,<br>
849     past_key_values_length=past_key_values_length,<br>
850 )<br>
851 encoder_outputs = self.encoder(<br>
852     embedding_output,<br>
853     attention_mask=extended_attention_mask,<br>
(…)<br>
861     return_dict=return_dict,<br>
862 )<br>
863 sequence_output = encoder_outputs[0]</p>
<p>File ~\Miniconda3\envs\env2clone\lib\site-packages\torch\nn\modules\module.py:1130, in Module._call_impl(self, *input, **kwargs)<br>
1126 # If we don’t have any hooks, we want to skip the rest of the logic in<br>
1127 # this function, and just call forward.<br>
1128 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks<br>
1129         or _global_forward_hooks or _global_forward_pre_hooks):<br>
 → 1130     return forward_call(*input, **kwargs)<br>
1131 # Do not call functions when jit is used<br>
1132 full_backward_hooks, non_full_backward_hooks = <span class="chcklst-box fa fa-square-o fa-fw"></span>, <span class="chcklst-box fa fa-square-o fa-fw"></span></p>
<p>File ~\Miniconda3\envs\env2clone\lib\site-packages\transformers\models\roberta\modeling_roberta.py:132, in RobertaEmbeddings.forward(self, input_ids, token_type_ids, position_ids, inputs_embeds, past_key_values_length)<br>
129         token_type_ids = torch.zeros(input_shape, dtype=torch.long, device=self.position_ids.device)<br>
131 if inputs_embeds is None:<br>
 → 132     inputs_embeds = self.word_embeddings(input_ids)<br>
133 token_type_embeddings = self.token_type_embeddings(token_type_ids)<br>
135 embeddings = inputs_embeds + token_type_embeddings</p>
<p>File ~\Miniconda3\envs\env2clone\lib\site-packages\torch\nn\modules\module.py:1130, in Module._call_impl(self, *input, **kwargs)<br>
1126 # If we don’t have any hooks, we want to skip the rest of the logic in<br>
1127 # this function, and just call forward.<br>
1128 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks<br>
1129         or _global_forward_hooks or _global_forward_pre_hooks):<br>
 → 1130     return forward_call(*input, **kwargs)<br>
1131 # Do not call functions when jit is used<br>
1132 full_backward_hooks, non_full_backward_hooks = <span class="chcklst-box fa fa-square-o fa-fw"></span>, <span class="chcklst-box fa fa-square-o fa-fw"></span></p>
<p>File ~\Miniconda3\envs\env2clone\lib\site-packages\torch\nn\modules\sparse.py:158, in Embedding.forward(self, input)<br>
157 def forward(self, input: Tensor) → Tensor:<br>
 → 158     return F.embedding(<br>
159         input, self.weight, self.padding_idx, self.max_norm,<br>
160         self.norm_type, self.scale_grad_by_freq, self.sparse)</p>
<p>File ~\Miniconda3\envs\env2clone\lib\site-packages\torch\nn\functional.py:2199, in embedding(input, weight, padding_idx, max_norm, norm_type, scale_grad_by_freq, sparse)<br>
2193     # Note [embedding_renorm set_grad_enabled]<br>
2194     # XXX: equivalent to<br>
2195     # with torch.no_grad():<br>
2196     #   torch.embedding_renorm_<br>
2197     # remove once script supports set_grad_enabled<br>
2198     <em>no_grad_embedding_renorm</em>(weight, input, max_norm, norm_type)<br>
 → 2199 return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)</p>
<p>IndexError: index out of range in self</p>

ptrblck: <p>Your post doesn’t describe any issue or asks any questions, so would you mind updating it?</p>

mima_mrb: <p>Hello, thank you. I’ve changed the post. Can you check it please ! thanks</p>

ptrblck: <p>Your input to the <code>nn.Embedding</code> layer is out-of-bounds as described in e.g. <a href="https://discuss.pytorch.org/t/embeddings-index-out-of-range-error/12582/27">this post</a>.</p>

mima_mrb: <p>thanks for your answer. But I am doing this I choosed for max sequence length a dimension less than 512 which is the dimension in the camembert architecture. But I still have the error.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/9/1911a8517c16b80900f10b1320dd13620071c487.png" data-download-href="https://discuss.pytorch.org/uploads/default/1911a8517c16b80900f10b1320dd13620071c487" title="error1"><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/9/1911a8517c16b80900f10b1320dd13620071c487.png" alt="error1" data-base62-sha1="3zLLBicoCzjcaajKr6BqD9XiePR" width="690" height="451" data-dominant-color="F1F1EE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">error1</span><span class="informations">883×578 30.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>I don’t know how the sequence length is used inside your model but it’s usually not defining the index range.</p>