Input, output must be on the current device

username: Gaborandi
cooked: <p>iam new to NLP and currently iam doing this code below wehre iam trying to save the Last Hidden States of the DistilBert model , the data is huge so iam using GPU<br>
but i don’t know why i face this problem ==&gt; Input, output and indices must be on the current device</p>
<p>my code ==&gt;</p>
<ul>
<li>
<p>tokenized = data[‘txt’].apply((lambda x: tokenizer.encode(x, add_special_tokens=True,truncation = True, max_length = 512)))</p>
</li>
<li>
<p>max_len = 0<br>
for i in tokenized.values:<br>
if len(i) &gt; max_len:<br>
max_len = len(i)</p>
</li>
</ul>
<p>padded = np.array([i + [0]*(max_len-len(i)) for i in tokenized.values])</p>
<ul>
<li>
<p>np.array(padded).shape</p>
</li>
<li>
<p>attention_mask = np.where(padded != 0, 1, 0)</p>
</li>
<li>
<p>input_ids = torch.tensor(padded).to(device)</p>
</li>
<li>
<p>attention_mask = torch.tensor(attention_mask).to(device)</p>
</li>
<li>
<p>with torch.no_grad():<br>
last_hidden_states = model(input_ids, attention_mask=attention_mask).to(device)</p>
</li>
</ul>
<p>= ---------------------------------------------------------------------------<br>
RuntimeError                              Traceback (most recent call last)<br>
 in <br>
1 with torch.no_grad():<br>
----&gt; 2     last_hidden_states = model(input_ids, attention_mask=attention_mask).to(device)</p>
<p>C:\Users\jod204.conda\envs\gpu_pyt\lib\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)<br>
887             result = self._slow_forward(*input, **kwargs)<br>
888         else:<br>
 → 889             result = self.forward(*input, **kwargs)<br>
890         for hook in itertools.chain(<br>
891                 _global_forward_hooks.values(),</p>
<p>C:\Users\jod204.conda\envs\gpu_pyt\lib\site-packages\transformers\models\distilbert\modeling_distilbert.py in forward(self, input_ids, attention_mask, head_mask, inputs_embeds, output_attentions, output_hidden_states, return_dict)<br>
478<br>
479         if inputs_embeds is None:<br>
 → 480             inputs_embeds = self.embeddings(input_ids)  # (bs, seq_length, dim)<br>
481         return self.transformer(<br>
482             x=inputs_embeds,</p>
<p>C:\Users\jod204.conda\envs\gpu_pyt\lib\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)<br>
887             result = self._slow_forward(*input, **kwargs)<br>
888         else:<br>
 → 889             result = self.forward(*input, **kwargs)<br>
890         for hook in itertools.chain(<br>
891                 _global_forward_hooks.values(),</p>
<p>C:\Users\jod204.conda\envs\gpu_pyt\lib\site-packages\transformers\models\distilbert\modeling_distilbert.py in forward(self, input_ids)<br>
105         position_ids = position_ids.unsqueeze(0).expand_as(input_ids)  # (bs, max_seq_length)<br>
106<br>
 → 107         word_embeddings = self.word_embeddings(input_ids)  # (bs, max_seq_length, dim)<br>
108         position_embeddings = self.position_embeddings(position_ids)  # (bs, max_seq_length, dim)<br>
109</p>
<p>C:\Users\jod204.conda\envs\gpu_pyt\lib\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)<br>
887             result = self._slow_forward(*input, **kwargs)<br>
888         else:<br>
 → 889             result = self.forward(*input, **kwargs)<br>
890         for hook in itertools.chain(<br>
891                 _global_forward_hooks.values(),</p>
<p>C:\Users\jod204.conda\envs\gpu_pyt\lib\site-packages\torch\nn\modules\sparse.py in forward(self, input)<br>
156         return F.embedding(<br>
157             input, self.weight, self.padding_idx, self.max_norm,<br>
 → 158             self.norm_type, self.scale_grad_by_freq, self.sparse)<br>
159<br>
160     def extra_repr(self) → str:</p>
<p>C:\Users\jod204.conda\envs\gpu_pyt\lib\site-packages\torch\nn\functional.py in embedding(input, weight, padding_idx, max_norm, norm_type, scale_grad_by_freq, sparse)<br>
1914         # remove once script supports set_grad_enabled<br>
1915         <em>no_grad_embedding_renorm</em>(weight, input, max_norm, norm_type)<br>
 → 1916     return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)<br>
1917<br>
1918</p>
<p>RuntimeError: Input, output and indices must be on the current device</p>
<p>– in the last line , i tried to run it with and without [to(device)] but same error each time<br>
how i can solve this problem</p>

username: AbdulsalamBande
cooked: <p><a class="mention" href="/u/gaborandi">@Gaborandi</a> , I assume what possible happened is that you are have loaded some tensors on GPU while others are on CPU. So if you made takes input1 and input2. Make sure both are on GPU and the model itself in on GPU. You can achieve it with the code below</p>
<pre><code class="lang-auto">device = "cuda:0"
model = model.to(device)
input1 = input1.to(device)
input2 = input2.to(device)
#Then you can say
output = model(input1, input2)
</code></pre>