Expected object of device type cuda but got device type cpu for argument #1 'self' in call to _th_mm

Kushagra_Bhatia: <p>This is my model. Only showing the __ init __() and forward() functions</p>
<pre><code class="lang-auto">class BERTplusAoA(nn.Module):
    def __init__(self, config, options): 
        super(BERTplusAoA, self).__init__()
        self.bert = BertModel.from_pretrained(
        options.model_name_or_path,
        from_tf=bool(".ckpt" in options.model_name_or_path),
        config=config,
        cache_dir=options.cache_dir if options.cache_dir else None,
        ) 
        self.l0 = nn.Linear(1024, 2)
        self.lq = nn.Linear(1024,256)
        self.lc = nn.Linear(1024,256)
        self.soft=nn.Softmax(dim=-1)
        self.wb = torch.sigmoid(nn.Parameter(torch.zeros(1)))
        self.we = torch.sigmoid(nn.Parameter(torch.zeros(1)))

def forward(self, input_ids, attention_mask, token_type_ids,start_positions,end_positions):
        sequence_out, _ = self.bert(input_ids, attention_mask=attention_mask, token_type_ids=token_type_ids)
        # (batch_size,num_tokens,1024)
        logits = self.l0(sequence_out)
        # (batch_size,num_tokens,2)
        start_logits,end_logits=logits.split(1,dim=-1)
        # (batch_size,num_tokens,1)        # (batch_size,num_tokens,1)
        start_logits=start_logits.squeeze(-1)
        end_logits=end_logits.squeeze(-1)
        # (batch_size,num_tokens)        # (batch_size,num_tokens)
        
        lb = F.softmax(start_logits,dim=-1)
        le = F.softmax(end_logits,dim=-1)

        hq,hc=self.splitting_(sequence_out,token_type_ids)
        #return hq,hc,sequence_out
        # hq, hc are list of tensors.

        hq,hq_len=self._batchify(hq,include_lengths=True)
        hc,hc_len=self._batchify(hc,include_lengths=True)
        # hq, hc have are tensors of dim: # (batch_size, max_length q or c, 1024)

        Hq = self.lq(hq)
        Hq_b,Hq_e = Hq.split(128,dim=-1)

        Hc = self.lc(hc)
        Hc_b,Hc_e = Hc.split(128,dim=-1)
..
</code></pre>
<p>I get the following error:</p>
<pre><code class="lang-auto">Epoch:   0% 0/20 [00:00&lt;?, ?it/s]
Iteration:   0% 0/6464 [00:00&lt;?, ?it/s]Traceback (most recent call last):
  File "run_techqa.py", line 623, in &lt;module&gt;
    main()
  File "run_techqa.py", line 617, in main
    model = train(args, train_dataset, model, optimizer, tokenizer, model_evaluator)
  File "run_techqa.py", line 223, in train
    outputs = model(**inputs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/apex/amp/_initialize.py", line 197, in new_fwd
    **applier(kwargs, input_caster))
  File "/content/MyDrive/IBM/TechQA-Base/techqa-master/model_techqa.py", line 109, in forward
    Hq = self.lq(hq)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/linear.py", line 87, in forward
    return F.linear(input, self.weight, self.bias)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py", line 1372, in linear
    output = input.matmul(weight.t())
RuntimeError: Expected object of device type cuda but got device type cpu for argument #1 'self' in call to _th_mm

</code></pre>

harsha_g: <p>Did you try sending <code>input</code> to the GPU?</p>

SANTOSH_S: <p>Hi <a class="mention" href="/u/kushagra_bhatia">@Kushagra_Bhatia</a>,<br>
The issue is with how you pass your inputs to the model during training. Can you post part of you code where you are training?</p>

Kushagra_Bhatia: <p>Hello, <a class="mention" href="/u/harsha_g">@harsha_g</a> and <a class="mention" href="/u/santosh_s">@SANTOSH_S</a>. I believe the dataset is on CUDA.<br>
Training loop is shown below. Also, args.device=device(type=‘cuda’)</p>
<pre><code class="lang-auto">   for _ in train_iterator:
        # train_iterator is basically range(0, total_epochs). However in case we resume training then
        # train_iterator becomes range(completed_epochs, total_epochs)
        # In every epoch we compute the mini-batches again
        train_sampler = RandomSampler(train_dataset) if args.local_rank == -1 else DistributedSampler(train_dataset)
        train_dataloader = DataLoader(train_dataset, sampler=train_sampler, batch_size=args.train_batch_size)
        epoch_iterator = tqdm(train_dataloader, desc="Iteration", disable=args.local_rank not in [-1, 0])
         
        for step, batch in enumerate(epoch_iterator):
            # batch is an element in dataloader. i.e. effectively a mini-batch

            # Skip past any already trained steps if resuming training. i.e. number of mini-batches trained already in the epoch
            if steps_trained_in_current_epoch &gt; 0:
                steps_trained_in_current_epoch -= 1
                continue

            model.train()
            batch = tuple(t.to(args.device) for t in batch)

            inputs = {
                "input_ids": batch[0],
                "attention_mask": batch[1],
                "token_type_ids": batch[2],
                "start_positions": batch[3],
                "end_positions": batch[4],
            }

            if args.model_type in ["roberta"]:
                del inputs["token_type_ids"] 
            # roberta doesn't require token_type_ids           

            outputs = model(**inputs)
            # model outputs are always tuple in transformers (see doc)
</code></pre>

harsha_g: <p>Then, I am assuming you also put your model on the GPU. Didn’t you? <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

Kushagra_Bhatia: <p>Yes I did put it on GPU.</p>
<pre><code class="lang-auto">def load_model(args, model_class, config):

    model = BERTplusAoA(config, args)
    

    if args.local_rank == 0:
        # Make sure only the first process in distributed training will download model &amp; vocab
        torch.distributed.barrier()

    model.to(args.device)
...
</code></pre>

harsha_g: <aside class="quote no-group" data-username="Kushagra_Bhatia" data-post="1" data-topic="86921">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kushagra_bhatia/48/25826_2.png" class="avatar"> Kushagra_Bhatia:</div>
<blockquote>
<p><code>BertModel</code></p>
</blockquote>
</aside>
<p>I am not entirely sure but could it be that you have another model <code>BertModel</code> inside your actual model, and does that need to be put on the device?</p>

Kushagra_Bhatia: <p>I imported it from transformers library.</p>
<pre><code class="lang-auto">from transformers import BertModel
</code></pre>

harsha_g: <p>I get that. I am just speculating if the following calls will help. I am not entirely sure. I’d just try those.</p>
<p><code>BertModel = BertModel.to(device)</code></p>
<p>or</p>
<p><code>self.bert = BertModel.to(device).from_pretrained(         options.model_name_or_path,         from_tf=bool(".ckpt" in options.model_name_or_path),         config=config,         cache_dir=options.cache_dir if options.cache_dir else None,         )</code></p>

Kushagra_Bhatia: <p>No, it doesn’t resolve the error. Rather throws another one:</p>
<pre><code class="lang-auto">  BertModel.to(options.device)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 425, in to
    return self._apply(convert)
AttributeError: 'torch.device' object has no attribute '_apply'
</code></pre>
<p>I don’t think we are supposed to pass the BertModel to the device. For example, the following code snippet is from huggingface’s official implementation of BertForQuestionAnswering model.</p>
<pre><code class="lang-auto">class BertForQuestionAnswering(BertPreTrainedModel):
    def __init__(self, config):
        super().__init__(config)
        self.num_labels = config.num_labels

        self.bert = BertModel(config)
        self.qa_outputs = nn.Linear(config.hidden_size, config.num_labels)

        self.init_weights()
</code></pre>

harsha_g: <p>Got it. Thanks for trying though. <img src="https://discuss.pytorch.org/images/emoji/apple/+1.png?v=9" title=":+1:" class="emoji" alt=":+1:"></p>

harsha_g: <aside class="quote no-group" data-username="Kushagra_Bhatia" data-post="1" data-topic="86921">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kushagra_bhatia/48/25826_2.png" class="avatar"> Kushagra_Bhatia:</div>
<blockquote>
<p><code>Hq = self.lq(hq)</code></p>
</blockquote>
</aside>
<p><a class="mention" href="/u/kushagra_bhatia">@Kushagra_Bhatia</a> Another keen look at the traceback and another thought. It’s evident the issue is at the <img src="https://discuss.pytorch.org/images/emoji/apple/point_up.png?v=12" title=":point_up:" class="emoji" alt=":point_up:" loading="lazy" width="20" height="20"> line. How about sending the <code>device</code> argument to the model and then putting <code>hq</code> and later <code>hc</code> on the device? I guess I was little sloppy earlier. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

Kushagra_Bhatia: <p>That actually solved my error. However, I am getting a new error now, which I am not able to resolve. It is as follows:</p>
<pre><code class="lang-auto">  File "run_techqa.py", line 623, in &lt;module&gt;
    main()
  File "run_techqa.py", line 617, in main
    model = train(args, train_dataset, model, optimizer, tokenizer, model_evaluator)
  File "run_techqa.py", line 223, in train
    outputs = model(**inputs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/apex/amp/_initialize.py", line 197, in new_fwd
    **applier(kwargs, input_caster))
  File "/content/MyDrive/IBM/TechQA-Base/techqa-master/model_techqa.py", line 113, in forward
    Hq = self.lq(hq)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/linear.py", line 87, in forward
    return F.linear(input, self.weight, self.bias)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py", line 1372, in linear
    output = input.matmul(weight.t())
RuntimeError: Expected object of scalar type Float but got scalar type Half for argument #2 'mat2' in call to _th_mm
</code></pre>
<p>I had used two functions in the forward part which I have not specified previously, these are:</p>
<pre><code class="lang-auto">   def splitting_(self,ids,type_ids):
      ans=[]
      for i,lis in enumerate(ids):
        typei=type_ids[i]
        ends=[]
        for j in range(len(typei)-1):
          if typei[j]!=typei[j+1]:
            ends.append(j+1)
        if len(ends)==1:
          qend=ends[0]
          dend=len(typei)
        else:
          qend,dend=ends[0],ends[1]
        qtns=lis[:qend]
        dtns=lis[qend:dend]
        temp=[qtns, dtns]
        ans.append(temp)
      q=[i[0] for i in ans]
      d=[i[1] for i in ans]
      return q,d

    # will take in list of n tensors and output a tensor of dimension n.
    # the list will contain tensors of sentence encoded using tokenizer.
    def _batchify(self,data, align_right=False, include_lengths=False):
      lengths = [x.size(0) for x in data]
      max_length = max(lengths)
      tens=torch.rand(1024,dtype=torch.float32)
      out=torch.stack([torch.stack([tens for _ in range(max_length)]) for _ in range(len(data))])
      for i in range(len(data)):
        data_length = data[i].size(0)
        offset = max_length - data_length if align_right else 0
        out[i].narrow(0, offset, data_length).copy_(data[i])
      if include_lengths:
          return out, lengths
      else:
          return out
</code></pre>
<p>Would really appreciate your help!</p>
<p>Note:<br>
I have already tried doing <code>hq=hq.float() and hq=torch.tensor(hq,dtype=torch.float32)</code></p>

harsha_g: <p><a class="mention" href="/u/kushagra_bhatia">@Kushagra_Bhatia</a> Glad, it worked. Now, onto the next one. Putting the two errors in context, it can be seen something’s off with the weights (<code>self.wb</code> and <code>self.we</code>) now and not <code>hq</code>. Try changing the types of the weights. I am sure that will take care of it. Let us know how it goes.</p>

Kushagra_Bhatia: <p>I don’t exactly understand, the traceback doesn’t explicitly mention <code>self.wb</code> and <code>self.we</code>. How do you infer about the two weights? They are defined as<code> self.wb = torch.sigmoid(nn.Parameter(torch.zeros(1)))</code>. Actually I wanted weights to combine two vectors and I have used them like: <code>ans_beg = torch.mul(self.wb,lb)+torch.mul(1-self.wb,s1)</code>. Basically it is just a weight between 0 and 1 that I wanted to define that can be learned by the model so that the two vectors could be effectively averaged. What do you think I need to change in the definition of wb and we?</p>

harsha_g: <p>Lecture time <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<pre><code class="lang-auto">File "/content/MyDrive/IBM/TechQA-Base/techqa-master/model_techqa.py", line 113, in forward
   Hq = self.lq(hq)
 File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 532, in __call__
   result = self.forward(*input, **kwargs)
 File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/linear.py", line 87, in forward
   return F.linear(input, self.weight, self.bias)
 File "/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py", line 1372, in linear
   output = input.matmul(weight.t())
RuntimeError: Expected object of scalar type Float but got scalar type Half for argument #2 'mat2' in call to _th_mm
</code></pre>
<p>If you carefully see the traceback, you will realize that the error happened at line <code>Hq = self.lq(hq)</code>. What’s happening there? You are making a call to the <code>forward</code> (the next line) of that model <code>lq</code>. What does it have? An <code>F.Linear</code> layer. And what happens in that layer? <code>input.matmul(weight.t())</code>. So, if you traceback, the <code>input</code> to the model <code>lq</code> is <code>hq</code> and the weight is? You guessed it right. <code>torch.sigmoid(nn.Parameter(torch.zeros(1)))</code>.</p>
<p>Now, the first error you posted said <code>RuntimeError: Expected object of device type cuda but got device type cpu for argument #1 'self' in call to _th_mm</code>. This means there’s an <code>mm</code> (matrix multiplication), happening on the device, which takes two inputs, the first one being <code>input</code> and the second one being <code>weight.t()</code>. So, the earlier error was solved by sending <code>input</code> aka ‘hq’ to the device. Likewise, now the second error will be taken care of by setting the datatype of <code>weight.t()</code> aka <code>self.wb</code>. How? By specifying <code>.float()</code> or <code>.double()</code> when you call <code>torch.zeros()</code>.</p>
<p>I know I wasn’t extremely clear but you get the point. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

Kushagra_Bhatia: <p>The <code>F.linear</code> comes from the <code>nn.Linear</code> from the <code>torch.nn</code> module. Inside the <code>nn.Linear</code> layer the code is as follows (official documentation from <a href="https://pytorch.org/docs/master/_modules/torch/nn/modules/linear.html#Linear" rel="nofollow noopener">https://pytorch.org/docs/master/_modules/torch/nn/modules/linear.html#Linear</a>):</p>
<pre><code class="lang-auto">class Linear(Module):

    __constants__ = ['in_features', 'out_features']
    in_features: int
    out_features: int
    weight: Tensor

    def __init__(self, in_features: int, out_features: int, bias: bool = True) -&gt; None:
        super(Linear, self).__init__()
        self.in_features = in_features
        self.out_features = out_features
        self.weight = Parameter(torch.Tensor(out_features, in_features))
        if bias:
            self.bias = Parameter(torch.Tensor(out_features))
        else:
            self.register_parameter('bias', None)
        self.reset_parameters()

    def reset_parameters(self) -&gt; None:
        init.kaiming_uniform_(self.weight, a=math.sqrt(5))
        if self.bias is not None:
            fan_in, _ = init._calculate_fan_in_and_fan_out(self.weight)
            bound = 1 / math.sqrt(fan_in)
            init.uniform_(self.bias, -bound, bound)

    def forward(self, input: Tensor) -&gt; Tensor:
        return F.linear(input, self.weight, self.bias)
</code></pre>
<p>As you can see the line</p>
<pre><code class="lang-auto">self.weight = Parameter(torch.Tensor(out_features, in_features)) defines the weight 
</code></pre>
<p>defines the weight that is passed to <code>F.linear</code> which performs the <code>output = input.matmul(weight.t())</code> operation.</p>
<p>Please correct me if I am wrong.</p>

harsha_g: <aside class="quote no-group" data-username="Kushagra_Bhatia" data-post="17" data-topic="86921">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kushagra_bhatia/48/25826_2.png" class="avatar"> Kushagra_Bhatia:</div>
<blockquote>
<p>Please correct me if I am wrong.</p>
</blockquote>
</aside>
<p>Right on the money.</p>
<p>So, did you manage to solve your latest error?</p>

harsha_g: <p>Ah, I see. I got that part wrong. I misread <code>Sigmoid</code> for <code>Linear</code>. <img src="https://discuss.pytorch.org/images/emoji/apple/man_facepalming.png?v=12" title=":man_facepalming:" class="emoji" alt=":man_facepalming:" loading="lazy" width="20" height="20"> But you got it right.</p>
<aside class="quote no-group" data-username="Kushagra_Bhatia" data-post="13" data-topic="86921">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kushagra_bhatia/48/25826_2.png" class="avatar"> Kushagra_Bhatia:</div>
<blockquote>
<p>I have already tried doing <code>hq=hq.float() and hq=torch.tensor(hq,dtype=torch.float32)</code></p>
</blockquote>
</aside>
<p>Did you try <code>double</code> or <code>float64</code> instead?</p>

Kushagra_Bhatia: <p>if I do <code>hq=hq.double() </code>or<code> hq=torch.tensor(hq,dtype=torch.float64)</code> before moving hq to the device I get the error:</p>
<pre><code class="lang-auto">RuntimeError: Expected object of scalar type Float but got scalar type Half for argument #2 'mat2' in call to _th_mm
</code></pre>
<p>If I first move to device then do <code> hq=torch.tensor(hq,dtype=torch.float64)</code>  I get:</p>
<pre><code class="lang-auto">Expected object of device type cuda but got device type cpu for argument #1 'self' in call to _th_mm
</code></pre>