Pytorch doesn't register model parameters to cuda backend

Shiva_Kumar: <p>I am trying to generate text using a sequence to sequence model in Pytorch from the Apocalypse Now movie script. My code is almost verbatim to <a href="https://github.com/LeanManager/NLP-PyTorch/blob/master/Character-Level%20LSTM%20with%20PyTorch.ipynb" rel="nofollow noopener">this one</a>.</p>
<p>The following is the model, a character level LSTM.</p>
<pre><code>class CharLSTM(nn.Module):

        def __init__(self, chars, hidden_size, n_layers=2, drop_out=0.5, lr= 0.001):

        super(CharLSTM, self).__init__()

       # Set all the hyperparameters of your network

        self.hidden_size = hidden_size
        self.n_layers = n_layers
        self.drop_out = drop_out
        self.lr = lr

        # set vocabulary and get indices for these
        self.chars = chars
        self.int2char = dict(enumerate(self.chars))
        self.char2int = {w : i for i, w in self.int2char.items()}

    # define the lstm network, this outputs the next char and cell state and hidden state

       self.lstm = nn.LSTM(input_size=len(self.chars), hidden_size=hidden_size, num_layers=n_layers, dropout=drop_out,
                   batch_first=True)
    # add dropout
       self.drop_out = nn.Dropout(drop_out)

       self.fc = nn.Linear(hidden_size, len(chars))

    self.init_weights()

     def forward(self, x, h_0):
        """compute current and hidden units, stack outputs and pass it to linear layer"""

         x, (h, c) = self.lstm(x, h_0)

         x = self.drop_out(x)
         x = x.view(x.size()[0]*x.size()[1], self.hidden_size)

          x = self.fc(x)

         return x, (h, c)

      def predict(self, char, h=None, cuda=False, top_k=None):
         """given a character, predict the next character in the sequence."""
    
         if cuda():
           self.cuda()
         else:
           self.cpu()
        
        # Initialize hidden state
        if h is None:
            h = self.init_hidden(1)

         # get the integer of character.
          ch = np.array([[self.char2int[char]]])

           # one_hot_encode
          one_hot = one_hot_encode(ch, len(self.chars))

           # convet to tensor
           one_hot_torch = torch.from_numpy(one_hot)
    
          if cuda():
             one_hot_torch = one_hot_torch.cuda()

          # create a tuple of the hidden state
          # this is what LSTM expects
         h = tuple([each.data for each in h])
         out, h = self.forward(ch, h)

         # Prob distribution over all the characters
         probs = F.softmax(out, dim=1).data

         # convert back prob to cpu if model was
         # set to gpu
    
          if cuda():
             probs = probs.cpu()

         # if top number of preds to get
         # wasn't pass, take the distribution over whole character length
         if top_k is None:
            top_ch = np.arange(len(self.chars))
         else:
            probs, top_ch = probs.topk(top_k)
           top_ch = top_ch.numpy().squeeze()

           # reduce dims of size 1
            probs = probs.numpy().squeeze()

           # sample from top_char with each probs of each character being
          # the char with higher prob will be chosen since dividing the
          # highest value with probs.sum() is what is the best.
          char = np.random.choice(top_ch, p=probs/probs.sum())

         return self.char2int[char], h

      def init_weights(self):

           initrange = 0.1

          self.fc.bias.data.fill_(0)
          self.fc.weight.data.uniform_(-1, 1)

       def init_hidden(self, n_seqs):

            weight = next(self.parameters()).data
           return (weight.new(self.n_layers, n_seqs, self.hidden_size).zero_(),
             weight.new(self.n_layers, n_seqs, self.hidden_size).zero_())
</code></pre>
<h1>instantiating an object of this class</h1>
<pre><code>    model = CharLSTM(chars, hidden_size=512, n_layers=2)
    model
</code></pre>
<p>CharLSTM(<br>
(lstm): LSTM(58, 512, num_layers=2, batch_first=True, dropout=0.5)<br>
(drop_out): Dropout(p=0.5)<br>
(fc): Linear(in_features=512, out_features=58, bias=True)<br>
)</p>
<p>The following is my train function where I send my inputs, targets for train and validation datasets, as well as my model.</p>
<pre><code>    def train(model, data, epochs=10, n_seqs=10, n_steps=40, lr=0.001, clip=5, val_frac= 0.1, cuda=False,
      print_every=10):

""" model: the model to b trained
    data: the data on which we train
    epochs: number of epochs to train for
    n_seqs" number of sequences in our batch
    n_steps: time step for each sequence
    lr: learning rate
    clip: value used to clip the network gradient to prevent exploding gradeint.
    val_frac: the fraction of data used for validation
    print_every: the number of seconds for which we print out model statistics
"""

    # change model to train mode
    **model.train()**

     # define optimizer and loss function
     optimizer = torch.optim.Adam(model.parameters(), lr=lr)
     criterion = nn.CrossEntropyLoss()


      # trin and validation split
      val_idx = int(len(data)*(1-val_frac))
      data, val_data = data[:val_idx], data[val_idx:]

      **if cuda:**
          ** model.cuda()**

        counter = 0
       n_chars = len(model.chars)

        # loop over epochs
            for epoch in range(epochs):

              # initialize hidden layer of the model
             h = model.init_hidden(n_seqs)

    # loop over batches
    for x, y in get_batches(data, n_seqs, n_steps):

        counter += 1

        # one hot encode
        x = one_hot_encode(x, n_chars)

        # convert to tensors
        inputs, targets = torch.from_numpy(x), torch.from_numpy(y)

        # move inputs and targets to cuda
        
        **inputs, targets = inputs.cuda(), targets.cuda()**
            
        # New hidden state being created to prevented backpropogating through the 
        # entire history
        h = tuple([each.data for each in h])

        # zero out gradient to prevent accumulation
        model.zero_grad()

        # get output and hidden
        out, h = model.forward(inputs, h)
        loss = criterion(out, targets.view(n_seqs*n_steps).type(torch.cuda.LongTensor))

        # backpropogate loss
        loss.backward()

        # use gradient clipping to prevent exploding gradient
        nn.utils.clip_grad_norm_(model.parameters(), clip)

        # take a step in the los surface
        optimizer.step()

        if counter % print_every ==0:

            # initilize hidden state for validation
            val_hidden = model.init_hidden(n_seqs)
            val_losses = []

            for x, y in get_batches(val_data, n_seqs, n_steps):

                x = one_hot_encode(x, n_chars)

                x, y = torch.from_numpy(x), torch.from_numpy(y)

                val_hidden = tuple([each.data for each in val_hidden])
                
                inputs, targets = x, y
                
                if cuda:
                  **inputs, targets = inputs.cuda(), targets.cuda()**
                  
                out, val_hidden = model.forward(inputs, val_hidden)

                val_loss = criterion(out, targets.view(n_seqs*n_steps).type(torch.LongTensor))

                val_losses.append(val_loss.item())

                print('Epoch:'.format(epoch+1),
                      'Steps:'.format(counter),
                      'train loss {.:4f}'.format(loss.item),
                      'val loss {.:4f}'.format(np.mean(val_losses)))
</code></pre>
<p>running this with batch_size, i.e.,  n_seqs= 128<br>
the sequence length, which here is n_steps,  in the sense of time steps = 100</p>
<pre><code>    # define batch_size and sequence length
    n_seqs, n_steps = 128, 100

    train(model, encoded, epochs=25, n_seqs=n_seqs, n_steps=n_steps, lr=0.001, cuda=True, print_every=10)
</code></pre>
<p>This throws the followin error</p>
<hr>
<p>RuntimeError                              Traceback (most recent call last)<br>
 in ()<br>
1 n_seqs, n_steps = 128, 100<br>
----&gt; 2 train(model, encoded, epochs=25, n_seqs=n_seqs, n_steps=n_steps, lr=0.001, cuda=True, print_every=10)</p>
<p>4 frames<br>
/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py in nll_loss(input, target, weight, size_average, ignore_index, reduce, reduction)<br>
1869                          .format(input.size(0), target.size(0)))<br>
1870     if dim == 2:<br>
-&gt; 1871         ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)<br>
1872     elif dim == 4:<br>
1873         ret = torch._C._nn.nll_loss2d(input, target, weight, _Reduction.get_enum(reduction), ignore_index)</p>
<p>RuntimeError: Expected object of backend CUDA but got backend CPU for argument <span class="hashtag">#2</span> ‘target’</p>
<p>A quick check of whether the model parameters were registered by quora returns false.</p>
<pre><code>    next(model.parameters()).is_cuda
    False
</code></pre>
<p>Working in the colab environment.</p>
<pre><code>    torch.cuda.get_device_name()
    'Tesla T4'
</code></pre>
<p>Which is perplexing since I moved my parameters to cuda. I don’t get where I am going wrong. Please help, thank you so much for your time.</p>

ptrblck: <p>It looks like the error message is thrown in the validation loop.<br>
While you are using:</p>
<pre><code class="lang-python">loss = criterion(out, targets.view(n_seqs*n_steps).type(torch.cuda.LongTensor))
</code></pre>
<p>in your training loop, you are not creating a CUDA tensor in the validation loop:</p>
<pre><code class="lang-python">val_loss = criterion(out, targets.view(n_seqs*n_steps).type(torch.LongTensor))
</code></pre>

Shiva_Kumar: <p>Thanks a bunch. I made a ton of other such silly mistakes throughout my code, but now the whole thing works well. Thanks again</p>

ptrblck: <p>I’m glad the code is working now.<br>
To avoid such issues in the future, I would use the <code>.to()</code> operation with a specified <code>device</code>, so that changing this single variable will switch between CPU and GPU:</p>
<pre><code class="lang-python">device = 'cuda:0'
# device = 'cpu'  # set for CPU run

targets = targets.to(device)
x = torch.randn(1, device=device)
model.to(device)
...
</code></pre>

Shiva_Kumar: <p>Thanks a lot, I shall practise this going forward. If I were to do this, switching the model back to CPU for prediction would just entail calling model.cpu() right?</p>

ptrblck: <p>This would be one approach, but I would recommend to set <code>device='cpu'</code> and use the same code if possible.</p>

Shiva_Kumar: <p>Thank you, will surely do.</p>