foldl/torch-forum huggingface dataset:
RuntimeError : Gradients are not CUDA tensors

Rizwan_Parvez: I am getting the following error while experimenting with a simple lstm layer. The forward propagation is fine but while computing loss.backward() I am getting the following error. any suggestions about why I am doing wrong?

smth: Can you post your simple script to see what you are doing wrong?

cheekala: I’m encountering the same error too. Followed is my train() function. def train(input_batch, input_batch_length, target_batch,target_batch_length, batch_size): <SCODE># Zero gradients of both optimizers
encoderchar_optimizer.zero_grad()
encoder_optimizer.zero_grad()
decoder_optimizer.zero_grad()

encoder_input = Variable(torch.FloatTensor(len(input_batch),batch_size,500))

for ix , w in enumerate(input_batch): 
    w = w.contiguous().view(15,batch_size)
    reshaped_input_length = [ x[ix] for x in input_batch_length] # [15 ,.. 30 times] * 128
    if USE_CUDA: 
        w = w.cuda()
        #reshaped_input_length =  Variable(torch.LongTensor(reshaped_input_length )).cuda()
    hidden_all , output = encoderchar(w,reshaped_input_length)
    encoder_input[ix] = output.transpose(0,1).contiguous().view(batch_size,-1)
    if USE_CUDA: 
        encoder_input = encoder_input.cuda()
  
temporary_target_batch_length = [15] * batch_size

#if USE_CUDA: 
    #target_batch_length =  Variable(torch.LongTensor(target_batch_length )).cuda()

encoder_hidden_all, encoder_output = encoder(encoder_input, target_batch_length)
decoder_input = Variable(torch.LongTensor([SOS_token] * batch_size))
decoder_hidden = encoder_output

max_target_length = max(temporary_target_batch_length)
all_decoder_outputs = Variable(torch.zeros(max_target_length, batch_size, decoder.output_size))

# Move new Variables to CUDA
if USE_CUDA:
    decoder_input = decoder_input.cuda()
    all_decoder_outputs = all_decoder_outputs.cuda()
    target_batch =  target_batch.cuda()
    ##Added by Satish 
    encoder_hidden_all  = encoder_hidden_all.cuda()
    encoder_output = encoder_output.cuda()
    decoder_hidden = decoder_hidden.cuda()
# Run through decoder one time step at a time
for t in range(max_target_length):
    decoder_output, decoder_hidden, decoder_attn = decoder(
        decoder_input, decoder_hidden, encoder_hidden_all
    )

    all_decoder_outputs[t] = decoder_output
    decoder_input = target_batch[t] # Next input is current target
    if USE_CUDA:
        decoder_input = decoder_input.cuda()

if USE_CUDA:
    all_decoder_outputs = all_decoder_outputs.cuda()
<ECODE> <SCODE>loss = masked_cross_entropy(
    all_decoder_outputs.transpose(0, 1).contiguous(), # -> batch x seq
    target_batch.transpose(0, 1).contiguous(), # -> batch x seq
    target_batch_length
)
loss.backward()

# Clip gradient norms
ecc = torch.nn.utils.clip_grad_norm(encoderchar.parameters(), clip)
ec = torch.nn.utils.clip_grad_norm(encoder.parameters(), clip)
dc = torch.nn.utils.clip_grad_norm(decoder.parameters(), clip)

# Update parameters with optimizers
encoderchar_optimizer.step()
encoder_optimizer.step()
decoder_optimizer.step()

return loss.data[0], ec, dc
<ECODE> any inputs what I’m doing wrong?
PackedSequence with DataLoader

miguelvr: Hey, I’m trying to reproduce some previous work I did with Theano in PyTorch, with RNNs. I want to be able to mask the sequences I pass as input to an RNN based model. This should be easy enough… However, there’s a couple of annoying issues that are bugging me: Cheers!

jekbradbury: 

miguelvr: What’s the best way to sort / rearrange tensors in PyTorch?

ShigekiKarita: is this correct usage? <SCODE>import torch
x = torch.autograd.Variable(torch.randn(5, 3, 2))
l = torch.nn.Linear(2, 2)
r = torch.nn.RNN(2, 2)

px = torch.nn.utils.rnn.pack_padded_sequence(x, [5, 3, 1])
ph = torch.nn.utils.rnn.PackedSequence(l(px.data), px.batch_sizes)
py, h = r(ph)
<ECODE>

jekbradbury: Yes, that looks right to me.

stefanonardo: Is this true? It is not working for me since nn.Linear requires attribute ‘dim’.

eplu: I’m observing the same behavior. I don’t think linear layers can take packed sequences as inputs.

nlgranger: 

Moore: 

Moore: Is there possible any better way now to apply the packed sequence with the linear layers?
How to store all hidden representation for each of the word in a sentence using LSTM?

wasiahmad: I am facing a problem to store the hidden variables (h and c) while I am looping through the words in a sentence. I am trying to store all the hidden state representation for each word in the sentence and use it for later computation. For example, if we have a sentence with 10 words, I want to store all the hidden states for each word in different Variables. I am using LSTM based simple encoder as follows. <SCODE>encoder_output, encoder_hidden = self.encoder(embedded, encoder_hidden)
<ECODE> RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation I have spent significant amount of time to find out the problem and found the reason but I am unable to solve it. I have tried using a list of Variable which should work but still getting the same error. <SCODE>embedded = self.embedding(input_variable)
encoder_output, encoder_hidden = self.encoder(embedded, (hidden_states[idx], cell_states[idx]))
hidden_states.append(encoder_hidden[0])
cell_states.append(encoder_hidden[1])
<ECODE> Even I tried to clone a Variable, but the problem is not resolved. Any help would be much appreciated. Thanks.

wasiahmad:
[Sovled] Pytorch-example openmt, ImportError: No module named utils.rnn

zeng: what cause this problem and how to solve this problem? the whl links depend on what you’ve installed before. <SCODE>pip uninstall http://download.pytorch.org/whl/cu80/torch-0.1.9.post2-cp27-none-linux_x86_64.whl 
<ECODE> <SCODE>pip install http://download.pytorch.org/whl/cu80/torch-0.1.10.post2-cp27-none-linux_x86_64.whl 
<ECODE>
CudaTensor Variable

IouJenLiu: I tried to wrapper autograd.Variable to send the data to the GPU every time I construct a Variable <SCODE>class Variable(autograd.Variable):
      def __init___(self, data, *args, **kwargs):
          data = data.cuda()
          super(Variable, self).__init___(data, *args, **kwargs)
  
  a = torch.randn(1,1)                                                                                                                                                                                    
  print(a)
  print(Variable(a))
  print(Variable(a.cuda()))
<ECODE> However, I got the output as follows: <SCODE>-0.2344
[torch.FloatTensor of size 1x1]

Variable containing:
-0.2344
[torch.FloatTensor of size 1x1]

Variable containing:
-0.2344
[torch.cuda.FloatTensor of size 1x1 (GPU 0)]
<ECODE> Does anyone get the same problem? Thank you!

smth: Here’s a proper working such wrapper. <SCODE>def Variable(data, *args, **kwargs):
    return autograd.Variable(data.cuda(), *args, **kwargs)
<ECODE>

IouJenLiu: It works. Thanks a lot!
Skip layer in pyTorch

wzhang35: Thank you all for the help in this fabulous forum. I was a Torch user, and new to pytorch. Right now I want to do something like “skip connection”. Below is the code I used in torch. How can I make a “skip connection” in pytorch <SCODE>main = nn.Sequential()
...
local conc = nn.ConcatTable()
local conv = nn.Sequential()
conv:add(SpatialConvolution(...))
conc:add(nn.Identity())
conc:add(conv)
main:add(conc)
main:add(nn.CAddTable())<ECODE>

DiffEverything: Assuming proper padding for compatible sizes - <SCODE>input = Variable(torch.Tensor(...))
conv_out =self.conv(input)
out = conv_out + input

<ECODE>

mattmacy:
Torch.transpose is too slow in GPU,slower than CPU

Kyle: <SCODE>import torch
import time
from torch.autograd import Variable

x = Variable(torch.randn(100,500))
cputimes = []
for sampl in (1000, 10000, 100000, 1000000):
    start = time.time()
    for i in range(sampl):
        y = torch.transpose(x,0,1)
    end = time.time()
    cputimes.append(end-start)
print(cputimes)

x = x.cuda(device_id=2)
gputimes = []
for sampl in (1000, 10000, 100000, 1000000):
    start = time.time()
    for i in range(sampl):
        y = torch.transpose(x,0,1)
    end = time.time()
    gputimes.append(end-start)
print(gputimes)
<ECODE> test2: <SCODE>In [16]: torch.cuda.set_device(2)

In [17]: %timeit torch.transpose(torch.FloatTensor(20,100),1,0)
The slowest run took 26.26 times longer than the fastest. This could mean that an intermediate result is being cached.
100000 loops, best of 3: 1.72 µs per loop

In [18]: %timeit torch.transpose(torch.cuda.FloatTensor(20,100),1,0)
The slowest run took 21.21 times longer than the fastest. This could mean that an intermediate result is being cached.
100000 loops, best of 3: 3.05 µs per loop<ECODE>

Kyle: It case my model run very slowly,how can I solve it

albanD: Hi, This code sample is slow only because of the python loop which calls c functions. To make it faster, you need to find a way to remove this loop. If in your case, you want to transpose a bunch of matrices, you could for example stack them in a single tensor and then call transpose on this tensor.

elanmart: <SCODE>class NoOp(torch.autograd.Function):
    def __init__(self):
        super().__init__()
        
    def forward(self, x):
        return x

def print_times(x, func, msg):
    start = time.time()
    for i in range(1000000):
        _ = func(x)
    t = time.time() - start
    print("{}: {:.5f}".format(msg, t))

tensor   = torch.randn(100, 500)
ndarray  = tensor.numpy()
variable = Variable(tensor)
    
print_times(tensor,   lambda x: x,             "Python noop")
print_times(ndarray,  lambda x: x.transpose(), "Numpy transpose")
print_times(tensor,   lambda x: x.t(),         "Torch transpose")
print_times(variable, lambda x: NoOp()(x),     "Autograd noop")
print_times(variable, lambda x: x.t(),         "Autograd transpose")

# output:
#
# Python noop: 0.07554
# Numpy transpose: 0.23783
# Torch transpose: 0.49813
# Autograd noop: 1.95098
# Autograd transpose: 3.72835
<ECODE>

Kyle: Actually I changed my code and remove the transpose op, but the model still run slower on GPU than CPU, here is some of my model code . <SCODE>x = x.unsqueeze(1)
x0 = F.relu(self.conv0_0(x)).squeeze(3)
x1 = F.relu(self.conv0_1(x)).squeeze(3)
x = torch.cat((x0,x1),1)

x = x.unsqueeze(1)
x0 = F.relu(self.conv1_0(x)).squeeze(2)
x1 = F.relu(self.conv1_1(x)).squeeze(2)

x0 = F.max_pool1d(x0, x0.size(2)).squeeze(2)
x1 = F.max_pool1d(x1, x1.size(2)).squeeze(2)
x = torch.cat((x0,x1),1)
<ECODE>

albanD:
Problem using softmax

pointW: here is the Variable before softmax after softmax, the biggest one of them became 1 and others became 0 my forward function <SCODE>def forward(self, x):
    x = F.relu(self.conv1(x))
    x = F.relu(self.conv2(x))
    x = F.relu(self.conv3(x))
    x = F.relu(self.conv4(x))
    x = F.relu(self.conv5(x))
    x = F.relu(self.conv6(x))
    x = F.relu(self.conv7(x))
    x = x.view(x.size(0), -1)
    x = F.softmax(x)
    return x
<ECODE> is there something wrong in my usage of softmax?

tom: Best regards Thomas

chenyuntc: the number  is too large,its exp overflow. I I think you should  add batchnorm layer before convolution layer.

pointW: 

pointW:
Strange output of torch.max()

melody-rain: <SCODE>_, max_probs = torch.max(output_var, 2)
print output_var.size()
print max_probs.size()
print torch.max(max_probs)
<ECODE> the outputs: <SCODE>(10L, 26L. 37L)
(10L. 26L, 1L)
37
<ECODE> So size of output_var is (10L, 26L. 37L) and with <SCODE>_, max_probs = torch.max(output_var, 2)
<ECODE> It is a bug of pytorch?

smth: Hi Melody, <SCODE>import torch
from torch.autograd import Variable

output_var = Variable(torch.randn(10, 26, 37).cuda())

_, max_probs = torch.max(output_var, 2)
print output_var.size()
print max_probs.size()

for i in range(1000):
    assert torch.max(max_probs).data[0] == 36
<ECODE> <SCODE>nvidia-smi
<ECODE>
How do I know whether pytorch is built with magma-cuda80?

wangg12: <SCODE>---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-6-9c860559505b> in <module>()
----> 1 torch.svd(torch.rand(3,3).cuda())

RuntimeError: No CUDA implementation of 'gesvd'. Install MAGMA and rebuild cutorch (http://icl.cs.utk.edu/magma/) at /data/wanggu/software/pytorch/torch/lib/THC/generic/THCTensorMathMagma.cu:280
<ECODE>

smth: 

wangg12: I set the cmake prefix to which conda，do I need to set it again when I rebuild pytorch？

smth: export CMAKE_PREFIX_PATH="$(dirname $(which conda))/../"

wangg12:
[resolved] Runtime error assigning hidden state

vvanirudh: I am using a GRUCell at each time-step to pass input and hidden state and get back the new hidden state in the following way: <SCODE>hidden_states[i, j, :] = GRUCell(input, hidden_states[i, j, :])
<ECODE> where <SCODE>hidden_states = Variable(torch.zeros(N, N, 1, hidden_state_size)).cuda()
<ECODE> As you can notice that instead of sending a batch of inputs, I am sending one input at a time (hence, the 1 in the tensor size). When I run this, I get a runtime error as follows: <SCODE>RuntimeError: in-place operations can be only used on variables that don't share storage with any other variables, but detected that there are 2 objects sharing it
<ECODE> Can anyone explain how I have created another variable with the same storage as hidden_states?

vvanirudh: Resolved. I was using hidden_states elsewhere in the code and creating a new variable that shared the storage.
Multiply feature map by a learnable scalar

Alexey_Chernyavskiy: 

fmassa: <SCODE>matrix = Variable(torch.rand(3,3))
scalar = Variable(torch.rand(1), requires_grad=True)
output = matrix * scalar.expand_as(matrix)
<ECODE>

Alexey_Chernyavskiy: in my Model definition: <SCODE>    self.multip = torch.autograd.Variable(torch.rand(1).cuda(), requires_grad=True)
    self.multip = self.multip.cuda()
<ECODE> in my Model forward: <SCODE>def forward(self, x):
  x1 = self.relu(self.conv1(x))
  x2 = self.relu(self.conv2(x))
  x1 = x1 * self.multip.expand_as(x1) # multiply x1 output by learnable parameter "multip"
  x = torch.add(x1, x2)
  return x
<ECODE>

fmassa: Quick example: <SCODE>class Model1(nn.Module):
    def __init__(self):
        super(Model1, self).__init__()
        self.multp = Variable(torch.rand(1), requires_grad=True)

class Model2(nn.Module):
    def __init__(self):
        super(Model2, self).__init__()
        self.multp = nn.Parameter(torch.rand(1)) # requires_grad is True by default for Parameter

m1 = Model1()
m2 = Model2()

print('m1', list(m1.parameters()))
print('m2', list(m2.parameters()))
<ECODE>

Alexey_Chernyavskiy: Now, how can I specify a special learning rate for this new param? I used to do something like this for my convolutional layers to make a specific smaller learning rate for layer ‘conv2’. UPD: I replaced my ‘custom’ Adam (see above) that had different learning rates by a regular Adam which has the same learning rate for all the layers, and it worked - my multp started to become updated during the course of training (it also worked with SGD). However my question with setting a special learning rate for the nn.Parameter remains.

fmassa: You can simply do something like <SCODE>optimizer = optim.Adam([{'params':[model.multp], 'lr':opt.lr*0.1}], lr=opt.lr)
<ECODE>

Alexey_Chernyavskiy: Great, it worked! Thank you.

Andrefmds: Hey, I was wondering if is possible to mutiply each feature map by a different scalar?

Chenhongyi_Yang: Maybe using a 1x1 conv with group=n_channels?

mbutt: Hi did you work it out? I also need to do something similar.
Efficient way to make datasets?

tymokvo: So I’m using a script to turn a directory of images in 5 subdirectories into a single tensor of size (730, 3, 256, 256) and a label tensor of size (730, 5) for 5 classes and then torch.utils.data to turn that into a TensorDataset and make/shuffle batches. The batches are then moved to the GPU individually at each iteration through the dataset during training. However, this isn’t a tenable practice for a very large dataset. Is there a better way to do this that I’m not seeing in the docs? It seems like there should be a simpler way to read images from disk into shuffled batches rather than having to put the whole thing into two tensors in system memory.

fmassa: 

tymokvo: Thanks! That’s pretty handy
[resolved] Has anyone noticed significant slowdown after upgrading to v0.1.11?

gaoking132: Yesterday I upgraded my pytorch to v0.1.11. I ran the same training script. I noticed that the time for each batch has increased from 0.8 seconds to 2.9 seconds. Nothing is changed except the pytorch. My old version is 0.1.9+67f9455. Now I am trying to upgrade to cudnn v6 to see whether it could solve my isse. I wonder whether anyone had the same problem with me. Yaozong

gaoking132: Problem solved!  After I used the pre-built binary install, everything works as fast as usual. Before that, I was building the pytorch manually pulled from the master branch.

jekbradbury: If I had to guess, you’re running on CPU and when compiled from source it wasn’t being linked properly with MKL.
How to demonstrate (or teach) learning theory using PyTorch

AjayTalati: Hi, There’s a lot of beautiful theory on function approximation, for example, is quite readable.

AjayTalati: I’d be interested in implementing them in PyTorch - I’ll be doing some DL theory teaching/demos over the summer. This would also be very helpful to others who are giving DL demos/teaching. Thanks a lot for your help, Aj

smth: In general, to give you my completely honest and regularized opinion, teaching something like “Learning Real and Boolean Functions” is way more valuable to teach in numpy than in pytorch. I wont pretend, numpy is much more accessible. Unless you need to really teach / showcase a gradient based learning method (SVM-SGD?) or teach GPU based whatever, you can maybe just stick to numpy.

tom: Best regards Thomas

Bhavya_Bhatt:
Mysterious nn.Parameter behavior

username1: <SCODE>import torch
from torch import nn

class Model(nn.Module):
    def __init__(self, deg):
	super(Model, self).__init__()
	self.deg = deg + 1

	self.theta = nn.Parameter(torch.ones(self.deg), 1)

     def forward(self, xs):
	phi = torch.cat([xs**i for i in range(self.deg)], 1)
	res = phi @ self.theta 
            # phi.mv(self.theta) # if not python 3.5 or later
	print(res)
	return res

model = Model(2)
#print(list(model.parameters()))
x = FloatTensor([1, 2,3, 4]).unfold(0,1,1)

print(model(x))
<ECODE> Here is the error that I get: <SCODE>Traceback (most recent call last):
  File "01-polynomial-fitting.py", line 40, in <module>
    print(model(x))
  File "/home/arseni/.anaconda/envs/mlt/lib/python3.6/site-packages/torch/nn/modules/module.py", line 206, in __call__
    result = self.forward(*input, **kwargs)
  File "01-polynomial-fitting.py", line 29, in forward
    res = phi @ self.theta
  File "/home/arseni/.anaconda/envs/mlt/lib/python3.6/site-packages/torch/tensor.py", line 357, in __matmul__
    return self.mv(other)
TypeError: mv received an invalid combination of arguments - got (Parameter), but expected (torch.FloatTensor vec)
<ECODE> <SCODE>Traceback (most recent call last):
  File "01-polynomial-fitting.py", line 40, in <module>
    print(model(x))
  File "/home/arseni/.anaconda/envs/mlt/lib/python3.6/site-packages/torch/nn/modules/module.py", line 215, in __call__
    var = var[0]
TypeError: 'float' object is not subscriptable
<ECODE> <SCODE>Traceback (most recent call last):
  File "01-polynomial-fitting.py", line 41, in <module>
    print(model(x))
  File "/home/arseni/.anaconda/envs/mlt/lib/python3.6/site-packages/torch/nn/modules/module.py", line 206, in __call__
    result = self.forward(*input, **kwargs)
  File "01-polynomial-fitting.py", line 29, in forward
    res = (self.theta @ phi.t())
  File "/home/arseni/.anaconda/envs/mlt/lib/python3.6/site-packages/torch/autograd/variable.py", line 770, in __matmul__
    return self.unsqueeze(0).mm(other).squeeze(0)
  File "/home/arseni/.anaconda/envs/mlt/lib/python3.6/site-packages/torch/autograd/variable.py", line 523, in mm
    output = Variable(self.data.new(self.data.size(0), matrix.data.size(1)))
AttributeError: 'FloatTensor' object has no attribute 'data'
<ECODE> What am I doing wrong ?

fmassa: <SCODE>print(model(Variable(x)))
<ECODE>

username1: Thanks for your answer. Now its working.
Argmax with PyTorch

MatthewKleinsmith: How can I use argmax with PyTorch? Meta: How could I have answered my own question? I searched the PyTorch docs and the PyTorch repo for “argmax” but got no results. Does it make sense to use argmax with a GPU?

fmassa: <SCODE>values, indices = tensor.max(0)
values, indices = torch.max(tensor, 0)
<ECODE>

platero: 

Andre_Holzner: 

platero: Thank you very much!

bgenchel: Any ideas for how to use this max function in a differentiable way? A custom loss function i’m writing has to do with the indices of max values. Not sure how to redo the loss function such that it uses differentiable components.

dpernes: I am not sure if I understand your problem. Do you want to have gradients with respect to indices? Well, indices are integers by definition and you cannot take derivatives of a function with respect to a variable that is defined over the integers only…

Brando_Miranda: what does the 0 do in the indexing?

BlakeWest: 

jchillin: you can now do torch.argmax(preds, dim=1) in version 0.4.0

chenchr: 

Brando_Miranda: what about argmax from the tensor itself like: <SCODE>import torch

x = torch.randn(3)
x.argmax(-1)
<ECODE> probably useful: Calculating accuracy of the current minibatch? 8 https://stackoverflow.com/questions/51503851/calculate-the-accuracy-every-epoch-in-pytorch 2 https://towardsdatascience.com/understanding-dimensions-in-pytorch-6edf9972d3be 3 How does one get the predicted classification label from a pytorch model? 6

Brando_Miranda: