foldl/torch-forum huggingface dataset:
How to get cuda variable gradient?

fxia22: why the result between CPU version and GPU version is different? How do we get grad for cuda variable? Thanks. CPU version: <SCODE>import torch
from torch.autograd import Variable

l = torch.nn.Linear(6,1)
input = Variable(torch.rand(10,6), requires_grad = True)
out = l(input)
target = Variable(torch.rand(10,1))
crt = torch.nn.L1Loss()
loss = crt(out, target)
loss.backward()
print input.grad
<ECODE> Output: GPU version: <SCODE>l = torch.nn.Linear(6,1).cuda()
input = Variable(torch.rand(10,6), requires_grad = True).cuda()
out = l(input)
target = Variable(torch.rand(10,1)).cuda()
crt = torch.nn.L1Loss().cuda()
loss = crt(out, target)
loss.backward()
print input.grad
<ECODE> Output: None

tom: The gradient does not work through .cuda(). So instead of use <SCODE>input = Variable(torch.rand(10,6).cuda(), requires_grad = True).
<ECODE> Best regards Thomas

fxia22: Thank you so much! -Fei

fabee: I just stumbled into the same problem. Can somebody explain the logic behind it? Seems very counter-intuitive.

smth:
Extract feature maps from intermediate layers without modifying forward()
Category: vision

jsalbert: I have this: <SCODE>def get_features_hook(self, input, output):
    print output.data.cpu().numpy().shape
    features = output.data.cpu().numpy()

model.features.module[37].register_forward_hook(get_features_hook)
model.forward(im_tensor)
<ECODE> Is there any way to extract that features value? Thanks

smth: There are many examples in this thread: Also a search will give you a few more examples: https://discuss.pytorch.org/search?q=register_hook

bhattad2: Hi, I am trying to extract feature outputs of the intermediate layers of pre-trained VGG 16 architecture and concatenate them. The in-built models in pytorch doesn’t have names for all its layers for VGG architecture. Therefore, I am unable to use register_forward_hook. Is there any other alternate way? I am trying to use something like below but I am not sure if gradients will be accumulated at the intermediate layers when I do back propagation as it is building up two computation graphs during instantiation of my model. I have something  like this <SCODE>def __init__(self):
        super(ModifiedVGG, self).__init__()
        k = [29, 22]
        model = models.vgg16(pretrained=True)
        self.layer1 =  nn.Sequential(*list(model.features.children())[:k[0]])
        self.layer2 =  nn.Sequential(*list(model.features.children())[:k[1]])
        self.conv1 = nn.Conv2d(512, 128, 3, padding=1)
        self.upsample1 = nn.UpsamplingBilinear2d(scale_factor=2)
def forward(self, x):
        conv_4_3 = self.layer2(x)
        conv_4_3 = F.relu(self.conv1(conv_4_3))
        conv_5_3 = self.layer1(x)
        conv_5_3 = F.relu(self.conv1(conv_5_3))
        conv_5_3 = self.upsample1(conv_5_3)
        concat_features = torch.cat([conv_4_3, conv_5_3], 1)
        return concat_features
<ECODE> Please let me know if this would work or not. Thank you!

smth: the simplest solution is to: copy over the vgg file change the name of the class modify the class (for example remove or replace layers as you see fit) use the function load_state_dict 196 to load the original VGG weights dict into this modified class.

bhattad2: Thank you. This would be the least error prone of all. Also, when I use register_forward_hook, do I need to worry about backward_hook or it will be taken care off automatically? Some more examples on register_hook would be appreciated. Thank you.

ne-bo: Thanks a lot this code works for me!

ssh983:
Indexing Multi-dimensional Tensors based on 1D tensor of indices

dasguptar: Hi, I have currently tried the following methods: Using gather:
seq_len = seq_len.view(-1, 1, 1).expand(output.size(0), 1, output.size(2))
output_extracted = torch.gather(output, 1, seq_len)
 Using index_select:
output_extracted = torch.cat([torch.index_select(output[i], 0, seq_len[i]) for i in xrange(output.size(0)])


chenyangh: 

dasguptar:
How to calculate the determinant of a variable?

maplewizard: 

AjayTalati: <SCODE>import torch
import numpy as np
from torch.autograd import Variable

a = Variable(torch.randn(5,5))
np.linalg.det(a.data.numpy())
<ECODE>

maplewizard: In fact, I want to get the gradient of the det w.r.t. each elements in the matrix.

Veril: 

dpernes: It’s sad that PyTorch does not have a determinant function. This basically makes it impossible (ok, very hard) to implement Gaussian mixture density networks with a full covariance matrix.

tom: the happy news is that between the last post and yours, we added documentations to make the Cholesky functions easier to find, so <SCODE>torch.potrf(a).diag().prod()
<ECODE> gives you the determinant. (The functions are exactly the same as in 0.1.12, too, but be sure to use the master docs.) Best regards Thomas

Yandong_Li: 

Yandong_Li: i can not use this function with new version of pytorch

tom: Either use the above on Tensors or Cholesky.apply with the linked austograd Function. Best regards Thomas

Yandong_Li: sure i can use this code torch.potrf(a).diag().prod() when a is a tensor but i need to do the operation to with autograd when i  call backward() function. Would you please help me solve this problems

tom: Hi, Best regards Thomas

Yandong_Li: Thanks for your help and i found that i got the square of the determinant with the code.

tom: If anyone is looking at this thread: Note that potrf has gained differentiability in master/0.3. Best regards Thomas

fehiepsi: 

tom: Best regards Thomas

fehiepsi: 

CNguyen:
Weird error before scripts end Exception TypeError: “‘NoneType’ object is not callable” by CuDNNHandle

norm_inf: Hi, I got this exception everytime my scripts contains torch model is finished training. Exception TypeError: "'NoneType' object is not callable" in <bound method CuDNNHandle.__del__ of <torch.backends.cudnn.CuDNNHandle instance at 0x7f32b48d0fc8>> ignored What is the cause ? Thanks

windweller: 

11145: I have the same problem. do you have solved it ?
Tunable Efficient Unitary Neural Networks (EUNN)

AjayTalati: Hi, I wonder if there’s been a PyTorch implementation of, It’s something that definitely seems to be a solid piece of work !

smth: There isn’t a PyTorch implementation of this publicly available as far as I know.

AjayTalati: Thanks! It’s in Tensorflow by one of the authors, Li Jing, Compared to LSTM, at least this is mathematically interpretable ! Multi-layer bi-directional LSTM works great, but you can’t do any theory on it?

flaport: Late to the party, but I will leave this here for anyone who bumps into this conversation.
How to calculate the gradient of images?
Category: vision

Michael: Thanks.

kmichaelkills: I guess you could represent gradient by a convolution with sobel filters. Maybe implemented with Convolution 2d filter with require_grad=false (where you set the weights to sobel filters).

yanp: 

Blade6570: Here is the code. It works perfectly. G_x=conv1(Variable(x)).data.view(1,256,512)

DhruvJawalkar: 

NjuHaoZhang: I have one of the simplest differentiable solutions. The idea comes from the implementation of tensorflow. The basic principle is: dx = I(x+1) - I(x), dy = I(y+1) - I(y), The code is as follows: <SCODE>def gradient_loss(gen_frames, gt_frames, alpha=1):

    def gradient(x):
        # idea from tf.image.image_gradients(image)
        # https://github.com/tensorflow/tensorflow/blob/r2.1/tensorflow/python/ops/image_ops_impl.py#L3441-L3512
        # x: (b,c,h,w), float32 or float64
        # dx, dy: (b,c,h,w)

        h_x = x.size()[-2]
        w_x = x.size()[-1]
        # gradient step=1
        left = x
        right = F.pad(x, [0, 1, 0, 0])[:, :, :, 1:]
        top = x
        bottom = F.pad(x, [0, 0, 0, 1])[:, :, 1:, :]

        # dx, dy = torch.abs(right - left), torch.abs(bottom - top)
        dx, dy = right - left, bottom - top 
        # dx will always have zeros in the last column, right-left
        # dy will always have zeros in the last row,    bottom-top
        dx[:, :, :, -1] = 0
        dy[:, :, -1, :] = 0

        return dx, dy

    # gradient
    gen_dx, gen_dy = gradient(gen_frames)
    gt_dx, gt_dy = gradient(gt_frames)
    #
    grad_diff_x = torch.abs(gt_dx - gen_dx)
    grad_diff_y = torch.abs(gt_dy - gen_dy)

    # condense into one tensor and avg
    return torch.mean(grad_diff_x ** alpha + grad_diff_y ** alpha)
<ECODE>

edgarriba: 

bumpybanana: You defined h_x and w_x, however you do not use these in the defined function. May I ask what the purpose of h_x and w_x are?
Set requires_grad to False but still slowing the speed

bilibili: <SCODE>class LCNPModel(nn.Module):
"""Container module with an encoder, a recurrent module, and a decoder."""

def __init__(self, inputs):
    super(LCNPModel, self).__init__()

    ....
    self.encoder_nt = nn.Embedding(self.nnt, self.dnt)
    self.word2vec_plus = nn.Embedding(self.nt, self.dt)
    self.word2vec = nn.Embedding(self.nt, self.dt) 

    self.LSTM = nn.LSTM(self.dt, self.dhid, self.nlayers, batch_first=True, bias=True)
    # the initial states for h0 and c0 of LSTM
    self.h0 = (Variable(torch.zeros(self.nlayers, self.bsz, self.dhid)),
            Variable(torch.zeros(self.nlayers, self.bsz, self.dhid)))

    .....

    self.init_weights(initrange)

    self.l2 = itertools.ifilter(lambda p: p.requires_grad == True, self.parameters())

def init_weights(self, initrange=1.0):
    self.word2vec_plus.weight.data.fill_(0)
    self.word2vec.weight.data = self.term_emb
    self.encoder_nt.weight.data = self.nonterm_emb      

    self.word2vec.weight.requires_grad = False
    self.encoder_nt.weight.requires_grad = False 
....
<ECODE> Hi, above is part of my code. In my code, I have a word2vec and word2vec_plus embeddings. So I would like an embedding which is initialized as word2vec pretrained vectors, but keep training it further. But when I use the optimizer, I would like to take the l2 norm of this embedding as the distance between the current embedding with the original word2vec embedding, which makes sense since I don’t want the new trained embedding to be too far away from the pretrained one. My problem is, when I set the word2vec.weight.requires_grad to False, and optimize parameters that require gradient, everything is fine but the training time is too slow after the first round. However, if I comment out everything with word2vec but only use word2vec_plus, everything is very fast. Since you can think of I am using word2vec just as a normal constant in my code, it is not supposed to slow down the model training. So my question is, if there is any way to speed up this process or is there anything that I am doing wrong? Thanks a lot!

jekbradbury: It’s probably the L2 norm distance between the two embedding matrices that’s taking forever to calculate, and there isn’t really a way around that for now (you may update only parts of the word2vec_plus embedding matrix at each iteration but you have to recompute the L2 norm over the whole matrix).

bilibili: Hi, The problem is that even when I comment out the word2vec embedding, which is a constant since I don’t require gradient for it, and optimize over everything else, I expect this is the same as not commenting out it (because the optimizer has nothing to do with it), it still gave me speed up. So I think even when requires_grad is false, the optimizer, for some reason, still looks at it?
Discussion: How to use a mediocre quality image dataset?
Category: vision

FuriouslyCurious: I am working with 2048x2048 px medical X-ray images, and I have two datasets.  One dataset is a _ dream come true <3 _ : nice, clean, and pixel-wise segmented.  This really helps in training a segmentation network. I also happen to have a second dataset, which is clean and nice but does not have pixel-wise segmentation.  Instead this dataset has a “Quadrant” feature which points to where the issue lies. What would you do with the second dataset?  Will love to hear some thoughts!
Newbie question: PyTorch not for production?

abagshaw: Really basic question here. I’ve read in a few places that PyTorch seems to be geared towards “development” and not production. Is there some reason why it is not suitable for production? The claims on the site, “The memory usage in PyTorch is extremely efficient”, “PyTorch is quite fast – whether you run small or large neural networks.” etc. seem to be all great benefits for running in production (and I suppose in development too)

smth: if you dont mind running python in production, then pytorch is ready for production. We also tend to always prioritize research flexibility over freezing specs / a more structured static model approach which is also suited for production.

abagshaw: Is PyTorch claiming performance improvements over libraries like Tensorflow even without C or cython?

smth: we dont claim performance improvements over any other library, we leave that exercise to the community.

Deeply: 

ptrblck:
Error about hooks in implementing customized modules

Shihan_su: Hi, I just implemented an customized modules as follows: <SCODE>def forward(self, x, para):
	''' 
	Implement eq 26 of ref paper for each batch.
	Input:
		para: dim(seq_len, batch, 121)
		x:    dim(seq_len, batch, 3)
	'''
	if x.size()[0] == para.size()[0]:
		self.seq_length = x.size()[0] - 1
		total_loss = 0
		for i in range(self.seq_length):
			# prepare parameters
			self.__get_para(i, x, para)
			normalpdf = self.__para2normal(self.x1, self.x2, self.m1, self.m2, self.s1, self.s2, self.rho) #dim (n_batch, 20)
			single_loss = self.__singleLoss(normalpdf)
			total_loss += single_loss
		return total_loss
	else:
		raise Exception("x and para don't match")


def __get_para(self, i, x, para):
	'''
	Slice and process parameters to the right form.
	Implementing eq 18-23 of ref paper.
	'''
	self.batch = x.size()[1]
	self.e = torch.sigmoid(-para[i,:,0])  # eq 18
	self.parameters = para

	# slice remaining parameters and training inputs
	self.pi, self.m1, self.m2, self.s1, self.s2, self.rho = torch.split(self.parameters[i,:,1:], 20, dim = 1) # dim(batch, 20)
	self.x1 = x[i+1,:,0].resize(self.batch, 1)  # dim(batch, 1)
	self.x2 = x[i+1,:,1].resize(self.batch, 1)
	self.et = x[i+1,:,2].resize(self.batch, 1)
	
	## process parameters
	# pi
	max_pi = torch.max(self.pi, dim = 1)[0]
	max_pi = max_pi.expand_as(self.pi)
	diff = self.pi - max_pi
	red_sum = torch.sum(diff, dim = 1).expand_as(self.pi)
	self.pi = diff.div(red_sum)

	# sd
	self.s1 = self.s1.exp()
	self.s2 = self.s2.exp()

	# rho
	self.rho = self.rho.tanh()


	# reshape ground truth x1, x2 to match m1, m2 because broadcasting is currently not supported by pytorch
	self.x1 = self.x1.expand_as(self.m1)
	self.x2 = self.x2.expand_as(self.m2)


def __para2normal(self, x1, x2, m1, m2, s1, s2, rho):
	'''
	Implement eq 24, 25 of ref paper.
	All input with dim(1, batch, 20)
	'''
	norm1 = x1.sub(m1)
	norm2 = x2.sub(m2)
	s1s2 = torch.mul(s1, s2)
	z = torch.pow(torch.div(norm1, s1), 2) + torch.pow(torch.div(norm2, s2), 2) - \
		2*torch.div(torch.mul(rho, torch.mul(norm1, norm2)), s1s2)
	negRho = 1 - torch.pow(rho, 2)
	expPart = torch.exp(torch.div(-z, torch.mul(negRho, 2)))
	coef = 2*np.pi*torch.mul(s1s2, torch.sqrt(negRho))
	result = torch.div(expPart, coef)
	return result

	
def __singleLoss(self, normalpdf):
	'''
	Calculate loss for single time stamp. eq 26
	Input: normalpdf (1,n_batch, 20).
	'''
	epsilon = 1e-20  # floor of loss from mixture density component since initial loss could be zero
	mix_den_loss = torch.mul(self.pi, normalpdf)
	red_sum_loss = torch.sum(torch.log(mix_den_loss))  # sum for all batch
	end_loss = torch.sum(torch.log(torch.mul(self.e, self.et) + torch.mul(1-self.e, 1 - self.et)))
	total_loss = -red_sum_loss - end_loss
	
	return total_loss/self.batch
<ECODE>

elanmart: Hi, <SCODE>class Loss(torch.nn.Module):
    def __init__(self):
        super(torch.nn.Module, self).__init__()

        ... # rest of the __init__ here
<ECODE>
How to do Convolution Backward manually

longcw: 

chenzhekl: Maybe there were something I missed. Why do you want to do backward propagation without forwarding first? The computational graph is built in the forwarding pass, without which it is not possible to do a backward propagation.

longcw: 

chenzhekl: Hi,

apaszke: 

longcw: Thank you for your reply.

magz:
Gradient w.r.t each sample

Suraj_Srinivas: Using model.[layer_name].weight.grad.data I am able to access the gradients for weights of a particular layer. However, I assume that these gradients are averaged across samples in a mini-batch. Is there a way for me to access the gradients of weights for each sample? I was able to obtain per-sample gradients for activations or neurons using register_hook, so not sure what to do about weights. Thanks!

smth: there is no way to access the gradients wrt weight for each individual sample. Gradients wrt weights in pytorch are always accumulated (even over the mini-batch). If you want gradients wrt each sample, you will have to run each sample individually through the network. This is because both the THNN backend and CuDNN dont support individual sample gradients wrt weight.
Max of a tensor and a scalar

vadimkantorov: There are workarounds, but an easy direct way of doing it would be more concise and probably efficient. Thanks!

elanmart: <SCODE>X = th.FloatTensor([
    [-1, 2],
    [2, -1]
])
scalar = th.FloatTensor([1])

th.max(X, scalar.expand_as(X))
<ECODE> gives you <SCODE> 1  2
 2  1
[torch.FloatTensor of size 2x2]
<ECODE> Please keep in mind that I’m playing with PyTorch for few days only, so that may not be the optimal solution.

vadimkantorov: 

smth: <SCODE>torch.clamp(tensor, max=value) # cmin
torch.clamp(tensor, min=value) # cmax
<ECODE> Edited to reflect what vadim said below

vadimkantorov: Seems it’s the other way around: It works but is a little confusing at first.

fmassa: 

HectorAnadon: Maybe someone is looking for this. Is there a way to do this without that much code? <SCODE>def min_(value1, value2):
    if is_number(value1) and isinstance(value2, torch.Tensor):
        return torch.clamp(value2, max=value1)
    elif is_number(value2) and isinstance(value1, torch.Tensor):
        return torch.clamp(value1, max=value2)
    elif isinstance(value1, torch.Tensor) and isinstance(value2, torch.Tensor):
        return torch.min(value1, value2)


def max_(value1, value2):
    if is_number(value1) and isinstance(value2, torch.Tensor):
        return torch.clamp(value2, min=value1)
    elif is_number(value2) and isinstance(value1, torch.Tensor):
        return torch.clamp(value1, min=value2)
    elif isinstance(value1, torch.Tensor) and isinstance(value2, torch.Tensor):
        return torch.max(value1, value2)

min_(2, torch.tensor(3))
min_(torch.tensor(3), 2)
min_(torch.tensor(3), torch.tensor(2))<ECODE>
Wired behavior of resize() function

Shihan_su: Hi, when I call resize of Variable, the underlying data changes as follows:

jekbradbury: 

pengalg: corect . but why resize change values in tensor ? It 's really weired.

colesbury: For example: <SCODE>>>> m = torch.arange(0, 25).view(5, 5)
  0   1   2   3   4
  5   6   7   8   9
 10  11  12  13  14
 15  16  17  18  19
 20  21  22  23  24
[torch.FloatTensor of size 5x5]
>>> x = m[:,0]
   0
   5
  10
  15
  20
 [torch.FloatTensor of size 5])
>>> x.resize_(5, 1)
 0
 1
 2
 3
 4
[torch.FloatTensor of size 5x1]
<ECODE>
Is pytorch for beginners in Deep Learning?

nafizh1: I have background in machine learning but I am a beginner in deep learning. A question for which I have not seen a direct answer anywhere is if the developers want it to be something that beginners get into DL with or this is geared towards deep learning researchers. Also, at the moment there seems to be not as much tutorials available for beginners as other frameworks such as Keras. Though, for me at least, I am finding pytorch to be not as high level as Keras, which is why I am really liking it. This helps me get a sense of what is happening under the hood when I am learning a new concept.

jekbradbury: It’s definitely meant to be accessible to beginners as well as researchers; part of the reason PyTorch is built the way it is so that it doesn’t have as much need for separate abstraction layers aimed at different use cases/user populations like tflearn or Keras. That should also make it easier to figure out how things work (at least until you get to the C backend) or move up/down the abstraction stack.
There are no graph nodes that require computing gradients

Shihan_su: https://github.com/ShihanSu/sequence-pytorch

miguelvr: what is datahp? what does the iterator return? I’m guessing that your problem is that you’re not casting the input and target data Variable type and therefore autograd can not compute the gradients… But again, I’m guessing! Didn’t spend too much time looking at your code.
How do I feed higher res images to model zoo models?
Category: vision

FuriouslyCurious: Can someone please explain what needs to be changed in following model definition to feed it 512x512 or 1024x1024 dimensions?  I am new from Keras and a bit lost here.  Thank you!

tymokvo: The size mismatch is coming from the output of the last avgpool layer feeding into the input of the resnet.fc module. To feed it larger images you will need to strip the existing fully connected layer and replace it with one with a larger input dimension. something like: <SCODE>your_net = models.resnet18(pretrained=True)`

new_num_features = *something bigger than 512*

your_net.fc = nn.Linear(new_num_features, 1000)
<ECODE> This will create untrained parameters, though. To use the pretrained models, I think you are restricted to image sizes below what you are using. this will show you the dimensions of the tensor that is created by whatever images you are using

FuriouslyCurious: 

raaj043: Hello, The CNN models like resnet, vgg and alexnet in pytorch are pretrained on which dataset?

achaiah: Would you be willing to share your code to do that? Thanks

macaodha: 

tymokvo: 

hyy123: I’d like to train the Inceptionv3 model with high-resolution images like 512x512. Do you know how to do it?
Installation error: ‘memcpy’ was not declared in this scope

ywu36: When I install the Pytorch package from source, it reports the following problem. My platform is: Ubuntu 16.06 + CUDA 7.5 Note that in Caffe installation, similar problem can be solved in here: By changing the CMakeLists.txt, I wonder whether we have some similar solutions in Pytorch.

ywu36: Further into this problem I found that it is caused by the version of gcc is too new: A usual way to solve it is to add a flag as: flags=-D_FORCE_INLINES before compiling. Is there any place I could insert this command installing Pytorch ?

apaszke: 

ywu36: Thanks for reply but the error is still there. I’m trying ‘CFLAGS’ instead of ‘CCFLAGS’.

ywu36: But still not working. Might because some other problems…

shiningsurya: Any update? Even I’m getting the same error.

smth: update your CUDA to 8.0

ken_zhang:
AttributeError: cannot assign module before Module.__init__() call

wasiahmad: I am getting the following error. AttributeError: cannot assign module before Module.__init__() call I have a class as follows. <SCODE>                                                   args.nlayers, args.dropout)
<ECODE> So, when I run the following line: question_classifier = QuestionClassifier(corpus.dictionary, embeddings_index, corpus.max_sent_length, args) What I am doing wrong here?

miguelvr: The first thing you should always do when you create a module is call its super constructor… So, your class should look like this: <SCODE>class QuestionClassifier(nn.Module):

    def __init__(self, dictionary, embeddings_index, max_seq_length, args):
        """"Constructor of the class"""
        super(QuestionClassifier, self).__init__()

        self.embedding = EmbeddingLayer(len(dictionary), args.emsize, args.dropout)
        self.encoder = EncoderRNN(args.emsize, args.nhid, args.model, args.bidirection, args.nlayers, args.dropout)
        
        self.drop = nn.Dropout(args.dropout)<ECODE>

AbdulsalamBande: