foldl/torch-forum huggingface dataset:
(Newbie Question) Getting the gradient of output with respect to the input

ckanbak: Hello all, I’m new to using neural network libraries, so I’m sorry if this is  a stupid question. I have a pretrained network with a 28x28 input(MNIST) image and 10 outputs. I want to get the gradient of one of those outputs wrt the input. Thank you very much in advance!

DiffEverything: 

albanD: <SCODE>import torch
a = torch.autograd.Variable(torch.Tensor(10,10))
print(a.requires_grad) # prints False
<ECODE>

DiffEverything: 

ckanbak: 

albanD: Hi, <SCODE>for digit in selected_digits:
    output[digit].backward(retrain_variables=True)
<ECODE>

Guan-Horng_Liu: <SCODE>net1.zero_grad(); net2.zero_grad()
x = Variable(torch.ones((1,20)), requires_grad=True)
y = net1(x)
z = net2(y)
z.backward(torch.ones(z.size()),retain_variables=True)
print(x.grad) # this will give non-zero value
print(y.grad) # this is all zero

net1.zero_grad(); net2.zero_grad()
y = torch.from_numpy(y.data.numpy())
y = Variable(y, requires_grad=True)
z = net2(y)
z.backward(torch.ones(z.size()),retain_variables=True)
print(y.grad) # this will give non-zero value<ECODE>

smth: 

fromLittleAcorns: Hi, would this still be the preferred approach in version 1.0, I can’t seem to get retrain_variable=True to work and am trying to do the same thing Thanks John

albanD: Hi,

fromLittleAcorns: If I do this will the gradients coming out of input increment each time or will they be overwritten.  I read that gradients are retained and only cleared when the model is zero grad is called.  If they are incremental then can I get each individual value but taking the difference of the previous value and the latest? Thanks again

albanD:
How to manipulate layer parameters by it’s names?

zeakey: So how can I set one specific layer’s parameters by the layer name, say “conv3_3” ? In pytorch I get the model  parameters via: <SCODE>params = list(model.parameters())

for p in params:
    print p.size()
<ECODE> But how can I get parameter according to a layer name and then change its values? <SCODE>caffe_params = caffe_model.parameters()

caffe_params['conv3_1'] = np.zeros((64, 128, 3, 3))
<ECODE>

smth: 

solidor: <SCODE>optimizer = optim.Adam([param for name, param in model.state_dict().iteritems()
                        if 'foo' in name], lr=args.lr)
<ECODE>

solidor: 

Nan_Meng: Hi Smth, Is there some way I can name a layer using a custom name. Thanks.

chsasank: 

Nan_Meng: thank you so much !!!

Emrah_Yigit: 
nn.Module.named_parameters() does not contain parameters like batch_norm’s running_mean and running_var
 
nn.Module.parameters() does not contain parameter names 
model.state_dict() does not contain parameters that we can update requires_grad property

An_Tran: <SCODE>for name, param in model.named_parameters():
    param.requires_grad = False```<ECODE>

Brando_Miranda: <SCODE># https://discuss.pytorch.org/t/how-to-get-the-module-names-of-nn-sequential/39682
# looping through modules but get the one with a specific name

import torch
import torch.nn as nn

from collections import OrderedDict

params = OrderedDict([
    ('fc0', nn.Linear(in_features=4,out_features=4)),
    ('ReLU0', nn.ReLU()),
    ('fc1L:final', nn.Linear(in_features=4,out_features=1))
])
mdl = nn.Sequential(params)

# throws error
# mdl['fc0']

for m in mdl.children():
    print(m)

print()

for m in mdl.modules():
    print(m)

print()

for name, m in mdl.named_modules():
    print(name)
    print(m)

print()

for name, m in mdl.named_children():
    print(name)
    print(m)
<ECODE>

whanafy: Just to be sure, this will replace the current module with this new module if the name is the same?

gilbertocunha: Just wanted to know, have you found any solution to this problem?

Alon41: For example, to scale a specific layer by a scalar c I did the following: <SCODE>weight_name = 'fc2.weight'
c = 100
for name, param in model.named_parameters():
    if param.requires_grad:
        if name == weight_name:
            param.data = c*param.data
<ECODE>
How to exclude Embedding layer from Model.parameters()?

ShawnGuo: I use an embedding layer to project one-hot indices to continuous space. However, during the training, I don’t want to update the weight of it. How could I do that?

smth: you can set the weight of the embedding layer to not require grad. <SCODE>m = nn.Embedding(...)
m.weight.requires_grad=False
<ECODE>

ShawnGuo: Oh, I see. Thank you very much.

ShawnGuo: <SCODE>ValueError: optimizing a parameter that doesn't require gradients
<ECODE> The optimizer is used in the following way: <SCODE>self.optimizer = optim.Adadelta(self.model.parameters(), lr=args.learning_rate)
<ECODE> And, model is defined as follow: <SCODE>class DecomposableModel(nn.Module):
    def __init__(self, word_embedding, config):
        super(DecomposableModel, self).__init__()
        self.name = 'DecomposableModel'

        self.drop_p = config['drop_p']

        self.word_dim = word_embedding.embeddings.size(1)
        self.embedding = nn.Embedding(word_embedding.embeddings.size(0), self.word_dim)
        self.embedding.weight = nn.Parameter(word_embedding.embeddings)
        self.embedding.weight.requires_grad = False
        # self.embedding_normalize()

        self.F = nn.Linear(self.word_dim, config['F_dim'])
        self.G = nn.Linear(2 * self.word_dim, config['G_dim'])
        self.H = nn.Linear(2 * config['G_dim'], config['relation_num'])

        self.cuda_flag = config['cuda_flag']

    def forward(self, p_ids, h_ids):
        ......
<ECODE>

albanD: Hi,
Operation between Tensor and Variable

Zafarullah_Mahmood: Suppose I do this: <SCODE>x = Variable(torch.rand(3,5))
y = torch.Tensor(torch.rand(3,5))
print(x + y)
<ECODE>

JerryLin: 

Zafarullah_Mahmood: 

vabh: <SCODE>In [2]: y =  torch.autograd.Variable(torch.Tensor(4))
In [3]: y.requires_grad
Out[3]: False<ECODE>

maggie: I also have question about this. Why is it designed like this?
How to use pack_padded_sequence in seq2seq models

VladislavPrh: Then when I sort sequences by length I get: So, h(hidden state) and c(cell state) from encoder will be incorrect for corresponding sentences in the decoder in the batch. How can I solve it? Or I shouldn’t use pack_padded_sequence? Thanks!

jekbradbury: You should keep track of the order somehow. It’s pretty common for seq2seq models to use some kind of attentional input feeding in the decoder which prevents nn.LSTM from being used for all decoder timesteps in one call; in that case I sort the batch based only on source sentence length and use an unrolled LSTM cell in the decoder. Code that sorts the batch together according to the source sentence length will be in torchtext shortly.

VladislavPrh: 

jekbradbury: 

vijendra_rana: 

DuaneNielsen: After attempting to implement seq2seq with packed_sequences, I have come to the conclusion that this is not really feasable as, It appears to me that fundamentally, packed sequences require a lot of fenangling to work as currently implemented.  It feels like they could benefit from a little more abstraction to make things a bit easier for the programmer. Happy to be corrected/proven wrong here!
What is action.reinforce(r) doing actually?
Category: reinforcement-learning

yunjey: Hi, Thanks.

apaszke: 

yunjey: 

zuoxingdong: 

apaszke: 

wjaskowski: 

jekbradbury: 

wjaskowski: 

jekbradbury: I think I see what you’re asking now. The gradient estimates from all the examples in the batch are added together in the stochastic node (there’s no implicit division by the batch size) so, depending on your use case, you may want to manually divide your rewards by the batch size.

abhigenie92: 

Crazyai: <SCODE>x = Variable(torch.Tensor([0.1, 0.9]), requires_grad=True)
y = torch.multinomial(x, 1)  # here, make sure y.data[0] = 1
r = torch.Tensor([2]).float()
y.inforce(r)
y.backward()
print x.grad.data
<ECODE>

volpato30: I implemented Vanilla Policy Gradient in this way and it works. However using the .reinforce method seems cleaner

zuoxingdong: To make it simple, let’s say 1D, given a mean and std, we have a sample = mean + std*eps, where eps ~ N(0, 1). It is not very clear to me why it is like that. Since if we have a sample, according to the formula,  d_sample/d_mean = 1. So, grad_mean = upward_gradient * d_sample/d_mean

lucasb-eyer: 

norm: 

zeng: I do not notice the paper you mentioned, can you give a PDF file or the link. Thanks.

lucasb-eyer: 

hughperkins: <SCODE>"""
Try some reinforce by hand and similar
"""

import torch
from torch import autograd, nn, optim
import torch.nn.functional as F
from torch.autograd import Variable
import numpy as np


def run_model(x, h1):
    torch.manual_seed(123)
    params = list(h1.parameters())

    x1 = h1(Variable(x))
    x2 = F.softmax(x1)
    a = torch.multinomial(x2)
    print('a', a)
    return x1, x2, a


def run_by_hand(params, x, h1):
    print('')
    print('=======')
    print('by hand')
    h1.zero_grad()
    x1, x2, a = run_model(x, h1)
    g = torch.gather(x2, 1, Variable(a.data))
    log_g = g.log()
    log_g.backward(- torch.ones(4, 1))
    print('params.grad', params.grad)


def run_pytorch_reinforce(params, x, h1):
    print('')
    print('=======')
    print('pytorch reinforce')
    h1.zero_grad()
    x1, x2, a = run_model(x, h1)
    a.reinforce(torch.ones(4, 1))
    autograd.backward([a], [None])
    print('params.grad', params.grad)


def run():
    N = 4
    K = 1
    C = 3

    torch.manual_seed(123)
    x = torch.ones(N, K)
    h1 = nn.Linear(K, C, bias=False)
    params = list(h1.parameters())[0]
    run_by_hand(params, x, h1)
    run_pytorch_reinforce(params, x, h1)


if __name__ == '__main__':
    run()
<ECODE> Result: <SCODE>=======
by hand
a Variable containing:
 1
 1
 0
 1
[torch.LongTensor of size 4x1]

params.grad Variable containing:
 0.6170
-1.3288
 0.7117
[torch.FloatTensor of size 3x1]


=======
pytorch reinforce
a Variable containing:
 1
 1
 0
 1
[torch.LongTensor of size 4x1]

params.grad Variable containing:
 0.6170
-1.3288
 0.7117
[torch.FloatTensor of size 3x1]
<ECODE> (Its actually almost the same amount of code in fact?)
How to simpler downsample an image tensor with bicubic?
Category: vision

orashi: Trying to downsample a batch of normalized image tensor but failed to get it work with transforms.Scale or PIL’s resize. Finally get it worked by : Is there a better way to do this?

smth: If not, yes this seems appropriate.

IanYeung: What about nn.AdaptiveAvgPool2d?

jia_lee:
Cannot Unsqueeze Empty Tensor

ycszen: When I load my data, I meet a problem: <SCODE>Traceback (most recent call last):
  File "trainer.py", line 41, in <module>
    for i, (rmap, label) in enumerate(trainloader):
  File "/usr/local/lib/python2.7/dist-packages/torch/utils/data/dataloader.py", line 174, in __next__
    return self._process_next_batch(batch)
  File "/usr/local/lib/python2.7/dist-packages/torch/utils/data/dataloader.py", line 198, in _process_next_batch
    raise batch.exc_type(batch.exc_msg)
RuntimeError: Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/torch/utils/data/dataloader.py", line 32, in _worker_loop
    samples = collate_fn([dataset[i] for i in batch_indices])
  File "/usr/local/lib/python2.7/dist-packages/torch/utils/data/dataloader.py", line 81, in default_collate
    return [default_collate(samples) for samples in transposed]
  File "/usr/local/lib/python2.7/dist-packages/torch/utils/data/dataloader.py", line 68, in default_collate
    return torch.stack(batch, 0)
  File "/usr/local/lib/python2.7/dist-packages/torch/functional.py", line 56, in stack
    return torch.cat(list(t.unsqueeze(dim) for t in sequence), dim)
  File "/usr/local/lib/python2.7/dist-packages/torch/functional.py", line 56, in <genexpr>
    return torch.cat(list(t.unsqueeze(dim) for t in sequence), dim)
RuntimeError: cannot unsqueeze empty tensor at /data/users/soumith/builder/wheel/pytorch-src/torch/lib/TH/generic/THTensor.c:530
<ECODE>

smth: you need to give more context

vmirly1: I also came across the same issue when trying to build a custom dataset: <SCODE>    def __getitem__(self, index):
            fname = self.train_filenames[index]
            image = Image.open(os.path.join(celeba_imgpath, fname))
            label = self.train_labels[index]
            return self.transform(image), torch.FloatTensor(label)

<ECODE> I think it was caused by the label given as a scalar value. I converted the label to a list, and then the problem was solved. Here is the corrected code: <SCODE>    def __getitem__(self, index):
            fname = self.train_filenames[index]
            image = Image.open(os.path.join(celeba_imgpath, fname))
            label = [self.train_labels[index]]
            return self.transform(image), torch.FloatTensor(label)
<ECODE>

Anshul_Paigwar: I had the same issue this solution worked for me. Thanks

sngth: I also had the same issue and is resolved with your solution. This saved a lot of time! Thanks
GPU slower than CPU on a simple RNN test code

aleingrosso: Hi, I wanted to write an RNN from scratch using the pytorch cuda capabilities and I ran some preliminary tests to compare the speed of the CPU vs GPU. The task is very simple and consists of a for loop mimicking the update of the internal state x in an RNN with recurrent weight matrix J. I’m using a Quadro K620 with cuda 8.0. When the size of x is N=1000 there seems to be a trade-off, with the GPU implementation consistently getting slower when the number of iterations increases (I ran some other tests with different sizes of the J matrix and this behaviour seems pretty systematic). This is an example of running times I get when running the enclosed script: <SCODE>cpu: [0.010117292404174805, 0.058980703353881836, 0.45785975456237793, 4.512230634689331]
gpu: [0.0019445419311523438, 0.05474495887756348, 0.7503962516784668, 7.011191129684448]
<ECODE> I’d really appreciate some help on this. Thanks in advance. The test script is the following: <SCODE>import numpy as np
import torch as tr
import math
import time

GPUID = 0
tr.cuda.set_device(GPUID)

N = 1000

J = tr.randn(N,N)
x = tr.randn(N)
r = tr.randn(N)
y = tr.randn(N)

Jn = J.numpy()
xn = x.numpy()
rn = r.numpy()
yn = y.numpy()

cputimes = []
for sampl in (100, 1000, 10000, 100000):
    start = time.time()
    for i in xrange(sampl):
        rn = np.tanh(xn)
        xn = Jn.dot(xn);
    end = time.time()
    cputimes.append(end-start)
print(cputimes)

Jc = J.cuda()
xc = x.cuda()
rc = r.cuda()
yc = y.cuda()

gputimes = []
for sampl in (100, 1000, 10000, 100000):
    start = time.time()
    for i in xrange(sampl):
        rc = tr.tanh(xc)
        xc = Jc.mv(xc);
    end = time.time()
    gputimes.append(end-start)
print(gputimes)
<ECODE>

wasiahmad: I experienced the same thing. A very simple network runs faster in CPU than GPU.

gaojun4ever: Can anyone help? I experienced the same thing.

hughperkins: Its totally normal that rnns run fast on cpus, compared to gpus. they send lots of teeny-tiny kernel launches, and the gpu cores spend all their time waiting for the next batch to arrive…

hughperkins: By the way, you’ll get faster per-example times by increasing the batch size, at the expense that effective learning rate will probably decrease.
Is it possible to keep intermediate results of forward for backward?

tom: Hello, I would like to implement a new torch.autograd.Function where the gradient is closely related to an intermediate result. Is there a way to store the intermediate result for the backward pass to avoid having to compute it again? (Similar to save_for_backward, but that explicitly isn’t it…) Thank you Thomas

smth: 

tom: Thanks! <SCODE>RuntimeError: save_for_backward can only save input or output tensors, but argument 2 doesn't satisfy this condition
<ECODE>

mxh000: The autograd system is designed to handle intermediate results, but only if they were formulated using torch.autograd.Variable() and torch tensor functions, afaik. The error is because one of the arguments is not a Variable.

lenscloth: The question is bit outdated, I encountered the same situation <SCODE>def forward(self, a, b, c):
     # okay
     # self.save_for_backward(a) 
              
     # wrong!!
     self.save_for_backward(a+1)
<ECODE> RuntimeError: save_for_backward can only save input or output tensors, but argument 2 doesn’t satisfy this condition

tom: Best regards Thomas

SivanK: 

arogozhnikov: I have the same question (I need to store info for backward). Unfortunately,  I don’t see where this requirement (store only input or output) comes from and how do I get around it (storing as a member in Function means you can use this function only once). Any ideas how to store intermediate things?

tom: I see two main options. Best regards Thomas <SCODE>class MyReLU(torch.autograd.Function):
    @staticmethod
    def forward(ctx, input):
        ctx.save_for_backward(input)
        return input.clamp(min=0)

    @staticmethod
    def backward(ctx, grad_output):
        input, = ctx.saved_variables
        grad_input = grad_output.clone()
        grad_input[input < 0] = 0
        return grad_input

<ECODE>

arogozhnikov: Ok, probably it is a way to go, where do I find an example with instantiating function? Values that I need to store are actually integer numbers that occur during forward pass, so I can’t follow this recipe.

arogozhnikov:
Erf and erfinv functions in PyTorch

Soniya: I have a torch.FloatTensor say A of size u,v,x,y. I have to calculate its erf and erfinv. Is there any way to do this in PyTorch. Please help me.

smth: What is Erf (and what is it’s inverse)? Is this an abbreviation?

Soniya: 

hadware: I’m actually interested by this as well, could you give us some updates if you find a solution?

AjayTalati: <SCODE>import torch
from scipy import special
## reference - https://docs.scipy.org/doc/scipy-0.14.0/reference/generated/scipy.special.erf.html

u = 1
v = 2
x = 3
y = 4

A = torch.randn(u,v,x,y)

special.erf( A.numpy() )
special.erfinv( A.numpy())
<ECODE>

Soniya: 

cerisara: Nope, it does not work, because I also need the error function but as an autograd function; I mean, I would like to insert this function in the computation graph and pytorch to automatically differentiate it ! If the function is not supported for now in pytorch, may I define my own custom autograd function ?

AjayTalati: 

tom: <SCODE>import torch
a_for_erf = 8.0/(3.0*numpy.pi)*(numpy.pi-3.0)/(4.0-numpy.pi)
def erf_approx(x):
    return torch.sign(x)*torch.sqrt(1-torch.exp(-x*x*(4/numpy.pi+a_for_erf*x*x)/(1+a_for_erf*x*x)))
def erfinv_approx(x):
    b = -2/(numpy.pi*a_for_erf)-torch.log(1-x*x)/2
    return torch.sign(x)*torch.sqrt(b+torch.sqrt(b*b-torch.log(1-x*x)/a_for_erf))<ECODE> To get an impression of how they look, you could plot it against the scipy.special functions like <SCODE>from matplotlib import pyplot
%matplotlib inline
import scipy.special
x = numpy.linspace(-2,2,100)
pyplot.subplot(1,2,1)
pyplot.title('erf')
pyplot.plot(x,erf_approx(torch.from_numpy(x)).numpy(), label="approx")
pyplot.plot(x,scipy.special.erf(x),'--', label="scipy")
pyplot.legend()
pyplot.subplot(1,2,2)
y = scipy.special.erf(x)
pyplot.title('erfinv')
pyplot.plot(y,erfinv_approx(torch.from_numpy(y)).numpy(), label="approx")
pyplot.plot(y,scipy.special.erfinv(y),'--', label="scipy")
pyplot.legend()<ECODE> (the %matplotlib is for jupyter). Best regards Thomas

AjayTalati: nice work !!! I found a very simple implementation of the sinkhorn-knopp matrix normalisation, we were looking at a while ago. It’s in MATLAB, but it’s very understandable, All the best, Aj

cerisara: Great ! Thank you, very useful !

zhangxiangxiao: One use case that I needed was to sample from a Gaussian VAE and do interpolation in uniform distributed variables. That is because u = erfinv(v) follows normal distribution if v follows uniform distribution via a technique called inverse sampling.

ira: 

tom: Awesome, thank you! Best regards Thomas

MAsad:
PyTorch vs Theano GPU RAM usage issue

vladimir: Hello, I’ve noticed that PyTorch (torch (0.1.10.post2), torchvision (0.1.7)) uses significantly more GPU RAM than e.g. Theano running a similar code. Unfortunately I cannot disclose the actual code but I think it should be possible to reproduce this behavior with the following simple sequential architecture (all dimensions are in form MINIBATCH_LENGTH x NUM_CHANNELS x H x W): N - minibatch size, e.g. 20, padding mode == “same” everywhere Main loop: <SCODE>net = Net()
net.cuda()
net.train()

optimizer = optim.SGD(net.parameters(), lr = ...)
criterion = nn.MSELoss()

input = Variable(torch.from_numpy(<your-Nx10x1024x1024-tensor>).cuda())
target = Variable(torch.from_numpy(<your-Nx10x1024x1024-tensor>).cuda())

for epoch in xrange(...):

    output = net(input)
    loss = criterion(output, target)
    
    net.zero_grad()
    loss.backward()
    optimizer.step()
<ECODE> On a GTX-1060 corresponding code takes around 30% more GPU RAM than its Theano counterpart (exection times are about the same for Theano and PyTorch versions). Is this something that can be fixed? Thanks,

smth: This might be one reason to explain the memory difference.

vladimir: thanks for your reply. This seems to be the case indeed (the maximum mini-batch size seems to be quite similar for both Theano and PyTorch). However, is there a way to constrain memory allocation? In Theano there’s lib.cnmem which can be assigned % of memory to pre-allocate. It’s a soft limit but still helpful when sharing GPU between multiple jobs.

smth: there is no way to constrain the memory to an upper bound right now.
4D Tensor indexing with `gather()`

zuoxingdong: Currently I would like to perform this numpy equivalent command <SCODE>a = np.random.randn(5, 4, 3, 3)
a[range(5), :, [0, 1, 1, 2, 0], [1, 2, 0, 1, 0]]
<ECODE>

fmassa:
Generic question about batch sizes

FuriouslyCurious: I was playing around with the MNIST example, and noticed the doing TOO big batches (10,000 images per batch) seem to be hurting accuracy scores. This got me thinking : What are the general rules of thumb you guys have found when it comes to batch sizes? Also, what is your favorite optimizer and activation function? Thanks!

vabh: The batch size is usually set between 64 and 256.

FuriouslyCurious: Thank you for the papers!  I was trying to minimize noise through massive batches.  The papers you provided are a great explanation of why massive batches may not work. Thanks!
LogLoss on Kaggle

FuriouslyCurious: <SCODE>LogLoss= −1/n∑i=1 to n [yi log(ŷ i) + (1−yi) log(1−ŷ i)],

where:
  n   : is the number of patients in the test set
  ŷi  : is the predicted probability of the image belonging to a patient with cancer
  yi  : is 1 if the diagnosis is cancer, 0 otherwise
<ECODE> <SCODE>torch.nn MLSM Loss:
loss(x, y) = - sum_i (y[i] log( exp(x[i]) / (1 + exp(x[i])))
                      + (1-y[i]) log(1/(1+exp(x[i])))) / x:nElement()
<ECODE> Thanks!

FuriouslyCurious: Hi Torchies, Bumping an old question… if someone has a paper / presentation comparing different loss functions and their effects, it will really help me out. Thanks!