foldl/torch-forum huggingface dataset:
How to assign values to tensor based on index array efficiently?

ypxie: <SCODE>tf.TensorArray.scatter(indices, value, name=None)

Scatter the values of a Tensor in specific indices of a TensorArray.

Args:
    indices: A 1-D Tensor taking values in [0, max_value). If the TensorArray is not dynamic, max_value=size().
    value: (N+1)-D. Tensor of type dtype. The Tensor to unpack.
name: A name for the operation (optional).

Returns:
    A new TensorArray object with flow that ensures the scatter occurs. Use this object all for subsequent operations.

Raises:

ValueError: if the shape inference fails.
<ECODE>

apaszke: 

Sun_Mingjie:
Non-determinisic results

supakjk: I run the follow code before definition of modules. (My model uses Embeding, Dropout, LSTM, and Linear layers.) <SCODE>torch.manual_seed(1000)
torch.backends.cudnn.enabled = False
torch.cuda.manual_seed(1000)
<ECODE> However, the final results are still different for each trial (regardless I use CPU or GPU). Are there any other things I should do to get the deterministic results given the same input? Thanks.

apaszke: 

apaszke: 

supakjk: 

fmassa: 

supakjk: 

fmassa: 

edgarriba: 

apaszke: 

edgarriba: 

apaszke: 

edgarriba: 

Samrat_Hasan: I am trying to understand the role of seeding in pytorch. For example, if I have a model trained with a specific seed, can I say, it will produce the same output for a specific input? While with no seed, its not guaranteed to produce the same output? One thing is also bothering me that if I train without setting any seed, why I would get different output for the same input given that there is no randomness associated with my model?

Houjing_Huang:
Format of weight parameters in KLDivLoss

mromaniuk: KLDivLoss can take a weight parameter but the docs don’t specify how it should be formatted. What should the format be?

apaszke: It should be a 1D tensor having as many elements as you have classes. We’ll have to add that to the docs, thanks for reporting that!

nofreewill: Can it? I can see nothing weight related in the docs, nor in the source.
Put a new top on ResNet

telesphore: I’m trying to replace the last layer of an imagenet trained model. <SCODE>model_imagenet = models.resnet18(pretrained=True)
model_imagenet

class ResNetNewTop(nn.Module):
    def __init__(self, old_model, num_classes=2):
        super(ResNetNewTop, self).__init__()
        self.bottom = nn.Sequential(*list(old_model.children())[:-1])
        self.top = nn.Linear(512, num_classes)  # TODO: input size (512) from bottom

    def forward(self, x):
        x = self.bottom(x)
        x = self.top(x)  # <<< Errors here
        return x

model = ResNetNewTop(model_imagenet, num_classes=2)
model.cuda()
model
<ECODE> This is throwing RuntimeError: matrix and matrix expected at [snip] /torch/lib/THC/generic/THCTensorMathBlas.cu:235 Can I put Sequential() into a new model directly? Or do I have to loop through the layers and build up the model that way?

apaszke: Additionally, in case of ResNet, you could just replace the final layer with a new one and it should work I think: <SCODE>model_imagenet.fc = nn.Linear(512, num_classes)
<ECODE>

avijit_dasgupta: I do this: <SCODE>model = models.__dict__[opt.arch](pretrained=True) #pretrained=False if you don't want to use pre-trained weights.
model.fc = nn.Linear(512, num_classes)<ECODE>

apaszke: 

avijit_dasgupta: Yes. It will only work for ResNets. Your answer is more generalized.

telesphore: Thank you! The diagnosis was spot on. I like the shortcut too; it helped me realize how loosely coupled all of the layers are.

drcege: <SCODE>feat = nn.Sequential(*list(resnet.children())[:-1])
fc1 = nn.Linear(a, b)
fc1 = nn.Linear(b, c)

<ECODE> the layer names in feat are lost.
Pretrained resnet model converted from caffe

ruotianluo: I’m using resnet to do feature extraction. I’m assuming the current resnet provided in model zoo is converted from fb.resnet.torch. Are you planning to convert the caffe model into pytorch version? (From my own experience, it seems the caffe one is better.)

smth: the current model is trained from scratch and matches accuracy to fb.resnet.torch model

ruotianluo: I realize that. The features just don’t work as well as the features extracted from the caffe model.
Construct new tensor on correct device based on input

ypxie: 

apaszke: 

wouter: Hi! I’m sorry to reply to an old post but I thought replying here would be better than starting a new topic. <SCODE>device = 1
X = torch.rand(10).cuda(device)  # or X = torch.rand(10)
<ECODE> How to make code below agnostic to whether X is on CPU/GPU and which device? <SCODE># Fill with numpy data
data = np.array([1, 2, 3])  # For example indices calculated by custom Python procedure
X.new().long().new(data)  # Works on CPU, error on GPU?
X.new().long().new(*data.shape).copy_(torch.from_numpy(data))  # Works but verbose
X.new(data.astype(float)).long()  # Inefficient, lossy?
# X.new(data)  # Error

# Or fill with zeros
X.new(10).long().zero_()  # Works, but inefficient?
X.new().long().new(10).zero_()  # Works, but verbose?<ECODE>

smth: 

wouter: 

bgobbi: What is the new alternative for this use case?

shlomia: 

Noam_Salomonski: This creates very ugly (and slow) code such as <SCODE>    if std.is_cuda:
        eps = torch.FloatTensor(std.size()).cuda().normal_()
    else:
        eps = torch.FloatTensor(std.size()).normal_()
<ECODE> Isn’t there a better way?

Noam_Salomonski:
How to get the class names to class label mapping

avijit_dasgupta: I am using ResNet-18 for classification purpose. I have used dataloader to load the data. But how get the label to class name mapping? Does it load in alphabetical order?

avijit_dasgupta: For future reference: <SCODE>def find_classes(dir):
    classes = os.listdir(dir)
    classes.sort()
    class_to_idx = {classes[i]: i for i in range(len(classes))}
    return classes, class_to_idx<ECODE>

zym1010: 

Ray_Luo: Something like this: <SCODE>dataset = datasets.ImageFolder(data_folder, transform=data_transforms)
print(dataset.class_to_idx)
<ECODE>

vipin14119:
Register_backward_hook on nn.Sequential

supakjk: <SCODE>import torch
import torch.nn as nn
from torch.autograd import Variable

a = nn.Sequential(nn.Linear(5,3), nn.Tanh(), nn.Linear(3,2))

def hookFunc(module, gradInput, gradOutput):
	print(len(gradInput))
	for v in gradInput:
		print v
a.register_backward_hook(hookFunc)

input = Variable(torch.randn(4,5))
output = a(input)

target = torch.FloatTensor(4,2).fill_(1)
output.backward(target)
<ECODE> <SCODE>Variable containing:
-0.5910 -0.7340 -0.4239
-0.5910 -0.7340 -0.4239
[torch.FloatTensor of size 2x3]

Variable containing:
 4
 4
[torch.FloatTensor of size 2]
<ECODE>

apaszke: Yes, you should never modify any arguments given to the hook in-place. If you want to replace grad input, you can do out-of-place operations on it, and return new values from the hook.

supakjk: 

apaszke: Like that: <SCODE>x = Variable(torch.randn(5, 5), requires_grad=True)
y = x + 2
y.register_hook(lambda grad: grad * 2)
y.sum().backward()
x.grad # is now filled with 2
<ECODE>

apaszke: But remember about that container hook problem. Do this only on primitive modules or Variables.

supakjk: Thanks!

apaszke: It works in the same way for modules: <SCODE>module.register_backward_hook(lambda module, grad_i, grad_o: grad_i * -1)
<ECODE>

supakjk: 

Qizhe_Xie: 

supakjk: That approach worked in my case.

WERush: 

dlmacedo: <SCODE>class DDReLU(nn.Module):
    def __init__(self):
        super(DDReLU, self).__init__()
        self.threshold = nn.Parameter(torch.rand(1), requires_grad=True)
        self.register_backward_hook(lambda module, grad_i, grad_o: (grad_i[0], grad_i[1]*0.01))
        #self.threshold.data.fill_(0.1)
        self.ReLU = nn.ReLU(True)

    def forward(self, x):
        print(self.threshold.data[0])
        return self.ReLU(x + self.threshold) - self.threshold
        #return self.ReLU(x) + self.threshold
<ECODE> Is the code above fine to change the relative learning rate of the new parameter? By relative learning rate, I mean: The parameter created has a learning rate that is 0.01 times the one used to the other model’s parameters.
About pytorch update

big_tree: Hi all, Anybody know what this means?

apaszke: Since the model structure is actually defined by the code, we’ve implemented a simple mechanism that saves the source code of each Module that you save, and when it’s loaded it compares it with the source that’s used to reinstantiate it. It’s meant to warn you that the model might now work differently, if you change your model code in the meantime. I didn’t realize it also worked for built in modules, we’ll have to disable that. You can safely ignore the warning, and overwriting the old checkpoint with a new one will make it disappear. Also, we recommend serializing only the state dict.

big_tree: Thanks for your help!
RuntimeError: could not compute gradients for some functions (CudnnRNN)

ponythewhite: Hi, Unfortunately getting the following behavior: <SCODE>  File "/home/jd/pytorch/examples/translation/translate_gpu.py", line 296, in trainEpochs
    loss = train(input_variable, target_variable, encoder, decoder, encoder_optimizer, decoder_optimizer, criterion)

  File "/home/jd/pytorch/examples/translation/translate_gpu.py", line 247, in train
    loss.backward()

  File "/home/jd/anaconda/lib/python2.7/site-packages/torch/autograd/variable.py", line 158, in backward
    self._execution_engine.run_backward((self,), (gradient,), retain_variables)

RuntimeError: could not compute gradients for some functions (CudnnRNN)
<ECODE> It works on the CPU. Any hints as to what might be causing this behavior? The RuntimeError sadly doesn’t mention which operations couldn’t get the gradient computed. Is there a list of supported/unsupported ops I could reference somewhere? Thanks,

apaszke: I don’t think it’s your fault. We’ll have to look into that. Thanks for the report.

ponythewhite: Thanks! Should I create an issue on Github, or will you? Would be great if I could track the status. Best,

apaszke: 

ponythewhite: Awesome, thanks! Would love to help with this, but can’t do much in C++.

apaszke: 

spro: Thank you! I also ran into this but forgot to report it.

spro: Got 0.1.9 hot off the press and I can confirm it works, approx 3.5x speedup on my GPU. Any ideas for making it faster? Is it a bad idea to make new Variables in the training loop?

apaszke: 

apaszke: And no, creating Variables is very very cheap, so you can do it at every step without any problem.

spro: 

apaszke: Yup, that’s been fixed! The fix initially introduced another bug, but that’s fixed as well.

Yingxin_Wu: <SCODE>class Linear(torch.autograd.Function):

    def forward(self, input, weight, bias):
        self.eps = 1e-16
        self.Z = weight.t()[None, :, :] * input[:, :, None]
        self.Zs = self.Z.sum(dim=1, keepdim=True)
        if bias is not None:
            self.Zs += bias[None, None, :]
        return self.Zs.squeeze(dim=1)

    def backward(self,R):
        print('linear')
        return (R[:, None, :] * self.Z / (self.Zs + (2 * (self.Zs >= 0) - 1) * self.eps)).sum(dim=2)

<ECODE> and run <SCODE>model.readout.backward(out)
<ECODE> Unfortunately, it reported as follows <SCODE>Traceback (most recent call last):

  File "G:\Explainer\_Geo_Exp_inter\infectionLRP.py", line 117, in <module>
    print(model.readout.backward(out))

  File "D:\Anaconda3\lib\site-packages\torch\tensor.py", line 195, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)

  File "D:\Anaconda3\lib\site-packages\torch\autograd\__init__.py", line 98, in backward
    allow_unreachable=True)  # allow_unreachable flag

RuntimeError: could not compute gradients for some functions
<ECODE> It has confused me for a few days and I don’t really kown what cause it and how to fix it. Yingxin Wu

albanD: Hi,
Discussion on pyTorch packages: which ones do you use?

Theodore: Hello everyone! This is my first post in the forums. I am a new user, starting my Torch and pyTorch experience and I am very excited to do so! I am still going over documentation, tutorials and such and considering further options. One thing I thought I would ask the more experienced people in the forum concerns packages. There’s quite a few packages in pyTorch (this is one of my favourite aspects of Torch actually!) and I was wondering if people would like to ‘pass the torch’ in a sense sharing their experiences with them. Which package was the best in loading data from a csv file? Which packages do you use more often when dealing with images? Which packages do you use more often when dealing with text and NLP? Which packages do you use for visualization of model performance and understanding of your results? Which packages do you think are missing and could be developed to further enhance pyTorch? Thanks in advance for any comments that might come! See you down the Torch road!
How can I make custom nn.Module with max pooling index?

Seungyoung_Park: I would like to custom nn.Module including max unpooling module. For example, if I declare MyModule1 and 2 which outputs the pooling index beside output signal and  takes the pooling index (pool_idx), respectively, as below. <SCODE>class MyModule1(nn.Module):
    def __init__(self):
    super(MyModule1, self).__init__()
    self.conv = nn.Conv2d(64, 3, 4)
    self.pool = nn.MaxPool2d(3, 3, return_indices = True)

    def foward(self, x):
        x = self.convl(x)
        x, pool_idx = self.pool(x)
        return x, pool_idx

class MyModule2(nn.Module):
    def __init__(self, pool_idx):
    super(MyModule, self).__init__()
    self.pool_idx = pool_idx
    self.conv = nn.ConvTransposed2d(3, 64, 4)
    self.unpool = nn.MaxUnpool2d(3,3)

    def forward(self, x):
        x = self.unpool(x, self.pool_idx)
        x = self.conv(x)
        return x
<ECODE> For example, <SCODE>class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.m1 = MyModule1()
        self.m2 = MyModule2()

    def forward(self, x):
        x, pool_idx1, pool_idx2 = self.m1(x)
        x = self.m2(x, pool_idx1, pool_idx2)
        return x
<ECODE>

apaszke: You shouldn’t save the pooling index in the constructor. This should work: <SCODE>class MyModule1(nn.Module):
    def __init__(self):
    super(MyModule1, self).__init__()
    self.conv = nn.Conv2d(64, 3, 4)
    self.pool = nn.MaxPool2d(3, 3, return_indices = True)

    def foward(self, x):
        x = self.convl(x)
        x, pool_idx = self.pool(x)
        return x, pool_idx

class MyModule2(nn.Module):
    def __init__(self):
    super(MyModule2, self).__init__()
    self.conv = nn.ConvTransposed2d(3, 64, 4)
    self.unpool = nn.MaxUnpool2d(3,3)

    def forward(self, x, pool_idx):
        x = self.unpool(x, pool_idx)
        x = self.conv(x)
        return x

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.m1 = MyModule1()
        self.m2 = MyModule2()

    def forward(self, x):
        x, pool_idx = self.m1(x)
        x = self.m2(x, pool_idx)
        return x
<ECODE>

Seungyoung_Park:
Some confusions about nn.Linear that bothered me ;-(

hbkunn: Linear layer requires 2D tensor. Does it mean that I got to use ‘view’ function to reshape the previous output every time I call it? But for some experiments BN and RELU are both needed afterwards. They both require 4D, for which I need to call reshape function again and again… and I can’t find one corresponding reshape function for nn.Sequence()? Is there any plan to add it?

Atcold: 

apaszke: 

hbkunn: thank you for replying me! I’m so sorry for not noticing BatchNorm1d in the document. Now my confusion is perfectly solved~

hbkunn: that’s reasonable enough~ Thanks for your replying!
Prefix parameter names in saved model if trained by multi-GPU?

ming: Hi, While doing inference I only use one GPU so the model failed to load the latter model file because the parameter names are not matching. I am wondering why parameter names are prepend the prefix? Can I trim the prefix and still use the model?

apaszke: Yes, you can just remove the prefix: <SCODE>state_dict = {k.partition('model.')[2]: v for k,v in state_dict}
<ECODE>

ming: Thanks for the reply! Removing prefix works. Curious why the prefix is needed? It creates inconvenience when we want to resume training a single-GPU-trained-model with multi-GPU, or pass a multi-GPU trained model to inference code that only uses one GPU? Also it looks the pretrained resnet doesn’t have ‘module.’ prefix, does it mean that they were trained on single GPU?

apaszke: No, they probably had the prefixes trimmed before serialization.

lunasdejavu: <SCODE>state_dict =checkpoint['model_dict']
state_dict = {k.partition('model.')[2]: v for k,v in state_dict}
model.load_state_dict(state_dict)
<ECODE>

Lynx_Commando: {k.partition(‘module.’)[2]:state_dict[k]  for k in state_dict.keys()}

pepeportolo:
Trainable Variable in Loss Function and Matrix Multiplication

melgor: Hi, could you check the example with trainable-variable? I was thinking about two cases: implementing a Linear layer without torch.nn. implementing the Loss layer with learable parameter (ex. Center Loss where mean-center is trainable parameters). Such stuff are easy in TesnotFlow, but I’m not sure if I coded it correctly in PyTorch Could you check if the idea of using PyTorch here is correct? Or I should change sth here.

fmassa: Hi! There are some issues with your first script: you need to define linear1 and linear2 as nn.Parameter instead of Variable, if not model.parameters() won’t see them just for information, nn.Linear implements it as torch.mm(input, weight.t() 61, as in torch7 BTW you might want to divide the loss by the batch size, that’s what’s the default in pytorch

waitwaitforget: I’ve implemented the same loss, however, the centers learned are nearly the same. I don’t know where is the problem.

jxgu1016: 

CKboss: A more easily implenment of center loss. <SCODE>class CenterLoss(nn.Module):

    def __init__(self,center_num,feature_dim):
        super(CenterLoss,self).__init__()
        self.center_num = center_num
        self.feature_dim = feature_dim
        self.center_features = nn.Parameter(torch.Tensor(self.center_num,self.feature_dim))
        nn.init.normal(self.center_features.data,mean=0,std=0.1)

    def forward(self,x,label):
        B = x.size()[0]
        center = torch.index_select(self.center_features,0,label)
        diff = x-center
        loss = diff.pow(2).sum() / B
        return loss
<ECODE>

lelegu: I had the same issue. My parameters learned are the same. Did you solve it and how? Thanks!
Select specific columns of each row in a torch Tensor

Kalamaya: There’s probably a simple way to do this, but owing to my noobness I do not know who to do this in PyTorch. Basically, let’s say I have a torch tensor like so: m = Variable(torch.randn(4,2)) Furthermore, I have a bunch of indices, given by inds, where inds is: <SCODE>inds
Variable containing
 1
 1
 0
 0
 [torch.LongTensor of size 4]
<ECODE> How can I do that? Thanks!

fmassa: <SCODE>m = torch.randn(4,2)
ids = torch.Tensor([1,1,0,0]).long()
print(m.gather(1, ids.view(-1,1)))
<ECODE>

Kalamaya: Thank you that’s exactly what I needed. Why cant we use normal slicing as in numpy?

fmassa: 

apaszke: 

fmassa: 

apaszke: Not really, this would require passing in a 2D list.

magnus_w: 

Qiqin_Dai: A follow up question: is the columns selection in Pytorch differentiable? Say the column index is obtained though a network, and the loss is computed based on the selected columns from the matrix. Will the back propagation go through?

Jan_Konopka: Yes indexing is fully differentiable in PyTorch! Obviously this only back-propagates into the indexed elements with the other elements receiving zero gradient. This is necessary with some techniques such as Policy Gradients however.

Enamex: How would you do the same when assigning to an existing tensor? Example: <SCODE>tensor.gather(-1, indices) = new_values_tensor
<ECODE>

usman095:
Convert to numpy cuda variable

pronics2004: How to convert cuda variables to numpy?

fmassa: <SCODE>cuda_tensor = torch.rand(5).cuda()
np_array = cuda_tensor.cpu().numpy()
<ECODE>

apaszke: 

Adel_Saleh1: <SCODE>import cv2
img=cv2.imread('xx.jpg')
img=cv2.resize(img,(32,32))
img = img.transpose((2,0,1))
img=np.expand_dims(img,axis=0)
img=img/255.0
img=torch.FloatTensor(img)
img=Variable(img)
img = img.cuda()

 output=net(img)

print(output)
print('---------------------------------------------------')

np_array = output.cpu()
print (np_array)
print('---------------------------------------------------')

np_array = np_array.numpy()
print (np_array)
print('---------------------------------------------------')
<ECODE> <SCODE>np_array = np_array.numpy()


AttributeError: numpy<ECODE>

spruceb: 

Adel_Saleh1: Thanks … fixed that

ahmedmazari: Hello, what’s wrong with my model <SCODE>m=torch.nn.Softmax()
model.eval()
preds = model(image)
temps=preds.cpu()
prob=torch.max(m(temps)*100)
<ECODE>

pvskand: 

dhpollack: Is there a reason that the numpy() call doesn’t check if the tensor is on the gpu and simply do .cpu().numpy()?

smth: 

Mona_Jalal: In my case I also had to use .detach() <SCODE>
output.cpu().detach().numpy()
<ECODE>

Golnoush123: <SCODE>loss_semi_adv_value += loss_semi_adv.data.cpu().numpy()[0]/args.lambda_semi_adv
IndexError: too many indices for array
<ECODE>
CUDA memory continuously increases when net(images) called in every iteration

Kalamaya: 

ruotianluo: This is because pytorch will build a the graph again and again, and all the intermediate states will be stored.

apaszke: 

Kalamaya: I think that did it! Ugh! I spent two days on this! >< Thanks though… so what was going on here?

Kalamaya: <SCODE>for i in xrange(100):
    out = net(input) 
    loss = someLossFunction(out)
<ECODE>

ruotianluo: The graph won’t be re-made. The 100 loops basically create 100 graphs. The 100 graphs share the same input and parameters, but all the intermediate variables of 100 graphs (although they could be the same) are saved separately.

Kalamaya: 

smth: <SCODE>for i in xrange(100):
    out = net(input)
    loss += someLossFunction(input) # BAD, because it keeps continuing the graph over the for-loop

    loss = someLossFunction(input)   # this is fine

    loss = someLossFunction(input)
    total_loss += loss.data[0]              # this is fine
<ECODE>

Kalamaya: Im just asking for the sake of understanding: Is what is happening, let me put it differently: Are you saying that this statement here, will make two graphs that are identical to each other? loss = someLossFunction(input1) + someLossFunction(input2) Is my conclusion correct?

smth: yes, your conclusion is correct.

spro: To clarify for RNN users that might come across this, we do actually want to keep the graph around for backprop-through-time. (Right? Or is there a better way?)

smth: 

heihei: This should be written in the tutorial!

hmishfaq: 

11112: hi, can the total_loss here be back propagated?

Felix_Lessange: this solved my problem

ddeng: This solved my problem too. I didn’t know call loss instead of loss.data[0] will cause rewriting the graph.

zhaoethz: This is really important. Everyone learning PyTorch should know this at the first place.

Albert_Christianto:
Elegant way to transpose a variable

meijieru: For example, A Tensor with shape [2, 3, 4] -> [4, 2, 3] In numpy, we could directly use np.transpose(Tensor, [2, 0, 1]). However, in pytorch, I could not find a elegant way to do it. Thanks for your help.

ruotianluo: torch.Tensor.permute…

meijieru: