foldl/torch-forum huggingface dataset:
A Workaround for HalfTensor lacking Stateless Methods

ajbrock: Hey guys, EDIT: SEE BELOW I’m working on using FP16 in DenseNets where concatenation is a key part of the architecture, but since HalfTensors don’t support stateless methods I can’t call cat(). I’ve thrown a workaround in by instantiating a temporary variable to store the concatenated results and then just dumping to it by indexing, but then I get an error calling backwards() that looks like it comes from the final layer: Is there an easy way to work around this, or is training in half precision not supported yet? I’m on CUDA8.0, cuDNN5105, Ubuntu14.04, pytorch1.9 (the latest version that you get through conda install) and a Maxwell titan X. I’m in a situation where memory costs are more important than speed so I can eat the slowdown of FP16 if it reduces memory useage. Aside, I get an error when calling torch.backends.cudnn.version() if I don’t call torch.backends.cudnn._loadlib() first (the “cuDNN not initialized” error). cuDNN still works, but when trying to check if a source build has succeeded this can be confusing. EDIT: Nevermind, it looks like nothing else other than F.linear has this issue, so switching over to the state’d version (three edits) fixed this and is now running smoothly and quickly in FP16. Dongers up for pytorch, woo! Probably would still make sense to add in a stateless cat() method, though, I suspect the way I’m doing it is inefficient. Thanks, Andy

smth: if you could give a repro for the F.linear issue, we’re happy to fix this asap. Open an issue on pytorch/pytorch with the repro.

ajbrock: Any ideas on if there’s a better workaround for the cat() issue would also be appreciated (I’m guessing the reason tensors don’t have a tensor.cat() function attached to them since it would change their shape/memory size?)

apaszke: It’s hard to say what would be a good choice of semantics for tensor.cat() (should it modify the tensor? should it be included in the list?), so we’d rather stick to the stateless one. If the method is implemented in THC, then exposing it is a matter of changing one line in our C wrappers. Otherwise, it will require some changes in the backends.
Documentation for reinforce()
Category: reinforcement-learning

cartage: Hi all, I’m having trouble understanding how and when to use a.reinforce(), despite the examples. In particular, why do I need it if I want to implement REINFORCE ? Thanks for your help

apaszke: 

apaszke: You could implement REINFORCE manually, it’s just a convenient way of doing that.

cartage: Thanks, that’s just what I needed !
Anyone tried the test code?

cosmmb: Hi there, I was trying to test if my pytorch is installed correctly on a CentOS-based server. I found there are a set of programs under test dir could do the job. But when I try to run run_test.sh, it always gives me the error message like: Anyone know what is going on here? I also tried on my mac, and everything works fine. Thanks!

smth: what is the version of python on the machine, and how did you install pytorch (wheel or conda)?

cosmmb: I use python 2.7 for both mac and centos. They are all installed with conda.

smth: 

cosmmb: thanks!

smth: ohhh, i think i know what’s happening. <SCODE>git clone https://github.com/pytorch/pytorch
cd pytorch
git checkout v0.1.9
cd test
./run_test.sh
<ECODE>

cosmmb: Thanks for your help! Now the error is gone.

dl4daniel: <SCODE>ERROR: test_serialization_map_location (__main__.TestTorch)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_torch.py", line 2713, in test_serialization_map_location
    tensor = torch.load(test_file_path, map_location=map_location)
  File "/Users/Natsume/miniconda2/envs/dlnd-tf-lab/lib/python3.5/site-packages/torch/serialization.py", line 222, in load
    return _load(f, map_location, pickle_module)
  File "/Users/Natsume/miniconda2/envs/dlnd-tf-lab/lib/python3.5/site-packages/torch/serialization.py", line 370, in _load
    result = unpickler.load()
AttributeError: Can't get attribute '_rebuild_tensor' on <module 'torch._utils' from '/Users/Natsume/miniconda2/envs/dlnd-tf-lab/lib/python3.5/site-packages/torch/_utils.py'>

<ECODE> <SCODE>pytorch                   0.1.10                   py35_1    soumith
<ECODE> How should I solve this error? Thanks!

smth: you are trying to install pytorch via binaries, but run tests for master. See my comment above.

dl4daniel: The error is gone with the following method: I don’t why but there is no error this time. Thanks

Royi: What if I installed PyTorch using Anaconda (Anaconda Navigator), how can I test the installation?
How can I get the loss of individual data

Seungyoung_Park: As long as I understand, the loss function can provide the mean loss of batch data or the loss sum of batch data. Currently, to get the individual loss at the evaluation mode, I use the batch size of one. Is there any way to get the loss of each data from the batch while the batch size of larger than one is employed ? Also, why is the loss value of the same input data different at each trial ? At the evaluation mode, are not the parameters fixed ?

Response777: You can write your own loss function instead of using the built-in one, then you can do every thing you want inside such function.

apaszke: 

Rinku_Jadhav2014:
What’s the difference between `torch.nn.functional` and `torch.nn`?

Response777: It seems that there are quite a few similar function in these two modules. What I want to know is if there were any other further difference, say, the efficiency?

jekbradbury: 

Cysu: 

Xixi: Correct me if I’m wrong. Seems like when using nn.functional   net.parameters() won’t find the parameters/weight. you need to specify them explicity

klory: I think this is correct, using nn.functional is lower-level. 1: you can’t benefit from the nn.sequential, so we have to manually define the parameters. However, you could still use the torch.optim to update the parameters in training.

govind_narasimman: I beleive both input and kernel are trainable. Can U explain what is extra in torch.nn or more inherited?? thanks
How to preprocess input for pre trained networks?
Category: vision

tsterin: Thank you very much

smth: 

tsterin: Thank you very much!

qianguih: Hi, it looks like the pixel intensities have been rescaled to [0 1] before normalization. It that right?

smth: 

qianguih: I see. Thank you very much!

ecolss: This is important information, I wonder it’s not put in the doc but in the example code?

mehdi-shiba: Agreed. If it wasn’t for this thread, I would have missed this important Normalization step for sure. It would be nice if it could be added to the documentation.

mattmacy: This is pretty key information. Without doing this, and only doing mean centering and stddev normalization of the original Hunsfield units, I need to keep batch normalization enabled during test to see reasonable results from my volumetric segmentation network. This should really be in bold somewhere.

youkaichao: 

Sphinxs: 

Deeply: 

Sphinxs: 

Deeply: Or, it is OK to use list sequences as shown in your case?

Sphinxs: 

Deeply: 

e3312f50f3ba76f35a60: If I use the pretrained model on ImageNet and fine-tune it on my own dataset, should I re-calculate the mean and std with my own dataset?

jdhao: Using the mean and std on ImageNet is pretty standard practice.  Since the mean and std are calculated using a million of images, the statistics is pretty stable. Also the pretrained model is trained using the mean and std in ImageNet. I do not recommend changing the mean and std to that on your small dataset.

r0mer0m: Thank you for your comment. I’ve been having that doubt a while ago. So basically what I infer from the comments as a summary is that the best practice is to leverage: in our specific task and dataset to check what’s the best option  to normalise.

Nikronic:
Variable(xxx).cuda() and Variable(xxx.cuda()), memory leak?

FrancoisFleuret: Hi, I have been hunting a memory leak on GPU for a few days, and there seems to be a pytorch issue. In the code bellow, I define a dummy Function that does nothing, and just forward through it a large tensor. Depending on .cuda() being inside or outside Variable(), there may or may not be a leak. Did I miss something? <SCODE>import torch

from torch import Tensor
from torch.autograd import Variable
from torch.autograd import Function

######################################################################
# That's not pretty
import os
import re
def cuda_memory():
    f = os.popen('nvidia-smi -q')
    fb_total, fb_used = -1, -1
    for line in f:
        if re.match('^ *FB Memory Usage', line):
            fb_total = int(re.search(': ([0-9]*) MiB', f.readline()).group(1))
            fb_used = int(re.search(': ([0-9]*) MiB', f.readline()).group(1))
    return fb_total, fb_used
######################################################################
class Blah(Function):
    def forward(self, input):
        return input
######################################################################

blah = Blah()

for k in range(0, 10):
    x = Variable(Tensor(10000, 200).normal_()).cuda()
    y = blah(x)
    fb_total, fb_used = cuda_memory()
    print(k, fb_used, '/', fb_total)

for k in range(0, 10):
    x = Variable(Tensor(10000, 200).cuda().normal_())
    y = blah(x)
    fb_total, fb_used = cuda_memory()
    print(k, fb_used, '/', fb_total)
<ECODE> prints: <SCODE>0 257 / 8113
1 265 / 8113
2 265 / 8113
3 265 / 8113
4 265 / 8113
5 265 / 8113
6 265 / 8113
7 265 / 8113
8 265 / 8113
9 265 / 8113
0 267 / 8113
1 267 / 8113
2 275 / 8113
3 283 / 8113
4 291 / 8113
5 299 / 8113
6 307 / 8113
7 315 / 8113
8 323 / 8113
9 331 / 8113<ECODE>

Response777: 

FrancoisFleuret: It is fixed in the git version.

Response777: Thanks for reminding.

apaszke: I’ll look into that. Thanks for the report.

apaszke: This should be now fixed in master. It was a reference cycle, so it wouldn’t get freed until the Python’s GC kicked in.
How could I build network structure equivelant to old nn.Concat or nn.Parallel?

Varg_Nord: 

Cysu: <SCODE>class InceptionBlock(nn.Module):
    def __init__(self, num_in, num_out):
        super(InceptionBlock, self).__init__()
        self.branches = [
            nn.Sequential(
                nn.Conv2d(num_in, num_out, kernel_size=1),
                nn.ReLU()),
            nn.Sequential(
                nn.Conv2d(num_in, num_out, kernel_size=1),
                nn.ReLU(),
                nn.Conv2d(num_out, num_out, kernel_size=3, padding=1),
                nn.ReLU()),
            ...
        ]
        # **EDIT**: need to call add_module
        for i, branch in enumerate(self.branches):
            self.add_module(str(i), branch)

    def forward(self, x):
        # Concatenate branch results along channels
        return torch.cat([b(x) for b in self.branches], 1)
<ECODE>

apaszke: 

Cysu: 

apaszke: 

Cysu: Oh. That’s perfect! Didn’t aware of this before. Thanks very much.
Directly getting gradients

yoonholee: Is there a way for me to directly compute the gradient of a variable w.r.t. another variable, like tf.gradients()?

Tudor_Berariu: <SCODE>import torch
from torch.autograd import Variable

x = Variable(torch.ones(10), requires_grad=True)
y = x * Variable(torch.linspace(1, 10, 10), requires_grad=False)
y.backward(torch.ones(10))
print(x.grad)
<ECODE> produces <SCODE>Variable containing:
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
[torch.FloatTensor of size 10]<ECODE>

yoonholee: As far as I can tell, there’s no way of accomplishing this using y.backward(). That’s why I was wondering if there was a standalone function that differentiated.

alexis-jacq: 

smth: we are working on enabling this. This will be in the next major pytorch release 2 months away. It wont be a full 2nd derivative but it will be the Hessian-vector product (whether it’s tf.gradients or in pytorch). We are tracking this feature as double-backprop / double backwards.

vsahil: 

BruceShakeham: Dear PyTorch dev,

yf225: 

albanD: Yes it will. Hopefully this will be done in the next weeks !

amitoz: Is this issue has be solved?

albanD: 

rodsveiga: Is it working? I tried that with v1.6. Same message with older versions.

albanD: Hi,
VGG: Exploding RAM with simple code, bug?
Category: vision

tsterin: 

fmassa: Hi, <SCODE>l = []
for i in range(10):
    out = vgg19(Variable(torch.randn([1,3,224,224])))
    l.append(out.data)
<ECODE>

tsterin: 

fmassa: <SCODE>Variable(torch.randn(1,3,22,224), volatile=True)
<ECODE>

tsterin: Thank you very much for your explanation!
Serializing HalfTensor Nets?

ajbrock: Digging through the issues and I can’t find anything on this particular instance: I’m trying to serialize a network that I’ve moved to the GPU and cast to HalfTensor. When I call torch.save(net,filename) I get the following error: <SCODE> File "<stdin>", line 1, in <module>
  File "torch/serialization.py", line 123, in save
    return _save(obj, f, pickle_module, pickle_protocol)
  File "torch/serialization.py", line 218, in _save
    _add_to_tar(save_storages, tar, 'storages')
  File "torch/serialization.py", line 31, in _add_to_tar
    fn(tmp_file)
  File "torch/serialization.py", line 189, in save_storages
    storage_type = normalize_storage_type(type(storage))
  File "torch/serialization.py", line 99, in normalize_storage_type
    return getattr(torch, storage_type.__name__)
AttributeError: 'module' object has no attribute 'HalfStorage'
<ECODE> I’ve tried casting to float, bringing the network back onto the cpu, and casting into a few other datatypes but it looks like a call to float() doesn’t change the underlying storage type in a way that would make this possible. If I don’t half() the tensor I can still save it just fine, but once I’ve called half() nothing I do apparently changes the underlying storage type back. Any tips (or if this has been fixed in a recent PR that I’m not seeing) would be appreciated. Thanks again for all the help, this has been an extremely pleasant experience thus far. Best, Andy

smth: 

ajbrock: It is, I’m currently just  .float().cpu().numpy() 'ing my weights and dumping them to .npz’s like I used to. Thanks for the response!
Gradients in variable obtained through narrowing

Tudor_Berariu: <SCODE>import torch
from torch.autograd import Variable

x = Variable(torch.linspace(1, 12, 12).view(3, 4), requires_grad=True)
x1 = x[:,:2]    # x1 is 3 x 2
x2 = x[:,1:]    # x2 is 3 x 3

y1 = 2 * x1
y2 = 3 * x2
y1.backward(torch.ones(3, 2))
y2.backward(torch.ones(3, 3))

print(x.grad)  # This is correct

# Variable containing:
#  2  5  3  3
#  2  5  3  3
#  2  5  3  3
# [torch.FloatTensor of size 3x4]

print(x1.grad)  # This is zero

# Variable containing:
#  0  0
#  0  0
#  0  0
# [torch.FloatTensor of size 3x2]<ECODE>

smth:
Newbie question: What are the prerequisites for running PyTorch with GPU?

yenson-lau: Hi everyone, I’m new to deep learning libraries, so apologies in advance if this is something I’m already supposed to know. I’ve used Theano before but guides for setting up the GPU there were very straightforward, also I was doing this using a WinPy instance on Windows. I come from a MATLAB background where I’m used to being able to play around with the variables and initialize things quickly, so naturally I felt like PyTorch may have more to offer. But there isn’t that much documentation on how to set things up from scratch with PyTorch. I have Anaconda installed on Ubuntu and have tried the basic functionality of PyTorch that way, but I’m not sure how to get it working with CUDA. Any advice would be appreciated!

moskomule: 

smth: if you want to use pytorch with an NVIDIA GPU, all you need to do is install pytorch binaries and start using it. We ship with everything in-built (pytorch binaries include CUDA, CuDNN, NCCL, MKL, etc.). So it’s a one-touch install: Install pytorch from binaries ( http://pytorch.org 1.2k has instructions) Try a simple program with CUDA: <SCODE>import torch
a = torch.randn(10).cuda()
print(a)
print(a + 2)
<ECODE>

moskomule: 

yenson-lau: <SCODE>a = torch.randn(10).cuda()
<ECODE> Just to make sure, I updated Pytorch with my Anaconda installation via the following <SCODE>conda install pytorch torchvision cuda80 -c soumith
<ECODE> Thanks again for the support!

yenson-lau: 

Sriharsha_Sammeta: Yup. You have to first do the CUDA binaries installation. if the above lines return True then you have Cuda working else not.

Dhruvrnaik:
Numerically unstable problem in matrix multiplication

yunjey: <SCODE>X = Variable(torch.randn(2, 30, 400))
Y = Variable(torch.randn(400, 400))

# 1st method
outs = []
for i in range(X.size(0)):
    out = torch.mm(X[i], Y)
    outs.append(out)
result1 = torch.stack(outs)  # shape of (2, 3, 4)

# 2nd method
result2 = X.resize(2*30, 400).mm(Y)
result2 = result2.resize(2, 30, 400)

# 3rd method
result3 = torch.bmm(X, Y.unsqueeze(0).expand(X.size(0), *Y.size()))

assert np.allclose(result1.data.numpy(), result2.data.numpy())  # this causes an error
assert np.allclose(result1.data.numpy(), result3.data.numpy())
assert np.allclose(result2.data.numpy(), result3.data.numpy())  # this causes an error
assert np.allclose(result2.data.numpy(), result3.data.numpy(), 1e-2)  # this doesn't cause an error<ECODE>

fmassa: 

joelmfonseca: Hi all, I am facing a similar issue: I have to implement a layer from a Theano implementation and reproducing it with the PyTorch framework leads to small numerical instability. Here is the code: <SCODE>import torch
import numpy as np
import theano
import theano.tensor as T

torch.manual_seed(7)

def func_pytorch(a, b):

    a = a.permute(0,2,1).contiguous().view(-1,400)
    b = b.view(400,-1)

    return torch.matmul(a, b).view(100,5,400,10).permute(0,2,3,1)

def func_theano(a, b):
    return T.tensordot(a, b, [[2], [2]])

def func_numpy(a, b):
    return np.tensordot(a, b, [[2], [2]])

a = torch.randn(100,400,5)
b = torch.randn(400,400,10)

a_p = a.permute(0,2,1)
b_p = b.permute(2,1,0)

out_pytorch = func_pytorch(a, b)
out_numpy = np.transpose(func_numpy(a_p,b_p), (0,3,2,1))

out_true = np.transpose(func_theano(a_p, b_p).eval(), (0,3,2,1))

np.testing.assert_allclose(actual=out_numpy, desired=out_true, rtol=1e-7) # OK
np.testing.assert_allclose(actual=out_pytorch, desired=out_true, rtol=1e-7) # 76% mismatch
<ECODE> Thanks in advance!
Error: ‘DataLoaderIter’ object has no attribute ‘shutdown’

vishal1796: 

smth: Without any context, we cannot reply with an answer. What are you looking for, is there a script to reproduce this error?

vishal1796: I am trying to implemen fast neural style. The training function is here <SCODE>for epoch in range(args.epochs):
    for iteration, batch in enumerate(data_loader):
        x = Variable(batch[0])
        x = batch_rgb_to_bgr(x)
        if args.cuda:
            x = x.cuda()
        y_hat = model(x)
        xc = Variable(x.clone())
        optimizer.zero_grad()
        loss = loss_function(args.content_weight, args.style_weight, xc, xs, y_hat)
        loss.backward()
        optimizer.step()
        print("===> Epoch[{}]({}/{}): Loss: {:.4f}".format(epoch, iteration, len(data_loader), loss.data[0]))
    torch.save(model.state_dict(), 'model_{}.pth'.format(epoch))
torch.save(model.state_dict(), 'model.pth')
<ECODE> and the dataloader code <SCODE>train_set = datasets.ImageFolder(args.dataset_path, transform)
data_loader = DataLoader(dataset=train_set, num_workers=args.threads, batch_size=args.batchSize, shuffle=True)
<ECODE>

smth: You can check this by doing: print(len(train_set)) <SCODE>a.png
cat/b.png
dog/c.png
<ECODE> Hope that helps figure out your issue.
Momentum update and in-place operators in optimizers

Ja-Keoung_Koo: In SGD optimizer code, <SCODE>                if momentum != 0:
                    param_state = self.state[p]
                    if 'momentum_buffer' not in param_state:
                        param_state['momentum_buffer'] = d_p.clone()
                    else:
                        buf = param_state['momentum_buffer']
                        d_p = buf.mul_(momentum).add_(1 - dampening, d_p)

                p.data.add_(-group['lr'], d_p)
<ECODE> It seems that the update in pytorch is something like this: <SCODE>v = momentum * v + (1-damping) * dp
p = p - lr * v
<ECODE> I felt just weird that learning rate is multiplied by both terms. Instead, I expect that <SCODE>v = momentum * v + (1-damping) * lr * dp
p = p - v
<ECODE> I just want to know it is a bug or intended.

apaszke: It’s intended. I think some frameworks use one definition, while others use the second one.
What’s the difference between a[0][1] and a[0 , 1]

splinter: <SCODE>		representation = Variable(th.zeros(batch_size , max_length , self.HIDDEN_SIZE * 2))

	for i in xrange(batch_size):
		for j in xrange(length[i]):
			representation[i][j] = th.cat((hidden_forward[max_length - length[i] + j][i]\
				, hidden_backward[max_length - 1 - j][i]) , 0)

	return representation
<ECODE> this code yields an error ’ RuntimeError: in-place operations can be only used on variables that don’t share storage with any other variables, but detected that there are 2 objects sharing it’ however, if I replace representation[i][j] with representation[i , j], the code runs just well. I’m wondering what’s the difference between these two ways to mention a particular part of a high-dimension tensor?

smth:
Detach the loaded model

alexis-jacq: Is there a correct way to detach a loaded model, when we don’t need to compute its gradients ? For the moment, I’m doing that: <SCODE>model = torch.load('mymodel.pth')
for variable in model.parameters():
    variable.detach_()
<ECODE>

smth: the correct way is to make the model’s parameters not require gradients: <SCODE>model = torch.load('mymodel.pth')
for p in model.parameters()
    p.requires_grad = False
<ECODE> <SCODE>myinput = torch.randn(10, 20) # example input
model = torch.load('mymodel.pth')
input = Variable(myinput, volatile=True)
output = model(input)
<ECODE>

alexis-jacq: Ah! good point for the input! Ok, thanks a lot!

FuriouslyCurious: NVM - sorry I figured it out my mistake.
Can I use dictionary in an extension of torch.autograd.Function when using GPU?

lili.ece.gwu: I have the following code in an extension of torch.autograd.Function: where y has type LongTensor, self.N and self.C have type long. it works fine if I use CPU, but got the following error if GPU is used: is this error because of I am using dictionary and GPU simultaneously or I did something else wrong? Thank you very much!

apaszke: 

lili.ece.gwu: I really appreciate your answer, but the next problem is how I can know the function is run in GPU or CPU, is there any flag indicates it? I tried to pass a bool argument from outside of the function but another error raises. Is it because of the torch.autograd.Function accepts Variable only? if so, how can I determine the function is run in GPU or not? Thank you!

apaszke: 

lili.ece.gwu: Many thanks!! it works now!!
Pytorch Imagenet Github Example - Training input variable not cuda?

nirvan: Why is the input variable not moved to cuda? Thank you!

smth: If your input has to use 2 GPUs, it’s more efficient to send the first half of the input to GPU1 and second half to GPU2, rather than sending the entire input to GPU1, and then sending half of it from GPU1 to GPU2. That’s the reason that the input is not transferred to the GPU at the code location that you pointed out.