foldl/torch-forum huggingface dataset:
How to specify GPU usage?

zhoubinxyz: I am training different models on different GPUs. I have 4 GPUs indexed as 0,1,2,3 I try this way: <SCODE>model = torch.nn.DataParallel(model, device_ids=[0,1]).cuda()
<ECODE> But actual process use GPU index 2,3 instead. and if I use: <SCODE>model = torch.nn.DataParallel(model, device_ids=[1]).cuda()
<ECODE> I will get the error: How to specify the GPU usage with index?

Seungyoung_Park: I am using Ubuntu 16.04. The GPU indexing are the same as you have. <SCODE>CUDA_VISIBLE_DEVICES=2,3 python xxx.py
<ECODE> In addition, I don’t think that dataparallel accepts only one gpu.

zhoubinxyz: 

apaszke: 

Seungyoung_Park: I am using pytorch 0.1.9 and Ubuntu 16.04. When I use CUDA_VISIBLE_DEVICES=2,3 (0,1), ‘nvidia-smi’ tells me that gpus 0,1 (2,3) are used. I do not know the reason, but the gpu id used in nvidia-smi and the gpu id used in pytorch are reversed. You can check it if you use Ubuntu 16.04.

shicai: 

albanD: 

zhoubinxyz: 

jramapuram: I’m curious about this as well. Can you currently use fractional GPU usage as in tensorflow? The tf equivalent is something like this: <SCODE>    with tf.device(FLAGS.device):
        gpu_options = tf.GPUOptions(per_process_gpu_memory_fraction=FLAGS.device_percentage)
        sess_cfg = tf.ConfigProto(allow_soft_placement=FLAGS.allow_soft_placement,
                                  gpu_options=gpu_options)
<ECODE>

Brando_Miranda: <SCODE>dtype = torch.FloatTensor
# dtype = torch.cuda.FloatTensor # Uncomment this to run on GPU
<ECODE> Also, do I really have to track how GPUs are assigned, I am fine with torch just doing its stuff automagically. In particular I would love to see how: is made into a GPU version of it.

saurabhjha21: Dear All, I am getting the below error: RuntimeError: cuda runtime error (2) : out of memory at /opt/conda/conda-bld/pytorch_1518244421288/work/torch/lib/THC/generic/THCStorage.cu:58 I think pytorch is not communicating with the Nvidia GPU, please advise.

ptrblck: 

saurabhjha21: Yes you are correct, it was fine after i restart the machine

Brando_Miranda: for a Unix command soln you can also do: <SCODE>export CUDA_VISIBLE_DEVICES=$i
<ECODE> though of course that only works if the scripts are independent and stuff like that…otherwise the other solutions here are probably better…

JannaShen: CUDA_VISIBLE_DEVICES=$i python main.py

labixiaoK: hi， do you have the answer？

DonghunP: Is there anyone who knows that… When I attach below code in python file(in main.py), <SCODE>import os
os.environment["CUDA_DEVICE_ORDER"] = "PCI_BUS_ID"
os.environment["CUDA_VISIBLE_DEVICES"] = "0"
<ECODE> The former one doesn’t specify(divide) GPU but, the latter one works well. It seems strange to me. Thanks ahead.

ptrblck: I wouldn’t recommend the first approach, since you would have to make sure these lines of code are imported before any other library, which might take the GPU. If some script imports PyTorch and these lines are executed afterwards, they won’t have any effect anymore. The second approach makes sure to mask the devices before running the Python script.

DonghunP: Totally understand thanks!!

LucasX:
Initalize the weights of nn.ConvTranspose2d

huxc_ustc: how shoud I initalize  the weights of nn.ConvTranspose2d ? like nn.Conv2d? is this any special for Pytorch <SCODE>for m in self.modules():
    if isinstance(m, nn.Conv2d):
        n = m.kernel_size[0] * m.kernel_size[1] * m.out_channels
        m.weight.data.normal_(0, math.sqrt(2. / n))
    elif isinstance(m, nn.BatchNorm2d):
        m.weight.data.fill_(1)
        m.bias.data.zero_()
<ECODE> infer :  the bias will automatically initialize with random value . is that right? <SCODE>def weights_initG(m):
    for p in m.modules():
        if isinstance(p,nn.Conv2d):
            n = p.kernel_size[0] * p.kernel_size[1] * p.out_channels
            p.weight.data.normal_(0, math.sqrt(2. / n))
        elif isinstance(p,nn.BatchNorm2d):
            p.weight.data.normal_(1.0, 0.02)
            p.bias.data.fill_(0)
        elif isinstance(p,nn.ConvTranspose2d):
            n=p.kernel_size[1]
            factor = (n+1)//2
            if n%2 ==1:
                center = factor - 1
            else :
                center = factor -0.5
            og = np.ogrid[:n,:n]
            weights_np=(1-abs(og[0]-center)/factor)* (1-abs(og[1]-center)/ factor)
            p.weight.data.copy_(torch.from_numpy(weights_np))
<ECODE>

chenyuntc: <SCODE>def weights_init(m):
    classname = m.__class__.__name__
    if classname.find('Conv') != -1:
        m.weight.data.normal_(0.0, 0.02)
    elif classname.find('BatchNorm') != -1:
        m.weight.data.normal_(1.0, 0.02)
        m.bias.data.fill_(0)

netG.apply(weights_init)
<ECODE> it should work.

david-leon: 

apaszke: 

acgtyrant: 

bille_du: 

chenyuntc: <SCODE>m.weight.data.normal_(1.0, 0.02).clamp_(min=0,max=2)
<ECODE>

bille_du: ok，thank you！by the way，if I want to clamp [-1, -0.1] and [0.1,1] 。How to operate？

chenyuntc: <SCODE>a.clamp_(min=-1,max=1)
a[a.abs()<0.1]=t.sign(a[a.abs()<0.1])*0.1
<ECODE>

bolt25: 

ptrblck: <SCODE>lin = nn.Linear(10, 10, bias=False)

with torch.no_grad():
    lin.weight.normal_(0.0, 1.0)
<ECODE>

bolt25: 

ptrblck:
How to delete PyTorch objects correctly from memory

111: I’m having an issue with properly deleting PyTorch objects from memory. With this Tensor: Then delete the object: CUDA memory is not freed up. Is there a clean way to delete a PyTorch object from CUDA memory?

albanD: Hi, It is because the cuda backend uses a caching allocator. This means that the memory is freed but not returned to the device.

111: Ah Thanks a lot! It’s really helpful!

nionjo: 

albanD: 

SANTOSH_S: 

albanD: Hi,
Freeze the learnable parameters of resnet and attach it to a new network
Category: vision

mderakhshani: I would like to know is this approach correct or not? Actually, in my point of view, only the 6 or 8 last layers have learnable parameters. Am I right?

apaszke: 

mderakhshani: 

apaszke: <SCODE>for param in net.features.parameters():
    param.requires_grad = False
<ECODE>

mderakhshani: 

mderakhshani: and here is its error:

albanD: 

mderakhshani: 

mderakhshani: My code: Error:

albanD: <SCODE>parameters = ifilter(lambda p: p.requires_grad, net.parameters())
<ECODE>

mderakhshani: 

albanD: 

apaszke: 

James_Chen: I am wondering whether to set .eval() for those frozen layers since they may still update their running mean and running var during training while not learning their parameters.
Problem on Variable.grad.data?

D_Kay: <SCODE>print(x.grad)
None

dir(x.grad)
['__class__', '__delattr__', '__doc__', '__format__', '__getattribute__', '__hash__', '__init__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__']
<ECODE> Whereas is it working fine on my personal computer , on PyTorch ‘0.1.9+aaf41c6’. Is this an issue or there is something that i Don’t know about? Thanks a lot !

albanD: Hi, <SCODE>x=torch.Tensor(5,5).normal_()
x=Variable(x,requires_grad=True)
print(hasattr(x.grad, "data")) # prints False
x.add(1).sum().backward()
print(hasattr(x.grad, "data")) # prints True
<ECODE>

D_Kay: Is there a way to easily init the gradient buffer ? I also wondered where I can get the latest patch notes? The last time I checked on github,  there were only the notes for the 0.1.9 version Thanks again !

albanD: Hi, For the 0.1.10 release note, I think Soumith is currently working on them and they should be on github soon.

D_Kay: I managed to get access to it by doing a “dummy” backprop in the _ _init _ _. That’s a quick and dirty solution, and I’m open to better ones. Thanks again !

apaszke: No, you can’t initialize it manually, but I don’t really see why would you need to do that (it’d be a tensor of zeros anyway). You can still access and modify the gradient, but only once the backward has been computed. Doesn’t this work for you: <SCODE>output = model(input)
loss(output).backward()
model.weight.grad.data # not None anymore. Can be modified
<ECODE>

ypxie: 

apaszke: I see, that’s actually a good point, we haven’t thought about that. We’ll have to solve it somehow

Ilya_Kostrikov: How did you perform the “dummy” backprop?

ypxie: I used the following workaround. <SCODE>        for shared_param in  shared_model.parameters():
            if not hasattr(shared_param.grad, 'data'):
                dummy_loss = 0
                for this_para in shared_model.parameters():
                    dummy_loss += torch.mean(this_para)
                dummy_loss.backward()
                break
<ECODE>

apaszke: So when exactly is it a problem? How are you implementing A3C? Are you sharing the main model parameters or does your training loop body look like this: <SCODE>loss = fn(input)
loss.backward()
copy_grads_to_shared_model(model, shared_model)
shared_model_optimizer.step()
copy_params_to_local_model(model, shared_model)

<ECODE>

apaszke: 

D_Kay: 

ypxie: 

apaszke: 

ypxie: Thank you~ For the safe issue, i noticed that DM’s paper explicitly said they don’t put a lock on the shared weights. For your solution to the shared grad to pytorch-a3c. <SCODE>def ensure_shared_grads(model, shared_model):
    for param, shared_param in zip(model.parameters(), shared_model.parameters()):
        if shared_param.grad is not None:
            return
        shared_param._grad = param.grad
<ECODE>

jingweiz: 

longhuei: So what’s the verdict here? Should we just remove the if condition? Because I can see no reason for the check. From what I’ve seen _grad is always accessible, and the global parameter grad is always updated to match the local, after I remove the if condition.

jhliew: 

xuehy:
Weird difference between function.forward(input) and function(input)

alexis-jacq: If I create a function and I apply it by calling forward method, the gradient computed seems independent of my backward() method and seems correct, even if the backward() was incorrect. For example, with this code: <SCODE>class Cube(Function):
    def forward(self,input):
        self.save_for_backward(input)
        return input*input*input

    def backward(self, grad_output):
        input, = self.saved_tensors
        # wrong backward function:
        return grad_output

cube = Cube()
input = Variable(torch.ones(2,2).double(), requires_grad=True)
output = cube(input).sum()
output.backward()
print(input.grad) # gives [[1,1],[1,1]] what does my backward do

input.grad.data.zero_()
output = cube.forward(input).sum()
output.backward()
print(input.grad) # gives [[3,3],[3,3]] the good gradient ?!
<ECODE>

jekbradbury:
Error using torch.load on update

temerick: I recently updated from pytorch 0.1.8 to 0.1.10 and I now get the following somewhat confusing error when I try to call torch.load on a file serialized with 0.1.8: <SCODE>----> 1 a = torch.load("file.pth7")

/usr/local/lib/python2.7/dist-packages/torch/serialization.pyc in load(f, map_location, pickle_module)
    220         f = open(f, 'rb')
    221     try:
--> 222         return _load(f, map_location, pickle_module)
    223     finally:
    224         if new_fd:

/usr/local/lib/python2.7/dist-packages/torch/serialization.pyc in _load(f, map_location, pickle_module)
    353     # try the legacy loader first, which only works if f is a tarfile
    354     try:
--> 355         return legacy_load(f)
    356     except tarfile.TarError:
    357         pass

/usr/local/lib/python2.7/dist-packages/torch/serialization.pyc in legacy_load(f)
    297                     args = pickle_module.load(f)
    298                     key, location, storage_type = args
--> 299                     obj = storage_type._new_with_file(f)
    300                     obj = restore_location(obj, location)
    301                     deserialized_objects[key] = obj

RuntimeError: Success
<ECODE>

apaszke: Huh, that’s surprising. No idea what could have caused that at the moment.

Cadene: <SCODE>save_checkpoint({
            'epoch': epoch + 1,
            'arch': options['model']['arch'],
            'state_dict': model.state_dict(),
            'best_prec1': best_prec1,
        }, is_best)
<ECODE> But when I try to load it with: <SCODE>torch.load(filename)
<ECODE> I get this error: <SCODE> Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/cadene/anaconda3/envs/vqa/lib/python3.6/site-packages/torch/serialization.py", line 222, in load
    return _load(f, map_location, pickle_module)
  File "/home/cadene/anaconda3/envs/vqa/lib/python3.6/site-packages/torch/serialization.py", line 377, in _load
    deserialized_objects[key]._set_from_file(f, offset)
RuntimeError: Success
<ECODE> It could be due to the fact that the data in my state_dict are of type torch.cuda.FloatTensor.
Adding Gaussion Noise in CIFAR10 dataset

Shiyu: If I want to add some Gaussion noise in the CIFAR10 dataset which is loaded by torchvision, how should I do it? Or, if I have defined a dataset by torch.utils.data.TensorDataset, how can I add more data samples there? Is any function like append( )? Thanks.

alykhantejani: There’s a few ways you can do this. <SCODE>import random

class RandomNoise(object):
    def __init__(self, probability):
         self.probabilit = probability
    def __call__(self, img):
        if random.random() <= self.probability:
            return img.clone().normal_(0,1)
        return img
          
cifar10_dataset = torchvision.datasets.CIFAR10(root="...", download=True, transform=input_transform)
<ECODE> <SCODE>def __getitem__(self, index):
    if index < 50000:
        return super(MySubClass, self).__getitem__(index)
    else:
        return torch.Tensor(size).normal_(0,1), torch.Tensor(1).fill_(class_label)

<ECODE>

maxiaoyuzdz: Hi, I use torchvision.transform to do it, it has a lambda function which you can customized a funciton to add noise to the data. But the CIFAR10 image is small just 32 * 32 * 10, after add sp or gaussion noise on them, the final result seems like not well .
Embeddings with half precision

skaae: <SCODE># Long 
embedding = nn.Embedding(embedding_dim=5, num_embeddings=10)
embedding.cuda()

x_device = torch.LongTensor([1,2,0,1]).cuda()
xv = Variable(x_device)
o = embedding(xv)
t = torch.zeros(o.size()).cuda()
o.backward(t)

# Half Crahes
embedding = nn.Embedding(embedding_dim=5, num_embeddings=10)
embedding.cuda().half()

x_device = torch.LongTensor([1,2,0,1]).cuda()
xv = Variable(x_device)
o = embedding(xv)
print(o)
t = torch.zeros(o.size()).cuda().half()
o.backward(t)<ECODE>

fmassa: <SCODE>if grad_output.is_cuda:
<ECODE>

skaae: Thanks. I’ll try to understand the Embedding code better and create a PR.
Swap axes in pytorch?

Response777: Is there equivalent operation in pytorch?

apaszke: 

Response777: Thanks, I have just checked the Docs, but it seems that I just miss it…

fmassa: <SCODE>a = torch.rand(1,2,3,4)
print(a.transpose(0,3).transpose(1,2).size())
print(a.permute(3,2,1,0).size())
<ECODE>

fmassa: Indeed, it can be a shortcut to use <SCODE>tensor.transpose_(0, 1)
<ECODE> instead of <SCODE>tensor = tensor.transpose(0, 1)
<ECODE>

rahulbhalley:
CUBLAS error if I replace torch.svd() with torch.symeig()

OCY: I am trying to get eigenvalues and eigenvectors of a real symmetric matrix A. Any idea about this? Thanks in advance

apaszke: What’s your PyTorch version? Did you do a binary or source install?

OCY: But still have the same error message.

smth: If possible, can you give me a 10-line program that produces this error? I will look at it and try to resolve it next week.

OCY: Because I changed a lot, I cannot reproduce that error, unfortunately. If I have the same problem, I will continue this post with reproduced code. Thanks!

David_Bau: Hey, I ran into this problem on a current build of pytorch (October 2017). Here’s a minimal repro - it’s triggered for me when the symmetric matrix is 129x129 or larger.
CUDA error when calling backward() on Tesla M2070

supakjk: I have two systems, where the first has GeForce GTX 780 Ti with CUDA 8.0 (driver version: 375.26) and the other has Tesla M2070 with CUDA 7.5.18 (driver version: 352.99). I tested the following simple code: <SCODE>import torch
from torch.autograd import Variable
a = Variable(torch.randn(3,4,5), requires_grad=True).cuda()
b = torch.randn(3,4,5).cuda()
a.backward(b)
<ECODE> The code works on the first machine but failed on the other machine as follows: <SCODE>THCudaCheck FAIL file=/users/PAS0396/osu7806/pytorch/torch/lib/THC/generic/THCTensorCopy.c line=65 error=46 : all CUDA-capable devices are busy or unavailable
Traceback (most recent call last):
  File "test.py", line 5, in <module>
    a.backward(b)
  File "/users/PAS0396/osu7806/anaconda3/lib/python3.6/site-packages/torch/autograd/variable.py", line 146, in backward
    self._execution_engine.run_backward((self,), (gradient,), retain_variables)
  File "/users/PAS0396/osu7806/anaconda3/lib/python3.6/site-packages/torch/autograd/_functions/tensor.py", line 163, in backward
    return grad_output.cpu()
  File "/users/PAS0396/osu7806/anaconda3/lib/python3.6/site-packages/torch/tensor.py", line 31, in cpu
    return self.type(getattr(torch, self.__class__.__name__))
  File "/users/PAS0396/osu7806/anaconda3/lib/python3.6/site-packages/torch/cuda/__init__.py", line 276, in type
    return super(_CudaBase, self).type(*args, **kwargs)
  File "/users/PAS0396/osu7806/anaconda3/lib/python3.6/site-packages/torch/_utils.py", line 33, in _type
    return new_type(self.size()).copy_(self, async)
RuntimeError: cuda runtime error (46) : all CUDA-capable devices are busy or unavailable at /users/PAS0396/osu7806/pytorch/torch/lib/THC/generic/THCTensorCopy.c:65
<ECODE>

albanD: Hi,

supakjk: 

apaszke: You might try building form source but it would require some additional patches (look for closed issues in the main repo). But we don’t support them officially.

supakjk: <SCODE>export CMAKE_PREFIX_PATH=/home/kimjook/anaconda3
conda install numpy mkl setuptools cmake gcc cffi
conda install -c soumith magma-cuda75
git clone https://github.com/pytorch/pytorch
cd pytorch
pip install -r requirements.txt
python setup.py install
<ECODE> Thanks! (I understand that supporting old devices is annoying, but I am somewhat frustrated since my almost the same model worked well on Torch7 doesn’t work on PyTorch.)

apaszke: That’s the issue I was thinking about, but maybe you don’t need it for some reason.

fazlerabbitanjil: 

smth: We should update that part. I’m doing it now. We started with the commitment of cc >= 2.0, but it has been infeasible, as 2.0 is simply too old and several newer APIs dont work on it.
REINFORCE in batch mode

sscommanderh: I have a feeling that this code need to be modified: <SCODE>for action, r in zip(model_Net.saved_actions, rewards):
    action.reinforce(r)
optimizer.zero_grad()
autograd.backward(model_Net.saved_actions, [None for _ in model_Net.saved_actions])
<ECODE> My question is how to pass the ‘r’ and ‘action’ in a batch mode in back-propagation ? It might related to reshape the ‘action’ values in a way to allow back propagation. Right now I came up with an idea that is to compute ‘r’ and ‘action’ in batch mode in forward passes, but update the gradients sequentially (1 sample at a time) in back-propagations (e.g., run ‘finish_episode’ several times). But it’s obviously not optimal.

jekbradbury: 

AjayTalati: Nothing, complicated, (or long to train like gym environments or Atari), just say a synthetic linear regression dataset with some noise added? This would be really helpful to newbs !
How does the backward pass get plumbed in and other rudimentary loss function questions

mattmacy: Since I’m just imitating what I see elsewhere there’s a few things I don’t understand about loss functions. My second question is what the last two lines are doing. What does the view(*repeat(… do? And what does mul_(tensor.expand_as(…) do? Last question is, does the DiceLoss class make sense? When I first started looking at adding a new loss function a few days ago it seemed overwhelming, but this seems like a rather trivial piece of code. Thanks in advance.

apaszke:
Bug in int8 ndarray to tensor conversion or in my understanding?

mattmacy: I can obviously preallocate an appropriately sized CharTensor and then copy myself, but it seems like that shouldn’t be necessary.

mattmacy: Trying to work around it with a ByteTensor as dst and ndarray of uint8 as follows: <SCODE>def copy_tensor(dst, src):
    for z in range(Z_MAX):
        for y in range(Y_MAX):
            for x in range(X_MAX):
                val = src[z][y][x]
                dst[z][y][x] = val
<ECODE> The labels masks are 192x160x192, so it really would be nicer to keep them as single byte tensors.

mattmacy: this does work - albeit slowly. And it’s actually so slow as to be useless. <SCODE>def copy_tensor(dst, src):
    for z in range(Z_MAX):
        for y in range(Y_MAX):
            for x in range(X_MAX):
                val = src[z][y][x]
                if val != 0:
                    dst[z][y][x] = 1
<ECODE> It appears that the only ndarray to tensor conversion that actually works is floating point.

albanD: Hi,

mattmacy: Converting to np.uint8 and then calling from_numpy works. Thanks.

xiao1228: Thank you very much
Anaconda Stopping kernel When I use my Customized Cost Function

mderakhshani: 

apaszke: Try running the code outside of an iPython notebook. It should print the full error then

mderakhshani: 

apaszke: Try running this: <SCODE>gdb python
<some output printed here>
> r your_script.py
<some more output. it will tell you that you got SIGSEV and drop into shell again>
> where
<a few lines looking like `#0 THPFloatTensor... `. Paste it in a GitHub gist
and put a link in this thread>

<ECODE>

mderakhshani:
How to reduce memory usage when one large networks called many times

qianguih: Hi, there, I am a newbie to PyTorch and I am really impressed by its speed and flexibility. Amazing work! But I have one question about the memory usage. Any suggestions would be welcome! Thank you! <SCODE>model = AntoEncoderDecoder()
output = Variable(torch.zeros(256,256))
for i in range(iterations):
    output = model( output )<ECODE>

albanD: Hi,

apaszke: 

qianguih: 

apaszke: 

qianguih: Yes, it is 0.1.10. I guess theano has optimized every computation graph a lot during the function compilation.

apaszke: Maybe. It’s hard to say what’s happening without having the model details.
Fast data loader for Imagenet

longcw: 

apaszke: How many workers are you using? What kind of disk are you using to store the dataset?

longcw: 

apaszke: Yes, the data transforms can be quite expensive.

Veril: Unfortunately all those mass consumption dataset loaders and feeders are far from optimal, everyone serious about this ends up writing their own performant code (or compensate by getting expensive SSDs and CPUs) but don’t publish it because it’s embarrassingly written. People often say you need to use SSDs and whatnot, but I use a couple of ancient HDDs and it’s enough to feed a duo 1080 setup. It’s just troublesome to write a properly multi threaded application for this, you don’t even need to do it in C if you have a good CPU.

Chong_Lv: More workers seems slower. The data loading is extremely slow with a low cpu load, which means the dataloader can’t make full use of the cpu.

gngdb: 

lkywk: did you store the images file after preprocessing?

gngdb: Yeah, but it’s a one-off cost. I just copied it to the other servers after.

lkywk: I run the script, but I meet the problem.

gngdb: Could be that you’re missing protobuf. You could install it with conda. If that works, I’ll add it to the README, I don’t have time to test it myself right now.

lkywk: I fixed it days ago. I think it is caused by protobuf.

charlesjiangxm: Hi, I’ve used your data but the loading speed is still 4s for batch=256, num_worker=1, which I think is not fast enough, is it normal?

gngdb: I used 4 workers, and each minibatch took 0.59s to process, including the time for forward and backward propagation. Maybe you should try with more workers?

Chenzhi_Jiang: Hi, I use your method. But when I train the imagenet, I meet the problem

Chenzhi_Jiang: I fixed it. Just need to downgrade msgpack to 0.5.6