foldl/torch-forum huggingface dataset:
GPU failure if training on the primary GPU?

ming: Hi, We have a machine with 4 Titan X GPUs - but we can only train Pytorch models on the 3 GPUs other than the primary one. As long as we utilize the primary GPU, pytorch would hangs after a few iterations and nvidia-smi would report GPU loss afterwards - only rebooting can recover the machine. We have tried to uninstall X-org from Ubuntu 16 desktop, or re-install Ubuntu 16 server without X. Disconnect display etc. It always causes GPU loss if the training utilizes the primary GPU. We also tried to use MXNET train on 4 GPUs, it goes well without seeing this problem. Any idea why Pytorch cannot train on the primary GPU? Any workaround we could try?

smth: Hi Ming, This could be because when you are using MXNet, the mxnet install did not end up using cudnn or nccl (for example) and runs fine.

ming: Thank you Soumith for the prompt reply! We are using Nvidia 375.26, CUDA 8.0 and CUDNN 5.1 which are the latest. I doubt it’s overheating issue because we can train the model using GPUs 1, 2, 3, but cannot use GPUs 0, 1, 2, where GPU 0 is the primary one. We also see cases that on machine with single GTX 1080 GPU, training pytorch model that maximize usage of GPU memory could cause GPU failure too. In any case, I will try installing pytorch from source and report back result.

phenom: Hi Ming, Were you able to solve the problem? I am also stuck at the same problem…
Convolution operator with groups for taking each convolved result

Ja-Keoung_Koo: In my case, I want to take each convolution output. In traditional convolution networks, if we assume that input channel (R,G,B) is 3 and output channel (J1, J2) is 2, <SCODE>J1 = F1 * R + F2 * G + F3 * B
J1 = F4 * R + F5 * G + F6 * B
<ECODE> where F1,F2,… denotes filters (say 3x3 filters) and * denotes convolution operator. In my case, I want to manipulate the each convolved result. So I want to take the followings, not output images: <SCODE>F1 * R, F2 * G, F3 * B,
F4 * R, F5 * G, F6 * B
<ECODE>

apaszke: 

Ja-Keoung_Koo: Thanks you for kind reply! <SCODE>       self.weight = Parameter(torch.Tensor(
            out_channels, in_channels // groups, *kernel_size))<ECODE>

apaszke: 

siki: Thanks to this Q&A, it helps me to understand groups operation.

juliohm: 

hinken: In a backprop context, you’d use groupwise convolution to either make the model splittable into more than one GPU (since the groups won’t have to “cross” data), or to force the training to split channels into different groupings of features (you can google on this, there are some nice explanations available). In a more general context, you could find any number of applications, lets say for simplicity that you just want to blur each R, G and B channel separately with a blur kernel and want RGB output (and you definitely don’t want to blur R with G etc.).

Yohan_Sumathipala: Quick follow-up: What is the order of the channels in the output image? Assuming input is RGB image and output is 6 channels (as in OP) is the order:

ptrblck: Let’s create a small example and have a look at the order: <SCODE># Set R=100, G=200, B=300
x = torch.FloatTensor([100, 200, 300]).view(1, -1, 1, 1)
x = Variable(x)

conv = nn.Conv2d(in_channels=3,
                 out_channels=6,
                 kernel_size=1,
                 stride=1,
                 padding=0,
                 groups=3,
                 bias=False)

# Set Conv weight to [0, 1, 2, 3, 4 ,5]
conv.weight.data = torch.arange(6).view(-1, 1, 1, 1)
output = conv(x)
print(output)
> Variable containing:
(0 ,0 ,.,.) = 
     0

(0 ,1 ,.,.) = 
   100

(0 ,2 ,.,.) = 
   400

(0 ,3 ,.,.) = 
   600

(0 ,4 ,.,.) = 
  1200

(0 ,5 ,.,.) = 
  1500
[torch.FloatTensor of size (1,6,1,1)]
<ECODE> The only way to get this result is the second guess.

Yohan_Sumathipala: Thanks so much, this is super helpful!

ixez: I try the following codes: <SCODE>import torch
from torch import nn

# Set (RGB)*2 = (0, 100, 200), (300, 400, 500)
x = (torch.arange(6, dtype=torch.double)*100).view(1, -1, 1, 1)

conv = nn.Conv2d(in_channels=6,
                 out_channels=2,
                 kernel_size=1,
                 stride=1,
                 padding=0,
                 groups=2,
                 bias=False)

# Set Conv weight to 1
conv.weight.data = torch.ones(6, dtype=torch.double).view(2, 3, 1, 1)

output = conv(x)

print(output)
<ECODE> output = (500, 5000). The result seems to be your Guess 2, RR GG BB.
Accessing objective function’s value from optimizer

moskomule: 
p.data is x
 
p.grad.data is the gradient of f at x
 where <SCODE>for group in self.param_groups:
    for p in group['params']:
         pass
<ECODE> Is there any good way to get the value f(x) from optimizer? Thank you.

apaszke: 

moskomule: 

apaszke: 

moskomule:
DataLoader gives double instead of float?

Veril: <SCODE>X_train = torch.from_numpy(X_train)
Y_train = torch.from_numpy(Y_train)
X_test = torch.from_numpy(X_test)
Y_test = torch.from_numpy(Y_test)

[all dtypes torch.FloatTensor confirmed]

train = torch.utils.data.TensorDataset(X_train, Y_train)
trainloader = torch.utils.data.DataLoader(train, batch_size=BATCH_SIZE, shuffle=True)


train_iter = iter(trainloader)
data = train_iter.next()
x, y = data		
print y
<ECODE> And I get torch.DoubleTensor x is torch.FloatTensor

apaszke: I can’t reproduce your issue. To resolve a problem we need a self-contained snippet that we can run.

Veril: The following reproduces this. <SCODE>import numpy as np

import torch
import torch.utils.data


X_train = np.random.uniform(-1, 1, (1000,11)).astype(np.float32)
Y_train = np.hstack((np.zeros(500), np.ones(500))).astype(np.float32)

X_train = torch.from_numpy(X_train)
Y_train = torch.from_numpy(Y_train)


print X_train
print Y_train


train = torch.utils.data.TensorDataset(X_train, Y_train)
trainloader = torch.utils.data.DataLoader(train, batch_size=128, shuffle=True)


train_iter = iter(trainloader)
data = train_iter.next()
x, y = data		
print x
print y
<ECODE> Maybe this is intentional but I can’t figure out why make a float into a double in that context, and why only the second one.

albanD: <SCODE>Y_train = torch.from_numpy(Y_train).view(-1, 1)
<ECODE> <SCODE> return self.data_tensor.narrow(0, index, 1), self.target_tensor.narrow(0, index, 1)
<ECODE> should solve the issue by always returning Tensor and not numbers. Would this break something I’m not aware of? (Let me know if you want me to send a PR for that)

Veril: copy_ doesn’t care if it gets data in shape (batch) or (batch,) ? A quick test shows no changes in loss behavior. <SCODE>input = Variable(torch.FloatTensor(BATCH_SIZE, dims).cuda())
label = Variable(torch.FloatTensor(BATCH_SIZE).cuda())

x, y = train_iter.next()
input.data.resize_(x.size()).copy_(x)
label.data.resize_(x.size(0)).copy_(y)<ECODE>

albanD:
How to save/load Torch models?

Russel_Russel: 

apaszke: 

kindlychung: 

apaszke: 

YunTang: Thanks Yun

smth: state_dict is easy to manipulate and load into other models. it’s also a simpler structure. Your model class definition might also not be picklable (possible in some complicated models) Other than this, it’s a matter of taste.

Nick_Brandaleone: Is there any work to export Pytorch models to new Apple format (CoreML), or convert to Keras (so it can be used by Apple devices)? Nick

Brando_Miranda: 

allenye0119: 

rajarsheem: 

smth: 

kirill-pinigin: Please tell me. Does Pytorch save model in h5df format file or another format?  What I must do to save dict model in h5df format?

singleroc: 

dhanyashri_b_v: When you save the pytorch model in .pth format. The entire model structure will get saved in it right. Again why do we have to keep the .py script along with .pth file? Thank you,
Easy way to compute outer products?

yoonholee: 

Cysu: 

yoonholee: Perfect, just what i needed. Thanks!

vitchyr: Is there a way to do this in batch mode? So if I have two batches of vectors, is there an easy way to compute the (batch) outer products?

tom: <SCODE>b = u.size(0)
m = u.size(1)
n = v.size(1)
u.unsqueeze(2).expand(b,m,n)*v.unsqueeze(1).expand(b,m,n)
<ECODE> Best regards Thomas

Avinab_Saha: 

tom: Best regards Thomas

Rahul_Soni: Hope this helps!
Implementing custom non-zero conv padding

Veril: Is there a way to implement your own padding algorithm? For example use reflect padding instead of the seemingly solely available one: zero padding.

apaszke: 

Veril: That would be a welcome addition. It could certainly be done in a hack-y way where you cut up variables, but that doesn’t seem like it would be very efficient.

Veril: Numpy has apply_along_axis but it doesn’t look like pytorch has that unless I’m missing something. I’m looking for a way to apply a custom (maybe learned too) convolution onto each feature map individually. So if you have a (64,32,128,128) tensor you want a say, (5,5) filter to be applied to all 64*32 (128,128) images. Maybe I’m not thinking in arrays right, what if I reshape it to (64*32,1,128,128) and do convolutions the usual way? Then reshape back? Seems like it should work, would reshaping it naively break things though?

apaszke: Hmmm yes I think that reshaping them like that would work.

Veril: Does it return an enlarged tensor based on the requested padding?

apaszke:
RuntimeError: cuda runtime error (59)

kundan2510: I would greatly appreciate any pointers.

kundan2510: I had an issue with the embedding layer. I had an index which was greater than the vocab-size. Fixing that resolved the issue. I still don’t understand how the error shown is related to that.

smth: Because of the asynchronous nature of cuda, the assert might not point to a full correct stack trace pointing to where the assert was triggered from.

ajbrock: STORY TIME! I recently re-installed my NVIDIA drivers (someone in the lab broke the server trying to install them, and for the life of me I have no idea what went wrong) and managed to compile the latest PyTorch build from source (which previously had thrown errors related to CUDA, suggesting that PyTorch unsurprisingly doesn’t compile if you’re using old drivers). Yay! It turned out the issue was in the line above, where I call index_select on a different tensor using a variable that can vary widely–the issue was that I clamped my indices incorrectly: out = inp.index_select(2, torch.clamp(index, 0, inp.size()[-1])).

smth: 

ajbrock: Or is the tensor.index_select function just something that’s hard-baked into CUDA that PyTorch doesn’t actually wrap? I might be misunderstanding how it’s put together.

apaszke: 

ajbrock: Sorry, I was just trying to do that as an example of indexing outside the allowable array size. Here’s a better example: returns RuntimeError: out of range at /data/users/soumith/miniconda2/conda-bld/pytorch-0.1.7_1485444530918/work/torch/lib/TH/generic/THTensor.c:379 Is it not possible to insert a value check in the index_select function to ensure that the elements of the second argument are all less than the size of the indexed tensor along that axis?

apaszke: It’s impossible, because we’d have to execute max and min over the whole tensor, synchronize the CPU with the GPU, and execute the index_select kernel only if the values are correct. It would be too slow.

ajbrock: Got it, that makes sense.

11129: This help me, thanks.

euwern: same here, thanks for the hint… I was converting some code from torch to pytorch. Lua is 1 based index but Python is 0 based index. My dataset was created for lua hence it is off by one.

gleb: I had this error when have created the model with the wrong number of classes:

miladiouss: 

Fengzi: I faced with the problem that is very similar: ‘reduce failed to synchronize: device-side assert triggered’,and the problem occurred in the circumstance that I used the BCEloss.finally I found the cause that my output is a mat consists of the elements that is negative numbers,but the implement of BCEloss is about log(),which has no meaning in the negative area, thus I add a sigmoid to constrain the output between 0-1and my problem is solved.

jdenim: Thanks

m0nster: 

tom: I’m not sure if I’m missing the obvious, but how about subtracting 1 from the labels? Best regards Thomas

m0nster: 

tom:
Backward size mismatch error when adding the bias term

Sandeep42: Hi, I am trying to implement a batch matrix multiplication like the first equation in this image. The weight and bias are defined as a parameter in the model. I am making a copy of the bias term to the entire batch. <SCODE>def batch_matmul_bias(seq, weight, bias, nonlinearity=''):
s = None
bias_dim = bias.size()
for i in range(seq.size(0)):
    _s = torch.mm(seq[i], weight) 
    _s_bias = _s + bias.expand(bias_dim[0], _s.size()[0])
    print _s_bias.size()
    if(nonlinearity=='tanh'):
        _s_bias = torch.tanh(_s_bias)
    _s_bias = _s_bias.unsqueeze(0)
    if(s is None):
        s = _s_bias
    else:
        s = torch.cat((s,_s_bias),0)
return s.squeeze()
<ECODE> The forward pass works, but when doing the backward pass, I am getting a size mismatch error. <SCODE>RuntimeError: sizes do not match at /data/users/soumith/miniconda2/conda-bld/pytorch-cuda80-0.1.7_1485448159614/work/torch/lib/THC/generated/../generic/THCTensorMathPointwise.cu:216
<ECODE> Can you help me fix it? Thank you.

albanD: <SCODE>import torch.nn.functional as F

def batch_matmul_bias(seq, weight, bias, nonlinearity=''):
  s = F.linear(seq, weight, bias)
  if nonlinearity=='tanh':
    s = F.tanh(s)
  return s
<ECODE>

Sandeep42: I just looked at the API carefully, and it looks like Linear supports batch samples and no bias as well, this will save a lot of time. Thank you!

albanD: Hi,

Sandeep42: I managed to fix it. For future reference, I was sloppy and did not properly reshape the bias term. Doing a transpose of the bias term is the one I forgot. <SCODE>_s_bias = _s + bias.expand(bias_dim[0], _s.size()[0]).transpose(0,1)
<ECODE> Thank you for the wonderful effort that you’ve put in here, debugging is a lot easier. BTW: Can you explain this?

Veril: Is there some straightforward way to reshape while using add_module() ? ATM I have to resort to <SCODE>class RESHAP(nn.Module):
	def __init__(self, nz):
		super(RESHAP, self).__init__()
		self.nz = nz

	def forward(self, input):
		return input.view(-1, self.nz, 1, 1)

	def __repr__(self):
		return self.__class__.__name__ + ' ()'
<ECODE> main.add_module(‘initial.{0}.reshape’.format(nz/4), RESHAP(nz/4))

smth: 

apaszke: 

Sandeep42: I am looking forward to that. The fact that you can just interactively work with Tensors, and debug them is already a great thing. Thanks!
Gradient for input when evaluate

meijieru: Hi, I want to use a pretrained model to supervise another model, which need to set the batch normalization into eval mode. However, with lstm in my pretrained model, it arises follow message: <SCODE>Traceback (most recent call last):
  File "main.py", line 153, in val
    cost.backward()
  File "/home/jrmei/.local/lib/python2.7/site-packages/torch/autograd/variable.py", line 145, in backward
    self._execution_engine.run_backward((self,), (gradient,), retain_variables)
  File "/home/jrmei/.local/lib/python2.7/site-packages/torch/autograd/function.py", line 208, in _do_backward
    result = super(NestedIOFunction, self)._do_backward(gradients, retain_variables)
  File "/home/jrmei/.local/lib/python2.7/site-packages/torch/autograd/function.py", line 216, in backward
    result = self.backward_extended(*nested_gradients)
  File "/home/jrmei/.local/lib/python2.7/site-packages/torch/nn/_functions/rnn.py", line 199, in backward_extended
    self._reserve_clone = self.reserve.clone()
AttributeError: 'CudnnRNN' object has no attribute 'reserve'
<ECODE> Do you have any suggestions? Thanks very much.

smth: 

apaszke: I see the bug. I’ll fix it today.

meijieru: <SCODE>import torch
import torch.nn as nn
from torch.autograd import Variable

rnn = nn.Sequential(nn.LSTM(10, 10, 2, bidirectional=True))
rnn = rnn.cuda()
rnn.eval()

i = torch.Tensor(2, 1, 10).cuda()
i = Variable(i, requires_grad=True)

o, _ = rnn(i)
o = o.mean()
o.backward(torch.ones(1).cuda())
<ECODE>

meijieru: 

smth: It will be today. I will be building binaries late today.

meijieru: Thanks very much. (~~~~~~~~~~~~~~~~~~)
Is that possible to save model in Lua “.t7” format to load from Lua Torch?

WildChlamydia: Subj, is that possible in a some way?

alexis-jacq: 

WildChlamydia: Sorry, I don’t understand how it will helps to serialize to LuaTorch format

apaszke: We don’t support that at the moment. PyTorch models are much more flexible than those in Lua, so it’s easier to go the other way around. You could try recoding your model and dumping the weights to HDF5 format.

WildChlamydia: 

apaszke:
Model parallelism in pytorch for large(r than 1 GPU) models?

ajdroid: Hi! I have a model that is too large to fit inside a single TITAN X (even with 1 batch size). I want to split it over several GPUs such that the memory cost is shared between GPUs. That is, place different parts of the same model on different GPUs and train it end-to-end. Questions:

apaszke: <SCODE>class MyModel(nn.Module):
    def __init__(self, split_gpus):
        self.large_submodule1 = ...
        self.large_submodule2 = ...

        self.split_gpus = split_gpus
        if split_gpus:
            self.large_submodule1.cuda(0)
            self.large_submodule1.cuda(1)

    def forward(self, x):
        x = self.large_submodule1(x)
        if split_gpus:
            x = x.cuda(1) # P2P GPU transfer
        return self.large_submodule2(x)
<ECODE>

ajdroid: This was so easy! I love your work with PyTorch. Minimum fuss! Cheers!

D-X-Y: 

Hengck: <SCODE>   def forward(self, x):
         
        x1= self.resnet_1(x[:,0:3,:,:]).cuda(0)
        x2= self.resnet_2(x[:,3:6,:,:]).cuda(1)

        flat  = torch.cat([x1, x2],1).cuda(???)
        logit = self.fc(flat).cuda(???) 
        return logit

<ECODE>

isalirezag: did you figure out how to do this?

dimtsi: 

LeiGuan: Dear apaszke, I am trying to implement inter-GPU communiation by using pytorch+mpi+gpu. Following are the tested code, which is designed to make sure that process0 runs on GPU0 and process1 runs on GPU1. However, the code can not be run successfully. Do you know why? def run(rank, size): mpirun noticed that process rank 1 with PID 0 on node osherlab exited on signal 11 (Segmentation fault). Your help will be appreciated. Thank you.

marcel1991: Would this also work for one 1 gpu with two sequential steps somehow? If my model is too large to fit on one gpu can I somehow do the forward/backward pass sequtially where I only have one part in gpu memory and somehow cache the other part for the backward pass later. Somehow like this: <SCODE>x = submodule1(x)
#somehow unload intermediate results of submodule1 from gpu here and cache for later backward pass 
#(and then load on gpu again when needed in backward pass of submodule1)
x = submodule2(x)
<ECODE> I could imagine how this works but then I don’t know how I would pass the gradients that come from submodule2 back to submodule1 and initiate the backward pass on submodule1.

RaayChen: Thx

shinx:
Quetions about train DCGAN

yichuan9527: 

Ismail_Elezi: I am not totally sure, because I am still learning PyTorch, so take my answer with a pinch of salt. input and label types are Variable. They are not Tensors, instead they are objects that encapsulate the Tensors and all their history (the operations that they have done). You neither can (you get a NameError if you try to do so) - nor want - to resize the Variable objects. What you want to do is to resize the Tensors within the variable and you can do that by using precisely the given code (data function).

apaszke: 

zip: Why is the input tensor resized in every iteration of the loop? Is there a computational overhead in doing this?
SGD Nesterov for Optim

ritchieng: 

ritchieng: <SCODE>from .optimizer import Optimizer, required


class SGD(Optimizer):
    """Implements stochastic gradient descent (optionally with momentum).
    Args:
        params (iterable): iterable of parameters to optimize or dicts defining
            parameter groups
        lr (float): learning rate
        momentum (float, optional): momentum factor (default: 0)
        weight_decay (float, optional): weight decay (L2 penalty) (default: 0)
        dampening (float, optional): dampening for momentum (default: 0)
        nesterov(bool, optional): enables Nesterov momentum (default: False)
        
    Example:
        >>> optimizer = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.9)
        >>> optimizer.zero_grad()
        >>> loss_fn(model(input), target).backward()
        >>> optimizer.step()
    """

    def __init__(self, params, lr=required, momentum=0, dampening=0,
                 weight_decay=0, nesterov=False):
        defaults = dict(lr=lr, momentum=momentum, dampening=dampening,
                        weight_decay=weight_decay, nesterov=nesterov)
        if nesterov and (momentum <= 0 and dampening != 0):
            raise ValueError("Nesterov momentum requires a momentum and zero dampening")
        super(SGD, self).__init__(params, defaults)

    def step(self, closure=None):
        """Performs a single optimization step.
        Arguments:
            closure (callable, optional): A closure that reevaluates the model
                and returns the loss.
        """
        loss = None
        if closure is not None:
            loss = closure()

        for group in self.param_groups:
            weight_decay = group['weight_decay']
            momentum = group['momentum']
            dampening = group['dampening']
            nesterov = group['nesterov']

            for p in group['params']:
                d_p = p.grad.data
                if weight_decay != 0:
                    d_p.add_(weight_decay, p.data)
                if momentum != 0:
                    param_state = self.state[p]
                    if 'momentum_buffer' not in param_state:
                        param_state['momentum_buffer'] = d_p.clone()
                    else:
                        buf = param_state['momentum_buffer']
                        buf.mul_(momentum).add_(1 - dampening, d_p)
                        if nesterov:
                            d_p.add_(momentum, buf)
                        else:
                            d_p = buf

                p.data.add_(-group['lr'], d_p)

        return loss
<ECODE>

apaszke: Yes, it’s going to be merged into master soon.

Royi: 

Ismail_Elezi: Accelerated gradient descent is not a momentum method, but it has been shown that it is closely related and the update rule can be rewritten as a momentum-like update rule.
The weird test dataset performance of pytorch

waitwaitforget: Hi, <SCODE>class MyDeepNeural(nn.Module):
def __init__(self, p_keep_conv):
    super(MyDeepNeural, self).__init__()

    self.conv = nn.Sequential()
    self.conv.add_module('conv1', nn.Conv2d(1, 32, kernel_size=3, padding=1))
    self.conv.add_module('relu1', nn.ReLU())
    self.conv.add_module('pool1', nn.MaxPool2d(kernel_size=2))
    self.conv.add_module('drop1', nn.Dropout(1 - p_keep_conv))
    self.conv.add_module('conv2', nn.Conv2d(32, 64, kernel_size=3, padding=1))
    self.conv.add_module('relu2', nn.ReLU())
    self.conv.add_module('pool2', nn.MaxPool2d(kernel_size=2))
    self.conv.add_module('drop2', nn.Dropout(1 - p_keep_conv))
    self.conv.add_module('conv3', nn.Conv2d(64, 128, kernel_size=3, padding=1))
    self.conv.add_module('relu3', nn.ReLU())
    self.conv.add_module('pool3', nn.MaxPool2d(kernel_size=2))
    self.conv.add_module('drop3', nn.Dropout(1 - p_keep_conv))

    self.fc = nn.Sequential()
    self.fc.add_module('fc1',nn.Linear(128*9, 625))
    self.fc.add_module('relu4',nn.ReLU())
    self.fc.add_module('fc2',nn.Linear(625,10))
    self.fc.add_module('softmax',nn.Softmax())<ECODE>

apaszke: Apparently that’s the model dynamics on your dataset. It happens.

waitwaitforget: Thank you very much. It solved my problem well.