foldl/torch-forum huggingface dataset:
Cache datasets pre-processing

bodokaiser: 

apaszke: 

jekbradbury: 

bodokaiser: Is there maybe something like a cache proxy which just wraps a custom python class and caches each calls?

apaszke: 

bodokaiser: Yep. That’s what I was looking for!

EKami: 

gonzales2010: joblib does an amazing job in caching to disk. This removes a lot of hassle and works great for me: <SCODE>from joblib import Memory
cachedir = '/data/cache/'
memory = Memory(cachedir, verbose=0, compress=True)

@memory.cache
def preprocess_file(file, ..params ):
    ... load data and do time consuming pre=processing
    sample['data'] = data
    sample['labels'] = labels
    return sample


class MyData(Dataset):
    def __init__(self, data_path, ... parameters):
        self.files  = list of files

    def __getitem__(self, index):
        sample = preprocess_file(self.files[index],  parameters....)     
        return sample
<ECODE>

Amirhj:
Sharding model across GPUs

claudiomartella: On a similar topic, give a GAN setting with a generator and a discriminator and two GPUs, what is the recommendation to speed-up the computation, given the dependency between discriminator and generator?

ajdroid: <SCODE>class MyModel(nn.Module):
    def __init__(self, split_gpus):
        self.large_submodule1 = ...
        self.large_submodule2 = ...

        self.split_gpus = split_gpus
        if split_gpus:
            self.large_submodule1.cuda(0)
            self.large_submodule1.cuda(1)

    def forward(self, x):
        x = self.large_submodule1(x)
        if split_gpus:
            x = x.cuda(1) # P2P GPU transfer
        return self.large_submodule2(x)
<ECODE>

claudiomartella: 

albanD: Hi,

ajdroid: @claudiomartella
Customized RNN cell which can accept packsequence

ruotianluo: Hi, I saw the packsequence class which presumably can speed up the RNN computation by ignoring the paddings. My question is, if I want to use my own cell, how can I build a RNN module which can accept Packsequence. I’ve looked up the code, but it seems nontrivial. I feel like it should be easily down by a wrapper. Is there any example use case or any suggestions?

smth:
Out of memory when optimizer.zero_grad() is called

david-leon: I’m asking  because I wonder why .zero_grad() would cause memory out. From my understanding this op is just to set param.grad.data to zero, why extra memory would be required?

colesbury: That’s strange. I can’t think of why that would happen.

smth: Is there a small script you can give to reproduce this? I am happy to look into what’s happening.

david-leon: I may need a while to reduce the code as small as possible. The error trace back is as follows: <SCODE>THCudaCheck FAIL file=/data/users/soumith/builder/wheel/pytorch-src/torch/lib/THC/generic/THCStorage.cu line=66 error=2 : out of memory
Warning: out of memory
Warning: out of memory

epoch = 0, loss =2.78738046, ER_train = 100.00, ER_batch = 100.00, time = 2.90s(2.90|0.00), progress = 0.00%, time remained = 1781.43h
epoch = 0, loss =2.77562714, ER_train = 98.44, ER_batch = 96.88, time = 0.73s(0.73|0.00), progress = 0.00%, time remained = 1983.91h
epoch = 0, loss =2.74634695, ER_train = 97.40, ER_batch = 95.31, time = 1.40s(1.40|0.00), progress = 0.04%, time remained = 5.93h

Warning: out of memory

Traceback (most recent call last):
  File "DIC_train_pytorch.py", line 397, in <module>
    optimizer.zero_grad()
  File "/home/David/App/anaconda3/lib/python3.5/site-packages/torch/optim/optimizer.py", line 136, in zero_grad
    param.grad.data.zero_()
RuntimeError: cuda runtime error (2) : out of memory at /data/users/soumith/builder/wheel/pytorch-src/torch/lib/THC/generic/THCTensorMath.cu:35
<ECODE> In the above trace back logs, “Warning: out of memory” is printed by my code to warn me that an out of memory exception (exactly the exception as shown in the last line of the above log) is catched. This exception would be raised by pytorch when input train data batch is big. After catching the exception, I’ll reduce the batch size and try the training procedure again.  The corresponding code snippet is as <SCODE>  optimizer.zero_grad()
    try:
        if device >= 0:
            score = model(Variable(torch.from_numpy(X)).cuda(device))
        else:
            score = model(Variable(torch.from_numpy(X)))
    except RuntimeError as e:
        if e.args[0].startswith('cuda runtime error (2) : out of memory'):
            print('Warning: out of memory')
            cached_data.extend(split_train_data([X, Y]))
            continue
        else:
            raise e<ECODE>

smth: <SCODE>CUDA_LAUNCH_BLOCKING=1 python script.py
<ECODE>

david-leon: The track back is the same, though.

apaszke: We’re probably missing a check somewhere so the error pops up only there. You’re likely working under a super heavy memory pressure, and the model doesn’t fit. What’s the last operation you do (loss fn + last op before)? Did you try reducing the batch size?

AdityaAS: I have the same error. Tried running the model with CUDA_LAUNCH_BLOCKING=1 but still the error pops up at optimizer.zero_grad(). Can anyone help me out? I can post the model and training snippet if needed. Thanks
Embedding returns an error that I dont understand

zeng: <SCODE>a = torch.Tensor(10)
a = Variable(a, requires_grad=True)
embedding = nn.Embedding(10, 100)
output = embedding(a)
<ECODE> <SCODE>---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
<ipython-input-23-11b80614bb73> in <module>()
      2 a = Variable(a, requires_grad=True)
      3 embedding = nn.Embedding(10, 100)
----> 4 output = embedding(a)

/home/zeng/code/tensorfold/lib/python2.7/site-packages/torch/nn/modules/module.pyc in __call__(self, *input, **kwargs)
    200 
    201     def __call__(self, *input, **kwargs):
--> 202         result = self.forward(*input, **kwargs)
    203         for hook in self._forward_hooks.values():
    204             hook_result = hook(self, input, result)

/home/zeng/code/tensorfold/lib/python2.7/site-packages/torch/nn/modules/sparse.pyc in forward(self, input)
     92             padding_idx, self.max_norm, self.norm_type,
     93             self.scale_grad_by_freq, self.sparse
---> 94         )(input, self.weight)
     95 
     96     def __repr__(self):

/home/zeng/code/tensorfold/lib/python2.7/site-packages/torch/nn/_functions/thnn/sparse.pyc in forward(self, indices, weight)
     42     def forward(self, indices, weight):
     43         assert indices.dim() <= 2
---> 44         assert not self.needs_input_grad[0], "Embedding doesn't " \
     45             "compute the gradient w.r.t. the indices"
     46 

AssertionError: Embedding doesn't compute the gradient w.r.t. the indices
<ECODE>

albanD: Hi,

cdjhz: Then, how to get some of the embeddings updated and some fixed?

albanD: If you want to achieve that, I think you will need to set the gradients corresponding to these embeddings to 0 manually after performing the backward pass.

Janinanu: 

albanD: Hi,

Janinanu: Thank you very much! After some adjustments, my code now works.

SHAILESH_KUMAR:
Exploiting sparsity in batch operations?

cjmcmurtrie: If I have a batch of sparse input matrices, and a dense batch of matrices : <SCODE>> mat1 = torch.zeros(4, 3, 5)
> mat1[1][1][1] = 1; mat1[2][2][2] = 1
> mat2 = torch.rand(4, 5, 6)
> torch.bmm(mat1, mat2)
<ECODE> Exploiting sparsity is quite an optimisation. If this isn’t available yet, which users might I liaise with to help out?

ebetica: Also, torch.smm(mat1_i, mat2) is also implemented, for sparse * dense -> sparse operations.

cjmcmurtrie: <SCODE>import torch
import torch.autograd as ann


mat2 = ann.Variable(torch.rand(4, 5, 6), requires_grad=True)
mats = [ann.Variable(torch.zeros(4, 3, 5), requires_grad=True) for _ in range(3)]
for i in range(len(mats)):
    result = torch.bmm(mats[i], mat2)
    print result.size()
<ECODE>

ebetica: Something like this: <SCODE>import torch

x = torch.rand(5,6)
# Sparse matrix of (0, 1) = 1; (2, 1) = 2, (3, 4) = 3
sparse = torch.sparse.FloatTensor(
    torch.LongTensor([[0, 2, 3], [1, 1, 4]]), # Indicies
    torch.FloatTensor([1, 2, 3])) # values

print(x)
print(sparse.to_dense())
print(torch.mm(sparse, x))
# This won't actually save space or compute, since it's so dense,
# but it will be a sparse tensor representation.
print(torch.smm(sparse, x))
<ECODE> Simply construct a list of your sparse tensors, and loop over them to do the batch mm.

cjmcmurtrie: Thanks once more! One last thing: <SCODE>sparse = torch.sparse.FloatTensor(
    torch.LongTensor([[0, 2, 3], [1, 1, 4]]),   # Indicies
    torch.FloatTensor([1, 2, 3]))   # values
<ECODE> Seems strange to me as you don’t define the sizes of the sparse matrix - it seems to arbitrarily pick the indices of the corner value as the size. What is the logic here?

ebetica: 

cjmcmurtrie: 

Ilya_Kostrikov: By the way, is anyone working on wrapping cusparse?

smth: martin raison is working on it, are you interested in collaborating?

Ilya_Kostrikov: Yes, sure! I’m not sure whether I’m familiar with pytorch internals enough to be able help. But I can try anyway.

siddharthachandra: I’d like to collaborate on writing a wrapper for cusparse, if you folks still need a hand.

smth: 

siddharthachandra: Thanks, looks good. Let me check it out.
What’s the equivalence of theano’s inc_subtensor()?

david-leon: To change the values of a subset elements of a tensor, in theano we have inc_subtensor(), what is the equivalence in pytorch?

albanD: Hi,

david-leon: Yes, I noticed there is torch.index_select() function. However this function returns a new tensor not a view, so if I do <SCODE>t2 = torch.index_select(t1,  axis, index)
t2 += 1.0
<ECODE>

smth: you can do standard numpy-like indexing: Try this: <SCODE>t1 = torch.randn(10, 5)
t2 = t1[:, 3]
t2.fill_(0)
print(t1)
<ECODE>

david-leon: 

ypxie: I think index_add_ is what you are looking for.

david-leon: Thanks, that’s exactly what I’m looking for.

apaszke: 

albanD: 

david-leon: 

apaszke: 

david-leon: 

apaszke: 

david-leon:
Easy way to loading localization dataset
Category: vision

Yunhun_Jang: Hi, I’d like to train model with class label and bounding box label at the same time, (e.g. CUB200 2011 dataset, ImageNet 12 CLSLOC dataset) and may preprocess with cropping/scaling (and they should be applied to both image and bounding box label.) Thanks!

amdegroot: Hey,

Yunhun_Jang:
How to train model with multithreading

xwgeng: 

smth: You can look at the hogwild training example we have:

xwgeng: 

apaszke: You can’t share data loaders among processes, it’s going to be very tricky (because of how Python’s multiprocessing works).

xwgeng: Thanks for the reply!

williamdjones: By passing the model as an argument to say the training function, is this not creating a separate local copy  of the model for each process? I ask because I followed the Hogwild! example with my own network/data and have found that while each process tends to have the same optimization behavior, there is variance in the weights at the end of training, at each epoch that I have a checkpoint for. I do expect variance in things such as loss over time because the processes should be loading different random subsets of the same data, but it doesn’t make sense to me why the weights would be different if calling model.share_memory() should be placing the model object in a shared memory segment where there should only be one copy of weights that are manipulated. The only explanation that I can think of as to why my weights/biases would vary across processes would be if the processes are actually only using a local copy of the model…but perhaps I am seriously misunderstanding something here.

williamdjones: to revise this slightly…I believe I understand what the “problem” actually may be. Since it is more likely that the processes finish each epoch at different times rather than all simultaneously at a point in time, and each process is calling optimizer.step() asynchronously then saving the parameters at the end of an epoch…it would make sense that the values of the weights would be slightly different due to this. So at the beginning of the training, all processes should have the same initial weights…but over the course of training it is likely that the processes will not actually have copies of the model that are identical to those of the other processes, but the parameters will generally be more or less similar.
How to add an ignore_label in loss layers?

Chenqi_Qi: 

smth: For example: <SCODE>import torch
import torch.nn as nn

nClasses = 10
ignore_classes = torch.LongTensor([4, 7]) # ignore class 5 and 8
weights = torch.ones(nClasses)
weights[ignore_classes] = 0.0
loss = nn.NLLLoss2d(weights)
<ECODE>
How can I share the variable in LSTM and LSTMCell

Ching_Kit_Wong: Since I am doing machine translation, so I want to train my network with LSTM module. But in testing phase, I want to copy the variable in LSTM to LSTMCell and roll-out the sequence by feeding the last lstm output to the next input.

smth: For this, you dont need to convert LSTM to LSTMCell from my understanding, you just need to make the timesteps of LSTM to be 1 and repeatedly use it.
Quick Detach() Question

ajbrock: If I call detach() on a variable after calling backward(retain_variables=True) on a variable further along the graph, will that free the buffers of the graph that led to the detach()'d variable? Example: net1 and net2 are modules, x is an input variable. I want to retain the state of net2 for processing later down the line, but I just want to accumulate gradients in net1 for the time being, no need to retain the graph. out1 = net1(x) out2 = net2(out1) loss = some_loss_function(out2) loss.backward(retain_variables=True) out1.detach_() Will this clear the buffers of net1? Thanks, Andy

albanD: Hi, <SCODE>out1 = net1(x)

in2 = Variable(out1.data, requires_grad=True)
out2 = net2(in2)

loss = some_loss_function(out2)

loss.backward(retain_variables=True)
out1.backward(in2.grad.data)

# You can call loss.backward again here
# But not out1.backward !
<ECODE>

ajbrock: Hmm, assuming detach() doesn’t free the graph, your way makes a lot of sense, and I think I could wrap that up into this chunk of code neatly. Thanks! Edit: a slightly more elegant way to do this (still following your same idea) might be to copy out1 to in2 with the out-of-place detach: in2 = out1.detach() I’ll try out both and report back.

alexis-jacq: If in2 is detached, you can’t call loss.backward at all, since net2(in2) are not taken into account by the computation of the gradient. When a variable is detached, the backward computations will not visit the branches that start from this variable (all the operations done on it).

apaszke: 

111319: <SCODE>import torch
x = torch.randn(10, 5, requires_grad = True)
w = x * x
a = w*w

a.sum().backward()
print(x.grad.sum())

w = x * x
a = w * w
w.detach_()

a.sum().backward()
print(x.grad.sum())
<ECODE> in this case, the output is: <SCODE>tensor(27.7943)
tensor(55.5887)
<ECODE>

albanD: 

111319: Thanks for replay. But why sometimes inplace operator cause a error in backward and sometimes not?

albanD: Which error do you sometimes see and sometimes not?

111319: Ou, I mean there are some inplace operators will cause an  error in backward, and some other operators will not. Back to the original question, is there a method that can detach a sub graph after a forward?

albanD: No there isn’t I’m afraid. We don’t allow modifying the graph.
Compute and report gradient norm of a hidden variable

dima: Thanks a lot! EDIT: reformulated my question.

dima: Thank you. It works for parameters. As far as I understand, the gradient for non-leaf nodes is deallocated when it is not needed.

dima: Veril, unfortunately not: <SCODE>    x[0].requires_grad = True
RuntimeError: you can only change requires_grad flags of leaf variables.
<ECODE>

apaszke: 

apaszke: It’s not deallocated, it just hasn’t been allocated. We’ve changed the strategy for allocating gradients in the recent release.

dima: All optimizers I know about require gradients wrt parameters which are always leaf nodes. So, it shouldn’t be a problem for implementing an optimizer. In my case I need those gradients only for debugging purposes. For those are interested, I ended up doing like this: <SCODE>class LazyContainer(object):
    def __call__(self, g):
        self.grad_norm = g.norm(2)
container = LazyContainer()
h.register_hook(container)
# later
print(container.grad_norm)
<ECODE>

xylcbd: you can also do like this: <SCODE>A = Variable(torch.ones(2), requires_grad = True)
B = A*2 
B.retain_grad()
C = B.norm()
C.backward()
print B.grad
<ECODE>
Permute elements of a tensor along a dimension

Nabarun_Goswami: Hi, I am trying to permute the feature map dimension in a tensor. As a very simplified case, One possible way I found was to do a index_select followed by cat. But, in case of a larger tensor, a lot of intermediate tensors will have to be created. Is there a better way to achieve this?

apaszke: 

Nabarun_Goswami: OK, so this is what I tried: <SCODE>>>> n = np.arange(480).reshape((5,4,4,6))
>>> a = torch.from_numpy(n)
>>> perm = torch.LongTensor([0,2,1,3])
>>> a[:, perm, :, :]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: indexing a tensor with an object of type LongTensor. The only supported types are integers, slices, numpy scalars and torch.LongTensor or torch.ByteTensor as the only argument.
<ECODE> Shouldn’t perm be a torch.LongTensor? Am I missing something here?

apaszke: 

Nabarun_Goswami: Getting the same error even when I remove the trailing :, :.

Nabarun_Goswami: I can probably do a dimension permute to bring the required dimension to dim0 and do the permutation and dimension permute back to original order. Seems like a hacky way to do it though.

apaszke: Ah yes, you’re right. We should have full indexing implemented soon.

harpone: 

tetratrio: This answer is a bit late but I stumbled upon this thread while trying to create a permutation matrix from permutation indexes so I’ll just share what I found works. <SCODE>permutation_indices = ...
size = permutation_indices.size(0)
permutation_matrix = torch.zeros(size, size)
permutation_matrix[torch.arange(size).long(), permutation_indices] = 1
<ECODE>

harpone: Not quite what I (or the OP) was looking for but thanks for the effort!

tetratrio: Oh yea thats an even simpler approach. Why didnt think of that. Thanks!

Tim_Oleskiw: Yeah as above, permutations can be achieved via matrix multiplication.  [a b] * [0 1; 1 0] => [b a], for example.

cuixing158_1: I do like this:

Daniel_Povey: Regarding how to permute vectors in PyTorch: there seems to be a function pytorch.permute(), but I can’t find any documentation for it, and when I try it doesn’t seem to work as I might expect (seems to be a no-op).

ptrblck: <SCODE>x = torch.randn(1, 2, 3)
print(x.shape)
> torch.Size([1, 2, 3])
x = x.permute(1, 2, 0)
> torch.Size([2, 3, 1])
<ECODE> print(x.shape)

Yuheng_Zhi: Did you want something like this? <SCODE>>>> import torch
>>> x = torch.randn(100, 57, 2)
>>> idx = torch.randint(57, size=(100, 57))
>>> permutated_x = x.gather(dim=1, index=idx.unsqueeze(-1).expand(x.shape))
>>> torch.all(x[0][idx[0]] == permutated_x[0])
tensor(True)
>>> torch.all(x[28][idx[28]] == permutated_x[28])
tensor(True)
<ECODE>
nn.ReLU(inplace=True) make an error

ycszen: <SCODE>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation
<ECODE> But this error just appears in some places. Why it occurs?

apaszke: 

ycszen: Ok. Thank you. It works after removing it.

osuemer: 

apaszke: It is the default value. Just don’t override it.
What is reshape layer in pytorch?

melody-rain: In torch7 it seems to be nn.View, but what it is in pytorch? What I want is to add a reshpe layer in nn.Sequential. Thanks.

apaszke: 

melody-rain: Hi, good example. Thanks. Thank you.

apaszke: 

ypxie: 

smth: 

ypxie: 

allenye0119: <SCODE>import torch.nn as nn

class Reshape(nn.Module):
    def __init__(self, *args):
        super(Reshape, self).__init__()
        self.shape = args

    def forward(self, x):
        return x.view(self.shape)
<ECODE>

ypxie: Thanks~ but it is still so many codes, a lambda layer like the one used in keras would be very helpful.

smth: 

Shisho_Sama: 

Noam_Salomonski: I have to ask why reshaping does not count as “trivial”? Current way of work forces me to separate logic of data flow to two separate places - definition of the nn.Sequential, and forward()

dsantiago: For high level DL, those layers are more confusing than beneficial…

alexp:
Pretrained learning with different input size

neuralix: Error Message in pretraining case: <SCODE>Traceback (most recent call last):
  File "main.py", line 314, in <module>
    main()
  File "main.py", line 68, in main
    model = models.__dict__[args.arch](pretrained=True)
  File "/home/dylee/.conda/envs/pytorch/lib/python2.7/site-packages/torchvision/models/alexnet.py", line 57, in alexnet
    model.load_state_dict(model_zoo.load_url(model_urls['alexnet']))
  File "/home/dylee/.conda/envs/pytorch/lib/python2.7/site-packages/torch/nn/modules/module.py", line 315, in load_state_dict
    own_state[name].copy_(param)
RuntimeError: inconsistent tensor size at /data/users/soumith/miniconda2/conda-bld/pytorch-cuda80-0.1.10_1488756735684/work/torch/lib/TH/generic/THTensorCopy.c:51
<ECODE>

apaszke: No, it’s supported. You must have some bug in your code.

neuralix: <SCODE>    own_state[name].copy_(param)
<ECODE> TO <SCODE>if name != 'classifier.1.weight' and name != 'classifier.1.bias':
    own_state[name].copy_(param)
<ECODE> By skipping parameter copying. I hope it’s helpful for someone.
Is there a way to upsample a image with non int scale_factor?

xiaochao_qu: For example, I want upsample my image with scale_factor 1.5.

apaszke: Do you need this transform to be differentiable, or just to use it as part of the data loading?

smth: <SCODE>import torch
import torch.nn as nn
from torch.autograd import Variable


inp = Variable(torch.randn(10, 3, 24, 24))
m = nn.UpsamplingBilinear2d(size=(55, 55))
out = m(inp)
print(out.size())
<ECODE>

xiaochao_qu: I need it to be differentiable, as I am doing a pixel-to-pixel task which requires the output size is different with the input size.

xiaochao_qu: Thanks. Is this operation differentiable?

smth: 

xiaochao_qu: 

meldefon: This appears to give an error saying that the upsampled size is not divisible by the original size when done with UpsampleNearest2d rather than UpsampleBilinear2d

vinaykumar2491:
Help for DQN (implementation of the paper)
Category: reinforcement-learning

transedward: Sorry, I know it’s really bad to ask help for checking if the implementation is correct. But I am rather new in deep RL, I don’t know if there is any debugging trick or method systematically for checking if my implementation is right. Unlike most supervised learning algorithm, by inspecting loss, we might have some idea about how to tune hyper-parameters (as in cs231n). But we can only see mean reward in RL(as far as I know) and I don’t know how to tune based on that. I tried inspecting the gradients and data before and after the part which I suspected there may contains bugs.

transedward: I got similar results with deepmind paper already.
How to assign gradients to model parameters manually?
Category: reinforcement-learning

onlytailei: 

smth: to answer your second question, you can create a mask tensor and index elements via this mask tensor: example: <SCODE>import torch

x = torch.randn(2, 3, 2)

mask = x.gt(0.5)
print(mask)

# mask is a torch.ByteTensor of same shape and size as x

print(x[mask])
<ECODE> Output: <SCODE>
(0 ,.,.) =
  0  0
  0  0
  0  1

(1 ,.,.) =
  0  1
  1  0
  1  0
[torch.ByteTensor of size 2x3x2]


 1.3140
 0.8220
 1.5169
 0.9264
[torch.FloatTensor of size 4]

<ECODE>

onlytailei: Then I think this method is useless.

apaszke: 

onlytailei: But gather always returns a tensor with same dimension as the input right?

apaszke: Yes, that’s the limitation.