foldl/torch-forum huggingface dataset:
New tutorials page
Category: Site Feedback

chsasank: Hi, We’ve replaced the old notebook based tutorials to better looking sphinx-gallery based tutorials. You can still download notebooks for individual tutorials. Do you have any feedback?

trypag: Nice work ! This work does really matter for the community, thanks !

chsasank: Thanks for the reply. Since I broke down the long and monolithic notebook tutorials into pieces and sections, I was of the impression that they are easier to read. Can you please illustrate your point with an example/screenshot?

trypag: As I said, what I liked in ipython, was that each step was contained in a single cell, then moving to another one meant changing of task. The whole page doesn’t seem as fluid to read as before. I don’t really know how to solve this, maybe it’s okay not having cells anymore and we can move to another design, but I don’t feel really comfortable with it right now. To me it requires thinking of how each step in the tutorial is designed, so that we can follow much easier. A first step may be to join a code panel and its associated result panel, without space between.

chsasank: 

shaun: @chsasank I really like the new tutorials page. Its easily accessible compared to the old one and is part of the documentation as well. Will you be adding more tutorials that others have written and available on github? Also, it’ll be nice to have a tutorial on using torchtext.

chsasank: 

shaun: @chsasank I’ll try to do that. Meanwhile here are my stars on Github, I think some of them are already on the tutorial: Awesome PyTorch 13 The Incredible Pytorch 9 Pytorch Tutorial 10

shaun: Hi, It is not in the “tutorial” format, but I thought I would share it with the community. If there is another way to share personal projects that might be of help to others please let me know. Thanks!

smth: thanks for sharing shaun!
How can I do the operation the same as `np.where`?

wangg12: 

DiffEverything: 

jekbradbury: 

jaromiru: I actually solved it by a workaround: <SCODE># differentiable equivalent of np.where
# cond could be a FloatTensor with zeros and ones
def where(cond, x_1, x_2):
    return (cond * x_1) + ((1-cond) * x_2)
<ECODE>

yxchng: 

truenicoco: 

Pete_Florence: thanks others for examples of other options for 0.3 and before

Pete_Florence: <SCODE>## this example prunes any elements in the vector above or below the bounds
vec = torch.FloatTensor([-3.2, 1000.0, 10.0, 639.0])
lower_bound = 0.0
upper_bound = 640.0
lower_bound_vec = torch.ones_like(vec) * lower_bound
upper_bound_vec = torch.ones_like(vec) * upper_bound
zeros_vec       = torch.zeros_like(vec)

def where(cond, x_1, x_2):
    cond = cond.float()    
    return (cond * x_1) + ((1-cond) * x_2)

vec = where(vec < lower_bound_vec, zeros_vec, vec)
vec = where(vec > upper_bound_vec, zeros_vec, vec)

in_bound_indices = torch.nonzero(vec).squeeze(1)
vec = torch.index_select(vec, 0, in_bound_indices)
<ECODE>

jpeg729: 

Pete_Florence:
How the following two classes interacts?

wasiahmad: I was following the example of SNLI classifier from pytorch official examples and found the following two classes. <SCODE>class Bottle(nn.Module):

    def forward(self, input):
        if len(input.size()) <= 2:
            return super(Bottle, self).forward(input)
        size = input.size()[:2]
        out = super(Bottle, self).forward(input.view(size[0]*size[1], -1))
        return out.view(*size, -1)


class Linear(Bottle, nn.Linear):
    pass
<ECODE> Can anyone share your insight?

pranav: 

DiffEverything: 

pranav: 

jekbradbury: 

pranav: I believe its best if we do not have such concepts at all - just simple Python and letting PyTorch/Model take the center stage
Tensor.sub works, Variable.sub fails with “inconsistent tensor size”

ndronen: <SCODE>import torch
from torch.autograd import Variable
x = torch.rand(1, 2, 3, 4)
# This succeeds.
x.sub(x.max()) 
v = Variable(x, volatile=True)
# This fails with "RuntimeError: inconsistent tensor size at /data/users/soumith/builder/wheel/pytorch-src/torch/lib/TH/generic/THTensorMath.c:827"
v.sub(v.max())<ECODE>

xwgeng: the output of v.max() is Variable with size 1, while x.max() is a scalar

ndronen: That begs the question whether that is the correct behavior.  Is it?

smth: 

ndronen:
Global max pooling?

Fangzou_Liao: Hi, I am looking for the global max pooling layer. Because in my case, the input shape is uncertain and I want to use global max pooling to make their shape consistent. But I do not find this feature in pytorch?

fmassa: You can use the functional interface of max pooling for that. In you forward function: <SCODE>import torch.nn.functional as F
output = F.max_pool2d(input, kernel_size=input.size()[2:])
<ECODE>

Ilya_Ezepov: You can do something simpler like <SCODE>import torch
output, _  = torch.max(input, 1)
<ECODE>
About MaxPool and MaxUnpool

KwuJohn: Hi all, I have got a problem about the pooling function, the code were shown below: <SCODE>input = Variable(torch.rand(1,1,64,64))
pool1 = nn.MaxPool2d(2, stride=2, padding=1, return_indices=True)
pool2 = nn.MaxPool2d(2, stride=2, return_indices=True)
unpool1= nn.MaxUnpool2d(2, stride=2)
unpool2= nn.MaxUnpool2d(2, stride=2, padding=1)

output1, indices1 = pool1(input)
output2, indices2 = pool2(output1)

output3 = unpool1(output2,indices2)
output4 = unpool2(output3, indices1)
<ECODE> How to solve this problem, I want to used the pooling function to get the ouput4 which with the same dimension as input. Thanks a lot!

fmassa: <SCODE>output3 = unpool1(output2, indices2, output_size=output1.size())
output4 = unpool2(output3, indices1, output_size=input.size())
<ECODE>

Yilin_Liu: Hello, even when I’ve specified the size I want, I still got this error…QAQ

coya: Works well for me.Thanks a lot

kshitijpatil09: I was playing around with max_pooling unpooling and the results I found are somewhat weird. Please let me know whether this is the desired behavior of unpooling or am I doing something wrong. These are the results of input size 320

 These are the results of input size 128



DezzardHD: Assuming the kernel size for the pooling operations is 2x2 with stride=(2,2) I guess the output of the MaxPool layer isn’t that confusing if you know how maxpooling works. The positions of the pixels from the Maxpool layer’s output are being placed according to the indices that are being outputted by the MaxPool layer. So it’s quite likely that you won’t get a chessboad like picture back. (That’s what I’ve thought at the beginning.)
How to concatenate list of pytorch tensors?

erogol: This is my current solution <SCODE> data = th.zeros([len(imgs), imgs[0].size()[0], imgs[0].size()[1], imgs[0].size()[2]])
 for i, img in enumerate(imgs): 
       print(img.size())
       print(img.type())
       data[i] = img<ECODE>

fmassa: 

Brando_Miranda: just in case you were wondering about the difference: stack cat

Hiperdyne19012: What if A is of shape (1,3,4) and B is (3,4)? I want the result to be (2,3,4). How do I do this?

harsha_g: <SCODE>a = torch.rand(1, 3, 4)
print(a.shape)
b = torch.rand(3, 4)
print(b.shape)
b = b.unsqueeze(0)
print(b.shape)
c = torch.cat([a, b], dim=0)
print(c.shape)
<ECODE>

Hiperdyne19012: Thank you, it works.
Accessing the Single-Segment Buffer Interface

seba-1511: Hi everyone, I was wondering if there was a way to access the single-segment buffer interface of CUDA Tensors. For example, when using pyCUDA, I used to do: <SCODE>def buff(ary):
    return ary.gpudata.as_buffer(ary.nbytes)
<ECODE> I’d like to use mpi4py with Pytorch, and while I can use the numpy interface for CPU tensors, this is not possible for GPU ones. This would be a nice trick while we wait for the distributed interface to be polished.

smth: you cannot use the buffer interface for CUDA tensors, but you can get the GPU pointer as an int: <SCODE>x = torch.randn(10).cuda()
print(x.data_ptr())
<ECODE>

seba-1511: Thanks for the fast answer.

smth: do you really need a PyBuffer? We attempted implementing the Buffer interface, but it is slightly different across many versions of python and impossible to implement without thousands of lines of code. Cant you work with the data pointer and size of tensor ?

seba-1511: The advantage of the Buffer interface is that mpi4py can take advantage of it. (Essentially making the whole of MPI available) I don’t know of a way to use it directly with address and length. Maybe using custom DataTypes, but I’d need to investigate that. Since I can afford a hacky solution (only need send/recv), I’ll try that while waiting for THDP. In any case, I’ll keep this thread up-to-date.

stsievert: Any updates on this? I’m interested in using mpi4py with Tensors. Does Tensor use the PyBuffer interface?

seba-1511: Not that I am aware of. I (quickly) tried to implement a buffer interface at that time, but was not successful.

stsievert: Does PyTorch work with MPI like other programs would? i.e., would the below work <SCODE>$ mpiexec -n 4 python torch_script.py
<ECODE> <SCODE>torch.distributed.init_process_group(backend='mpi', world_size=4)
<ECODE>

smth: 

Stone: To use MPI you have to build from source, I think it’s kind of annoying.

seliad: So, is there a workaround? torch.distributed MPI operations are very limited (+requires building from source), while mpi4py supports more operations and seems like excellent library.
Why different results when multiplying in CPU than in GPU?

tunante: I am not sure if this is Pytorch related…apologies if not. <SCODE>In [1]: %cpaste
Pasting code; enter '--' alone on the line to stop or use Ctrl-D.
import numpy as np
import torch

a = torch.from_numpy(np.random.rand(5000,100000).astype(np.float32))
b = torch.from_numpy(np.random.rand(5000,100000).astype(np.float32))

c = a.cuda()
d = b.cuda()

print(a.dot(b))
print(c.dot(d))
:<EOF>
124996952.0
124997016.0
<ECODE>

tom: Hi, (I have also run into this e.g. when computing the mean over all images on a dataset with float32.)

tunante: Thanks, you are right about the float64. The number of different digits is similar (depends on the experiment), but they are way more closer numbers. <SCODE>import numpy as np
import torch

a = torch.from_numpy(np.random.rand(5000,100000).astype(np.float64))
b = torch.from_numpy(np.random.rand(5000,100000).astype(np.float64))

c = a.cuda()
d = b.cuda()

print(a.dot(b))
print(c.dot(d))::::::::::
:<EOF>
125000868.65247717
125000868.65247723
<ECODE>

kmichaelkills: CPU: <SCODE>(((a + b) + c) + d)
<ECODE> GPU: <SCODE>((a+b) + (c+d))
<ECODE>

smth: 

tunante: 

smth: it is because GPUs have thousands of cores, and doing a map-reduce style computation best exploits the parallelism of GPUs
Problem reshaping an LSTM output with view() when batch_first=True

miguelvr: Hey there, I’m trying to implement a pytorch module that wraps any module and applies the wrapped module’s operation to every time step of the input. This is pretty much the same as Keras’ TimeDistributed wrapper. To do so, I want to simply reshape the input to two dimensions, apply the operation, and then reshape it back. RuntimeError: input is not contiguous at /Users/soumith/code/pytorch-builder/wheel/pytorch-src/torch/lib/TH/generic/THTensor.c:231 Hopefully, someone here can help me solve this problem… Cheers

jekbradbury: 

ecolss: @jekbradbury <SCODE>rnn = th.nn.LSTM(5, 10, 1, batch_first=True)  # batch_first

x = th.autograd.Variable(th.randn(1, 2, 5))  # batch size 1
out, state = rnn(x)
print(out.is_contiguous())    # TRUE

x = th.autograd.Variable(th.randn(2, 2, 5))  # batch size 2
out, state = rnn(x)
print(out.is_contiguous())    # FALSE

rnn = th.nn.LSTM(5, 10, 1)   # seq_len first

x = th.autograd.Variable(th.randn(2, 1, 5))  # batch size 1
out, state = rnn(x)
print(out.is_contiguous())    # TRUE

x = th.autograd.Variable(th.randn(2, 2, 5))  # batch size 2
out, state = rnn(x)
print(out.is_contiguous())    # TRUE<ECODE>

jekbradbury:
How to turn list of varying length tensor into a tensor

Shihan_su: It seems that I cannot create tensor with varying length on any dimension.

smth: 

ZeweiChu: These two functions help you pack (already) padded sequences and pad (already) packed sequences. I think the purpose is not to help you pad sequences, but to save some spaces. Check out the following example:

apaszke: 

ecolss: 

smth: 

ecolss: As illustrated in the release notes, PackedSequence holds data and a list of sequence sizes of a packed sequence batch. Then why would we have to first pad the seq to equal length manually and then pass it to pack_padded func to generate this PackedSequence object? Wouldn’t it be easier to just construct the object directly from the original seq without manual padding?

lgelderloos: 

ecolss: I’m not following up on this, but I agree with you on this.

nilbot: 

RylanSchaeffer: For anyone wondering if PyTorch has created a function that takes a list of tensors and pads them to be the same length, the following function does the trick: Example: <SCODE>>>> import torch
>>> l = [torch.Tensor([1., 2.]), torch.Tensor([3.])]
>>> torch.nn.utils.rnn.pad_sequence(l, batch_first=True, padding_value=0)
tensor([[1., 2.],
        [3., 0.]])
<ECODE>
ImageNet example is crashing

FuriouslyCurious: <SCODE>Process Process-4:
Traceback (most recent call last):
  File "/conda3/envs/idp/lib/python3.5/multiprocessing/process.py", line 249, in _bootstrap
    self.run()
  File "/conda3/envs/idp/lib/python3.5/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/conda3/envs/idp/lib/python3.5/site-packages/torch/utils/data/dataloader.py", line 36, in _worker_loop
    data_queue.put((idx, samples))
  File "/conda3/envs/idp/lib/python3.5/multiprocessing/queues.py", line 349, in put
    obj = ForkingPickler.dumps(obj)
  File "/conda3/envs/idp/lib/python3.5/multiprocessing/reduction.py", line 50, in dumps
    cls(buf, protocol).dump(obj)
  File "/conda3/envs/idp/lib/python3.5/site-packages/torch/multiprocessing/reductions.py", line 113, in reduce_storage
    fd, size = storage._share_fd_()
RuntimeError: unable to write to file </torch_6487_1133870694> at /data/users/soumith/miniconda2/conda-bld/pytorch-cuda80-0.1.10_1488757768560/work/torch/lib/TH/THAllocator.c:267<ECODE>

tom: Hi, Best regards Thomas

Nord786: 

ml9951: This solved it for me.  Thanks!
How convert a list of torch.LongTensor to one torch.LongTensor?

zeng: I want to pull a batch of Variable to one Variable, or one Tensor?[ type = ‘list’  ----> type = ‘torch.LongTensor’] How can I deal with this problem.

albanD: Hi,

SpandanMadan: You may want to use torch.stack and torch.squeeze in conjunction. The first converts a list of tensors to a single stacked tensor, and the latter removes all dimensions=1, thus making your data compact.
Training on Caltech101: Not Learning

dlmacedo: Dear friends, I am trying to adapt the ImageNet example to use PyTorch to train on Caltech101, an also important dataset. I am using ImageFolder just like it is used in ImageNet example, but the model VGG from vision.models is not learning (Loss is not going to zero): Question: Do I need to explicit convert JPG to BMP in some way? Or this should not be necessary since we are using PIL library anyway? Thanks in advance, David

smth: 

QuantScientist: Were you able to do this at the end? I am trying to do the same. Thanks.
How to reach model attributes wrapped by nn.DataParallel?

erogol: What is the way to reach model attributes inside DataParallel?

andyhx: i come across the same problem as u  met,how do u solve it,thank u

fmassa: You can access them by using <SCODE>model.module.rnn
<ECODE> but it’s not exposed in the public API, so it might change in the future.

oneTaken: 

MrTuo: What if I chage the attribute by this way? Will the models on other GPUs change synchronously?

Victor_Maldonado: <SCODE>    device =  "cuda:0"

    graph = Text2Mel().to(device)
    # set the training flag
    graph.train()
<ECODE> i tried ‘dataparallel’ but i’m noob in pytorch and alway i have this error: <SCODE>AttributeError: 'DataParallel' object has no attribute 'attention'
<ECODE> I would appreciate any help, thanks!

hobi: <SCODE># simple fix for dataparallel that allows access to class attributes
class MyDataParallel(torch.nn.DataParallel):
    def __getattr__(self, name):
        try:
            return super().__getattr__(name)
        except AttributeError:
            return getattr(self.module, name)
        
#     def __setattr__(self, name, value):
#         try:
#             return super().__setattr__(name, value)
#         except AttributeError:
#             return setattr(self.module, name, value)

def load_weights(base_model_name, model, epoch):
    """
    Loads previously trained weights into a model given an epoch and the model itself
    :param base_model_name: name of base model in training session
    :param model: the model to load weights into
    :param epoch: what epoch of training to load
    :return: the model with weights loaded in
    """

    pretrained_dict = torch.load('{}/{}_{}_{}.pt'.format(args.model_path, base_model_name, epoch, args.lr))['state_dict']
    model_dict = model.state_dict()
    pretrained_dict = {k: v for k, v in pretrained_dict.items() if k in model_dict}
    model_dict.update(pretrained_dict)
    model.load_state_dict(model_dict)

    return model

model = r2plus1d50()
model = MyDataParallel(model, device_ids=[0, 1])

# Copying the layer from epoch 13 to model
epoch = 13
head_model = r2plus1d50()
head_model = MyDataParallel(head_model, device_ids=[0, 1])
head_model = load_weights(base_model_name, head_model, epoch)
# head1 is the name of the attribute that we want to copy over
head = getattr(head_model, "head1")

# set attribute to change the attribute
setattr(model.module, "head1", head)

<ECODE>
Cublas runtime error : library not initialized at /data/users/soumith/builder/wheel/pytorch-src/torch/lib/THC/THCGeneral.c:383

Dennis: Thank you

wasiahmad: I am also getting the same error when I run my code in multiple GPUs. But the error is not consistent, sometime I get it, sometime not.

Yufeng_Ma: I am facing the same error. So is this related with machines that have multiple GPUs?

Yufeng_Ma: I think I’ve found the workaround. When we do .cuda(), we may specify the GPU device we want to load data or model to make sure they are on the same GPU.  For example, <SCODE>net = Net()
net = net.cuda( 0 )

input = Variable(torch.randn(1, 1, 32, 32))
input = input.cuda( 0 )
output = net(input)
<ECODE>

Rohith_AP: 

ShawnGuo: 

ShawnGuo: I’m also facing the same issue. Although I removed the cache files in .nv directory, same error would be raised when running my code.

ws123: Thanks!

vince62s: any help ?

smth: CUDA_VISIBLE_DEVICES is 0-indexed. PyTorch is also 0-indexed.

vince62s: 

lifematrix: Steven

oneTaken: I’m finetuning the vgg19_bn on my own dataset, and I faced the same problem too.

zjtgit: Thanks! It worked for me.

guoqiang_Wei: 

tiantong: when I use pytorch 0.3, it works. But when I use 0.4 compiled from master, my code throws this error. remove nv doesn’t work.

xypan1232: 

Brando_Miranda: whats the source of this problem?

Brando_Miranda: is there not a way to set these indices globally once for everything?

Brando_Miranda: I think: <SCODE>export CUDA_VISIBLE_DEVICES=$i
<ECODE> is what Im looking for.
Does RNN work with DataParallel wrapper?

erogol: If I wrap RNN with DataParallel, it seems like output is not consistent with the target size. For instance, if the batch size is 32 and 2 gpus are active then 16 instances per gpu are processed. However, these instances should be aggregated in the end to get the whole batch of 32 instances for  loss function. But when I use rnn, aggregation is not happening and model outputs only 16 instances which is conflicting with the target value size. I don’t know it makes sense ?

smth: 

erogol: This is strange. If custom RNN module returns the both RNN state and the output, DataParallel does not work and the problem above appears. If you just return output of RNN then things are fine
ImageFolder() hangs in wait forever

FuriouslyCurious: I tried to use Amos’ DenseNet and Soumith’s ImageNet examples and replaced CIFAR10 dataset with ImageFolder dataset: Original: Modified: But loading process hangs forever.  Keyboard interrupt shows:

bodokaiser: 

FuriouslyCurious: Thanks!

bodokaiser: Some things I would try:

FuriouslyCurious: Thanks Kaiser - I am starting debugging now and will look in to all three points you mentioned.

apaszke: Also, are you running your script in Docker?

FuriouslyCurious: Looks like models are hardcoded with 224x224 dimensions.  I am looking into it now.
Segmentation fault when loading weight

david-leon: Previously this runs with no problem, actually two training processes are still running (on another two GPUs), however this breaks when I want to start an additional training process.

smth: it seems that you are running out of memory on the GPUs

david-leon: No, the target GPU is idle, and there’re still 22GB memory available on this GPU.

david-leon: 

david-leon: I’ve tried this: <SCODE>    def map_loc(storage, loc):
        if loc.startswith('cuda'):
            return storage.cuda(device)
        else:
            return storage
    print('model weights loading...')
    model.load_state_dict(torch.load(model_file,map_location=map_loc))
    print('model weights loaded')
<ECODE> And still, exception raised: <SCODE>model weights loading...
THCudaCheck FAIL file=/data/users/soumith/builder/wheel/pytorch-src/torch/csrc/generic/serialization.cpp line=145 error=2 : out of memory
Traceback (most recent call last):
  File "PTR_evaluation_pytorch.py", line 197, in <module>
    model.load_state_dict(torch.load(model_file,map_location=map_loc))
  File "/home/David/App/anaconda3/lib/python3.5/site-packages/torch/serialization.py", line 222, in load
    return _load(f, map_location, pickle_module)
  File "/home/David/App/anaconda3/lib/python3.5/site-packages/torch/serialization.py", line 377, in _load
    deserialized_objects[key]._set_from_file(f, offset)
RuntimeError: cuda runtime error (2) : out of memory at /data/users/soumith/builder/wheel/pytorch-src/torch/csrc/generic/serialization.cpp:145
<ECODE> The target device is idle with over 20GB memory free.

smth: there was a bug in the serialization where remapping devices still used the device memory. this is fixed in master. i am working on binaries of version 0.1.11 and that will have this fix.

david-leon: 

Zhang_Wen: 

will_soon: Is because the version of pytorch?

todpole3: Met the same problem today.

Jonson: I got the same problem, do you figure out how to solve it yet?

ptrblck: 

Jonson: 

Debatrix: <SCODE>if torch.cuda.is_available() and cfg.use_gpu is not None:
    device = torch.device(use_gpu)
else:
    device = torch.device("cpu")
checkpoint_data = torch.load(checkpoint_path, map_location=device)
model.load_state_dict(checkpoint_data['model'])
optimizer.load_state_dict(checkpoint_data['optimizer'])
<ECODE> <SCODE>checkpoint_data = dict(
                    optimizer=optimizer.state_dict(),
                    model=model.state_dict(),
                )
<ECODE>
Converted Trained PyTorch Model to Caffe

sauhaardac: Is there a way to convert a pretrained pytorch model to use in inference in Caffe? I torch.save() the dictionary of the net. How can I get the trained model loaded into Caffe?

smth: at this time, there is no such conversion tool.

wk910930: 

sauhaardac: Nope, I actually ended up converting all our inference code to PyTorch and my entire lab followed suit. No regrets, Pytorch is awesome!