foldl/torch-forum huggingface dataset:
Tensor slice in pytorch?

gaoking132: Is there any function that behaves similarly to the function of tf.slice in pytorch?

jekbradbury: 

gaoking132: I just found a similar post regarding using indexing to slice a tensor. So my question is “Is index slicing same as using narrow function?” Thanks

fmassa: 

gaoking132: Thanks. This is what I mean.

Rohit_Keshari: Please let me know if I am doing something wrong

nicozhou: so, how do we slice the tensor become patches using narrow?
Tensors are on different GPUS

ycszen: <SCODE>img = Variable(img).cuda()
label = Variable(label).cuda()
<ECODE> <SCODE>Traceback (most recent call last):
  File "trainer.py", line 53, in <module>
    outputs = resnet(rmap)
  File "/usr/local/lib/python2.7/dist-packages/torch/nn/modules/module.py", line 202, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/torch/nn/parallel/data_parallel.py", line 92, in forward
    outputs = self.parallel_apply(replicas, scattered, gpu_dicts)
  File "/usr/local/lib/python2.7/dist-packages/torch/nn/parallel/data_parallel.py", line 102, in parallel_apply
    return parallel_apply(replicas, inputs, kwargs)
  File "/usr/local/lib/python2.7/dist-packages/torch/nn/parallel/parallel_apply.py", line 50, in parallel_apply
    raise output
RuntimeError: tensors are on different GPUs
<ECODE>

smth: 

ycszen: 

nicklhy: I met the same problem in my program yesterday. It seems that this error only happens when device_ids[0] is not 0. <SCODE>(Pdb) l
 29         for param in module.parameters():
 30             if param in seen_params:
 31                 continue
 32             seen_params.add(param)
 33 B           param_copies = Broadcast(device_ids)(param)
 34  ->         for param_copy, remap in zip(param_copies, param_remap):
 35                 remap[param] = param_copy
(Pdb) p param_copies[0].get_device()
0
(Pdb) p param_copies[1].get_device()
3
(Pdb) p device_ids
[2, 3]
<ECODE> I am still working on this and have not got any solutions so far.

melody-rain: I got the same problem. Is it a bug of PyTorch?

nicklhy: 

mehdi-shiba: 

nicklhy: Please change your code to xx.cuda(device=gpus[0]) instead of xx.cuda(device=0).

nicklhy: By moving all variable tensors and models into gpus[0](not gpu 0), the first model replica will be on the right device you selected(gpus=[xx, xx, xx]) even though the broadcast function will still ignore it.

mehdi-shiba: 

lakehanne: has this been fixed in PyTorch?

hyqneuron: What’s wrong??

smth: 

stacked.twix: 

botcs: In case if I have, what is the solution?

jarrelscy: My 2 cents worth. I’m used to creating lists of Conv objects as I did in Keras and placing them into a class attribute. Turns out this doesn’t work if you’re using PyTorch and Data parallel as it has to be added using add_module.

smth: 

jarrelscy: Thanks! That is a great tip.

kits: I am also curious what to do if my model has tensor and variables used in self?

furlat:
PyTorch example with CNN-based object detection

pevogam: Hi guys, I was wondering is there any example or at least pull request in progress regarding a PyTorch example with CNN-based object detection? For instance a short enough code on the COCO detection dataset? I looked in the examples on GitHub but at least I couldn’t find anything similar.

fmassa: 

YossiB: switch between meta-architectures (SSD/Faster-RCNN) and base architectures (Resnet/Inception/…) test training tricks such as hard-negatives mining many other tweaks relevant for object detection

fmassa: 

skrish13: citing few more (prolly some better too)

chenyuntc: 

Amir_Rosenfeld: 

ignacio-rocco: @Amir_Rosenfeld

Amir_Rosenfeld:
Resnet101 performs differently at different batch_size

Rigel_Hsu: I use resnet pretrained model to fine tune my own dataset. Everthing seems ok but I found that at the testing phase, the result is different due to different batch_size setting of test_loader. I change the pretrained model to alexnet, everthing goes well and the result is consistent. Why?

fmassa: 

Rigel_Hsu: Sorry, I forgot it. Thanks for your help, it fixes my problem.
Monte Carlo dropout and independent dropout masks within mini batches?

mattmacy: One could parallelize MCD inference by having multiple instances of a given item in a mini-batch. However, in order for that to work the dropout masks have to be independent for all the members of a given mini-batch. Normally it would be faster to re-use a mask across members so I’m curious how it’s done in torch. Thanks.

ngimel: In pytorch dropout masks are independent for all the samples in the minibatch (a single mask the size of the whole input is generated).

mattmacy: Good to hear. Thanks.
What does next(self.parameters()).data mean?

wasiahmad: <SCODE>def init_weights(self, bsz):
    """Initialize weight parameters for the encoder."""
    weight = next(self.parameters()).data
    num_directions = 2 if self.bidirectional else 1
    if self.rnn_type == 'LSTM':
        return (Variable(weight.new(self.n_layers * num_directions, bsz, self.hidden_size).zero_()),
                Variable(weight.new(self.n_layers * num_directions, bsz, self.hidden_size).zero_()))
    else:
        return Variable(weight.new(self.n_layers * num_directions, bsz, self.hidden_size).zero_())
<ECODE> I have two question.

Response777: 

manal_elaidouni: <SCODE>class Model(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc = nn.Linear(10,1)
        
model = Model()
weight = next(model.parameters()).data

print(weight)
# Output :  tensor([[ 0.0398,  0.0729, -0.2676,  0.2354, -0.0853,  0.1141,  0.0297,  0.0257, \ 
         -0.1303,  0.2208]])

print(type(weight), weight.device, weight.dtype, weight.requires_grad)
# Output : (torch.Tensor, device(type='cpu'), torch.float32, False)

var = weight.new(4,5)

print(type(var), var.device, var.dtype,  var.requires_grad)
# Output : (torch.Tensor, device(type='cpu'), torch.float32, False)

<ECODE>
How to initiate parameters of layers

alan_ayu: 

chenyuntc: <SCODE>m = t.nn.Conv2d(16, 33, 3, stride=2)
m.weight.data.normal_(0, 0.01)
m.bias.data.fill_(0)
<ECODE> <SCODE>m = t.nn.Conv2d(16, 33, 3, stride=2)
xavier_uniform(m.weight.data)
<ECODE>

alan_ayu:
Tensors as arguments

lbollar: Hello, I am new to Pytorch so sorry for any naiveté or ignorance. I am doing a little bit of experimenting replicating the functionality of distributions in tensorflow.contrib.  While working on a sample generator for random normal, I tried to pass in first erroneously a torch.Size type to randn and then after converting this to tensor, still received an error. So the args supplied to randn need to be integers then. I notice that a lot of tensorflow arguments are tensors. Let’s say I store or derive through a function call the shape or size of my normal distribution parameters as regular integers. Is there possibly a performance hit that can take place when things get more complex by having to derive and pass in integers instead of working with tensors as arguments?

smth: <SCODE>x = torch.randn(10, 20)
y = torch.randn(*x.size()) # should work
<ECODE>
Deep/Sharp Mask or Mask R-CNN
Category: vision

ceberly: Greetings and apologies if this is just a lazy question, but has anyone implemented either of the FAIR deep/sharp mask or the more recent r-cnn papers? Thanks, really enjoying learning about pytorch.

smth: From what I know, there are no public implementations of these papers in pytorch yet.

ceberly: 

smth: i’m also thinking of hacking up MaskRCNN (but not deepmask / sharpmask). Maybe we could work off a common repo.

ceberly: I would love that. Not sure if I’m qualified to help but it would be a good learning experience.

bfortuner: I would be interested, too. Although from the looks of it, it won’t be a trivial project. It will be great to implement the instance segmentation component. I haven’t found that piece in any of the existing implementations of fast/faster RCNN. Everyone seems to stop at object detection. Also the pytorch ecosystem for segmentation is a bit small, so this could be a big win for pytorch getting a state of the art segmentation model into the model zoo.

Akshay_Deep_Lamba: 

deepcode: did anyone start this? I tried getting either of these 2 work, but they dont work: I would love to  be able to do this in pytorch.

Tgaaly: 

tshrjn: Any update on Pytorch implementation of Mask RCNN ?

Cat: Commenting for updates

pfis: Maybe someone can give me a hint on what this modification is for and how it works or a paper which descripes it. Thanks in advance

rohun: Commenting for updates. Did anyone try this?

jvj_iit: I’m trying to implement Deepmask with Pytorch, so far I have defined the Joint Loss Function, and the model’s learn-able parameters. I was working on the training phase, and as the paper says that training must be done in an alternative back-propagation fashion across the two branches, I have written the code for the same. But there is some problem with training, I tried to train the model with a Fake Dataset, for minibatches other than the first mini-batch the loss of the model is turning out to be NAN. Can somebody help me with this?
Why torch.cat() makes a cuda Variable to a non-cuda Variable?

wasiahmad: I am trying to run a code where I am getting error in the following code snippet. <SCODE>combined_representation = torch.cat([self.encoder_hidden_states1[last_time_step_sent1][0],
                                 self.encoder_hidden_states2[last_time_step_sent2][0]], 1)
if self.config.cuda:
     combined_representation = combined_representation.cuda()

print(combined_representation.size()) # prints torch.Size([16, 600])
scores = self.linear(combined_representation)
<ECODE>

smth:
Support for bidirectional_dynamic_rnn?

mattmacy: Has any work been done on supporting dynamic unrolling of inputs as in TF’s {bidirectional_}dynamic_rnn?

jekbradbury: <SCODE>#TensorFlow (should be run once, during `__init__`)
cond = lambda i, h: i < tf.shape(words)[0]
cell = lambda i, h: rnn_unit(words[i], h)
i = 0
_, h = tf.while_loop(cond, cell, (i, h0))


#PyTorch (should be run for every batch, during `forward`)
h = h0
for word in words:
    h = rnn_unit(word, h)
<ECODE>

mattmacy: Thanks. The python/ops/rnn.py code is so involved / convoluted I thought that there must be something more going on that that.

mattmacy: 

ngimel: 

elanmart: Also, note that you can use pack_padded_sequence 93 to allow sequences of different length inside a minibatch it’s perfectly fine to use minibatches with different sequence lengths in every call to LSTM If your application allows that, using nn.LSTM instead of manually unrolled nn.LSTMCell you can easily observe a 10x speedups.

mattmacy: Can I just pack/pad the sentences in the minibatch and feed that to a BiLSTM?

ngimel:
torch.nn.DataParallel does not accept None for device_ids in 0.1.11

jtremblay: I just updated to .11 and I think the api for calling torch.nn.DataParallel has changed. In the previous version if I only had one GPU I would call the function with None passed as the device_ids. Now if I pass None I get the following error: <SCODE>  File "/home/jtremblay/anaconda2/envs/py3/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 96, in data_parallel
    output_device = device_ids[0]
TypeError: 'NoneType' object is not subscriptable
<ECODE>

meijieru: 

smth: Here is the script I used, along with it’s output: <SCODE>import torch
from torch.autograd import Variable
import torch.nn as nn
import platform

print('Python version: ' +  platform.python_version())
print(torch.__version__)


print('Trying out device_ids=None')

model = nn.Linear(20, 30).cuda()
net = torch.nn.DataParallel(model, device_ids=None)


inp = Variable(torch.randn(128,20).cuda(), requires_grad=True)
out = net(inp)
out.backward(torch.ones(out.size()).cuda())

print('Passed')
<ECODE> Output: <SCODE>Python version: 3.6.0
0.1.11+b13b701
Trying out device_ids=None
Passed
<ECODE>

smth: For now, you can do: device_ids = list(range(torch.cuda.device_count()))

jtremblay: Sorry for the late reply I am travelling. I should have provided an example or do a PR. I have been using the ids from now. It was an easy fix for upgrading my scripts. But thank you so much for your time.

raaj043: Hello, I am getting Torch: unable to mmap memory: you tried to mmap 0GB error. I have 12 GB RAM, 1 GPU core and the datasize is 7GB. Ideally it should not give this error. I think i am making mistake in cuda and dataparallel, but unable to figure it out.  Attached image contains the details. Please help!!
[resolved] PySlice_AdjustIndices Error

Seungyoung_Park: I am using python 3.6 on Ubuntu 16.04. I have just updated pytorch via conda. But, I got an following error when torch was imported. Please help me how to solve this problem.

smth: this makes no sense yet, because python 3.6 should be having that. Let’s try to figure this out. Can you please paste your output of the following command: <SCODE>ldd /home/sypark/anaconda3/lib/python3.6/site-packages/torch/_C.cpython-36m-x86_64-linux-gnu.so
<ECODE>

Seungyoung_Park: He had the same problem. The output is below.

smth: it would help me if you gave the output of the command: <SCODE>python --version
<ECODE> Thanks.

Seungyoung_Park: My python version is below. <SCODE>Python 3.6.0 :: Anaconda 4.3.1 (64-bit)
<ECODE> Also, I updated pytorch by the command as <SCODE>conda update pytorch torchvision -c soumith
<ECODE> Thanks.

smth: this is helpful, thank you. I’ll try to fix this error by tomorrow in the new binaries.

Seungyoung_Park: Thanks for your help!!

smth: 

Seungyoung_Park: I will check it. Thanks for your help.

Seungyoung_Park: Thanks.

smth: what is your application? is your code available somewhere to look?

Seungyoung_Park: My application is Inception-resnetv2 based autoencoder. I would like to share my code to solve this problem, but I am not allowed to do this by the company for a while. I will post the github address if I am allowed. Sorry.

mderakhshani: So what shoud i do then? How can I solve this?

leegang: 

smth: 

leegang: conda install pytorch torchvision cuda80 -c soumith @smth

smth: that is weird. what is your output of: <SCODE>python --version
<ECODE>

leegang: 

smth: <SCODE>conda uninstall pytorch
conda install pytorch -c soumith
<ECODE>

leegang: <SCODE>[ec2-user@ip-172-31-40-200 Notebooks]$ conda install pytorch -c soumith
Fetching package metadata .....
......
Solving package specifications: .

Package plan for installation in environment /home/ec2-user/anaconda3:

The following NEW packages will be INSTALLED:

    pytorch: 0.1.11-py360_4cu80 soumith [cuda80]

Proceed ([y]/n)?
y

ec2-user@ip-172-31-40-200 Notebooks]$ conda list | grep pytorch
    pytorch                   0.1.11              py360_4cu80  [cuda80]  soumith<ECODE>
Extract sub region of conv feature map

Rigel_Hsu: Is there any module in pytorch can extract sub region of conv feature map such as spatial transformer (given the affine matrix) or roi pooling layer(given the window’s 4-tuple coordinates)?

fmassa: 

blackyang: 

blackyang: 

simo23: the rois are defined as [roi_id , left_coord , top_coord, right_coord, bot_coord] = [ roi_id , x1, y1 , x2, y2 ] I noticed that it is not mentioned in the code, so maybe it can help someone.
What should I do if the output of my model can’t be the input of the loss function?

pointW: I’m trying to implement a policy gradient method in RL and the output of my model need some more calculations before computing the loss. What should I do with my output and the loss function in such case?

AjayTalati: Maybe posting some code of the calculations you want do might be helpful to understand what you want to do?

pointW: Here is part of my codes: <SCODE>    self.steps_done = 0
    self.matches_done = 0
    self.win_count = 0

def forward(self, x):
    x = F.relu(self.conv1(x))
    x = F.relu(self.conv2(x))
    x = F.relu(self.conv3(x))
    x = F.relu(self.conv4(x))
    x = F.relu(self.conv5(x))
    x = F.relu(self.conv6(x))
    x = F.relu(self.conv7(x))
    x = x.view(x.size(0), -1)
    x = F.softmax(x)
    return x
<ECODE> <SCODE>output = model(Variable(epstate.type(dtype)))

discounted_epr = discount_rewards(epreward)
discounted_epr -= torch.mean(discounted_epr)
discounted_epr /= torch.std(discounted_epr)

discounted_epr.resize_(discounted_epr.size()[0], 1)
discounted_epr = discounted_epr.expand(discounted_epr.size()[0], 81)

epy = Variable(epy, requires_grad=False)
discounted_epr = Variable(discounted_epr, requires_grad=False)

loss = (epy - output).mul(discounted_epr).pow(2).mean()

optimizer.zero_grad()
loss.backward()
optimizer.step()<ECODE>

pointW: 

AjayTalati:
How to repeat a vector batch wise?

AjayTalati: <SCODE>>>> images_vec.size() 
torch.Size([100, 784])
<ECODE> <SCODE>torch.Size([100, 784, 28])
<ECODE> Here’s the batch loading code, <SCODE>import torch 
import torchvision.datasets as dsets
import torchvision.transforms as transforms

# Hyper Parameters
sequence_length = 28
input_size = 28*28
batch_size = 100

# MNIST Dataset
train_dataset = dsets.MNIST(root='../data_tmp/',
                            train=True, 
                            transform=transforms.ToTensor(),
                            download=True)

# Data Loader (Input Pipeline)
train_loader = torch.utils.data.DataLoader(dataset=train_dataset,
                                           batch_size=batch_size, 
                                           shuffle=True)

for i, (images, labels) in enumerate(train_loader):
    #images_repeated = images.view(-1, sequence_length, input_size) 
    images_vec = images.view(-1, input_size)
    #images_repeated = torch.randn(100,784,28) # pre-allocate
    images_repeated = torch.randn(batch_size, input_size, sequence_length) # pre-allocate
    #for j in range(sequence_length):
        #images_repeated [,,j] = images_vec
    y_true = labels
    if i == 1:
        break

images.size()
images_vec.size() 
<ECODE> Cheers, Aj

AjayTalati: OK, answer my own question, this seems to work? <SCODE>>>> images_repeated = images_vec.repeat(1,sequence_length) 
>>> images_repeated = images_repeated.view(-1,input_size, sequence_length)
>>> images_repeated.size()
torch.Size([100, 784, 28])
<ECODE> I guess it could be checked by something like this? <SCODE>images_repeated[99,783,0] == images_vec
<ECODE> It should be all true? I expect there must be a more robust or cleaner way to do all this though?

elanmart: <SCODE>X = torch.randn(100, 700)
X = X.unsqueeze(2).repeat(1, 1, 28)
<ECODE> Or you can use expand: this will only create a view, without copying any data. <SCODE>X = torch.randn(100, 700)
X = X.unsqueeze(2).expand(100, 700, 28)
<ECODE>

AjayTalati: 

AjayTalati: What I mean is, <SCODE>X = torch.randn(100, 700)
X_tmp = X.unsqueeze(2).repeat(1, 1, 28)

X[1] == X_tmp[1,:,0] # should be all true - want to check it vectorially
<ECODE>

elanmart: 

AjayTalati: 

tom: If you want to stay in pytorch (e.g. for GPU arrays), you can do (X[1] != X_tmp[1,:,0]).sum()==0 . The caveat is that NaN != NaN (X != X seemed to be easiest way to check for NaN in pytorch a while back). Best regards Thomas

AjayTalati: 

feipan664: 

Honghe: <SCODE>X = torch.randn(100, 700)
X = X.unsqueeze(2).expand(100, -1, -1)
<ECODE>
Do any PyTorch AMIs exist on AWS?

ethancaballero: I searched pytorch and didn’t see any.

elanmart: Edit: ofc. if you don’t need gpu, anaconda alone will suffice.

ethancaballero: Ubuntu 16.04 + Python3 + PyTorch ami just appeared:

mjdietzx: 

smth: 

smth: i dont know why you have to download CUDA (or especially CuDNN). We already ship both with our binaries. The only thing required is an NVIDIA driver on the machine.

smth: 

b0noi:
Copying weights from one net to another

rbrigden: Is there a canonical method to copy weights from one network to another of identical structure?

fmassa: <SCODE>net1.load_state_dict(net2.state_dict())
<ECODE>

FuriouslyCurious: How does deep copy / canonical copy differ from normal weights loading?

fmassa: it avoids you from having to do <SCODE>model1 = Model()
model2 = Model()

model2.load_state_dict(model1.state_dict())
<ECODE> and instead you only do <SCODE>model1 = Model()
model2 = copy.deepcopy(model1)
<ECODE>

shubhamjain0594: As far as I have seen the code “load_state_dict copies only parameters and buffers”. Does deepcopy also copies only _parameters and _buffers or the hooks as well?

qq456cvb: deep copy will recursively copy every member of an object, so it copies everything

spnova12: If using deepcopy in my experience, optimizer does not work …

Navneet_M_Kumar: Example: weights_new = k*weights_old + (1-k)*weights_new How can I do this?

bhushans23: 

D-X-Y: Hi, have you found a effective way to do this thing?

Navneet_M_Kumar: I am doing this for each layer. I believe there must be a better method but this works for now.

Navneet_M_Kumar: 

roee: I think you need to reinitialize the optimizer using the new copied model and then you can copy the optimizer inner values from one to the other, a bit of a mess. I would probably stop at reinitializing the optimizer.

bhushans23: 

roee: Less messy than what I assumed, good to know! thx!

Brando_Miranda: does one know to to properly solve the polykia averaging issue mentioned? The solution mentioned doesn’t work I believe:

Navneet_M_Kumar: <SCODE>def polyak_update(polyak_factor, target_network, network):
    for target_param, param in zip(target_network.parameters(), network.parameters()):
        target_param.data.copy_(polyak_factor*param.data + target_param.data*(1.0 - polyak_factor))
<ECODE>

Krishna_Garg: load_state_dict worked for me. copy.deepcopy failed for me with run time error “Only Tensors created explicitly by the user (graph leaves) support the deepcopy protocol at the moment”