foldl/torch-forum huggingface dataset:
Gradient computed on CPU but nor computed on GPU?

Donglai_Xiang: I am writing some code for something similar to RoI pooling. The gradient propagates back well when I use CPU but not on GPU? Does anyone have any idea? Thanks a lot. A demo is like this. CPU: <SCODE>out = torch.zeros(1, 3, 6, 6)
vout = Variable(out)
fmap = np.arange(3 * 6 * 6).reshape((1, 3, 6, 6))
tmap = Variable(torch.from_numpy(fmap).float(), requires_grad=True)

mask = torch.zeros(1, 6, 6).byte()
mask[0, 2:5, 2:5] = 1
mask = Variable(mask.expand(1, 3, 6, 6))
masked = tmap.masked_select(mask).view(3, -1)
pooled = torch.max(masked, 1)[0][:, 0]
vout[0, :, 0, 0] = pooled
# similar to the operation above
mask = torch.zeros(1, 6, 6).byte()
mask[0, 3:6, 3:6] = 1
mask = Variable(mask.expand(1, 3, 6, 6))
masked = tmap.masked_select(mask).view(3, -1)
pooled = torch.max(masked, 1)[0][:, 0]
vout[0, :, 1, 1] = pooled

a = torch.mean(vout)
a.backward()
print tmap.grad
<ECODE> GPU: <SCODE>out = torch.zeros(1, 3, 6, 6)
vout = Variable(out).cuda()
fmap = np.arange(3 * 6 * 6).reshape((1, 3, 6, 6))
tmap = Variable(torch.from_numpy(fmap).float(), requires_grad=True).cuda()

mask = torch.zeros(1, 6, 6).byte().cuda()
mask[0, 2:5, 2:5] = 1
mask = Variable(mask.expand(1, 3, 6, 6))
masked = tmap.masked_select(mask).view(3, -1)
pooled = torch.max(masked, 1)[0][:, 0]
vout[0, :, 0, 0] = pooled

mask = torch.zeros(1, 6, 6).byte().cuda()
mask[0, 3:6, 3:6] = 1
mask = Variable(mask.expand(1, 3, 6, 6))
masked = tmap.masked_select(mask).view(3, -1)
pooled = torch.max(masked, 1)[0][:, 0]
vout[0, :, 1, 1] = pooled

a = torch.mean(vout)
a.backward()
print tmap.grad
<ECODE> The result is None. I am using version 0.1.9.

apaszke: <SCODE>tmap_leaf = Variable(...)
tmap = tmap_leaf.cuda()
<ECODE> or better <SCODE>tmap = Variable(torch.from_numpy(...).float().cuda(), requires_grad=True)
<ECODE>

Donglai_Xiang: Oh, I get it. Thank you very much!
How to apply a simple model’s parameters to a complicated model(two different model))

hongyuan: I have written a simple model(only the  horizontal  part of above model) and trained the parameters. Now, I want to apply the simple model’s parameters to the above model(a different one). The complicated model can’t inherit from the simple one(or use simple one as a part),  So,is there any way to do this work?

Jordan_Campbell: This will only work if each of the horizontal layers in your diagram are identical. If so then you can save the trained model parameters: <SCODE>torch.save(simple_model.state_dict(), "path/to/file.pth")
<ECODE> and then load them into each of the horizontal layers: <SCODE>filepath = "path/to/file.pth"

complex_model = [network() for i in range(num_layers)]

for layer in complex_model:
    layer.load_state_dict( torch.load( filepath ))
<ECODE>
What is the equivalent of Theano’s images2neibs

Adriano_Pinto: In order to partition an image into smaller patches, during training, in Theano we can take advantage of theano.tensor.nnet.neighbours package that contains images2neibs, neibs2images. How can it be performed in PyTorch?

apaszke: No, I’m afraid we don’t have any operation like that. You can open a feature request in the main repository if you want.

Veril: You can hack up a conv operation to do this. If you want a patch of say 8x8 just do a 8x8x64 convolution with zero padding and let the kernel be 1’s in different positions with all zeros. After you do this your 1x1x64 will be your 8x8 patches. This will probably be very inefficient with maximally sparse convolutions, but unless you intend to do this in an iterative manner it shouldn’t be noticeable. And should be faster than any loop you can cook up probably. With some extra effort you can make a better array programming solution, something involving reshapes and permutes probably.

fmassa: 

pvskand:
How to load caffe models in pytorch

kirk86: Thanks a lot!

apaszke: 

kirk86: 

apaszke: Not anytime soon I’m afraid. There are a lot of high priority tasks.

apaszke: But it should be quite simple for someone to add that. You just need to read protobufs and translate graphs into modules.

kirk86: gotcha ya. Thanks a lot for the suggestions.

xiaoxing_zeng: 

apaszke: 

sunitha_kanuri: 

Neo_li: 

aadharna: Good evening, Following your advice apaszke, I downloaded loadcaffe, and transformed the caffe model + prototxt file into a model.t7 file. The model which gets created is: When I try to use torch.utils.serialization.load_lua(‘imdb.t7’) the result is that the .t7 model is corrupted. As such, I was hoping to get some guidance. Here is the error code: Thank you for your time! Similarly, Neo_li’s solution is not working.
nn.DataParallel(model).cuda() hangs

thnkim: Hi, <SCODE>Traceback (most recent call last):
  File "/home/polphit/anaconda3/lib/python3.6/multiprocessing/process.py", line 249, in _bootstrap
    self.run()
  File "/home/polphit/anaconda3/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/home/polphit/anaconda3/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 28, in _worker_loop
    r = index_queue.get()
  File "/home/polphit/anaconda3/lib/python3.6/multiprocessing/queues.py", line 343, in get
    res = self._reader.recv_bytes()
  File "/home/polphit/anaconda3/lib/python3.6/multiprocessing/connection.py", line 216, in recv_bytes
    buf = self._recv_bytes(maxlength)
  File "/home/polphit/anaconda3/lib/python3.6/multiprocessing/connection.py", line 407, in _recv_bytes
    buf = self._recv(4)
  File "/home/polphit/anaconda3/lib/python3.6/multiprocessing/connection.py", line 379, in _recv
    chunk = read(handle, remaining)
KeyboardInterrupt
<ECODE> I tried on two different machines and got the same issue. Ubuntu 16.04, conda 4.3.14, pytorch installed from source, python 3.6.0.final.0 requests 2.12.4 CUDA 8.0 cuDNN 5.1 When I run the same code on a machine without conda, and python3, it works well.

apaszke: That’s a stack trace of a data loader process, can you paste a full error into a gist and link it here?

thnkim: (I have two GPU devices) Flow is <SCODE>i (input) -> netA ---> netB -> x (output #1)
                   +-> netC -> y (output #2)
                   +-> netD -> z (output #3)
<ECODE> If this is not helpful to guess the cause, I would like to simplify my codes to reproduce the issue with minimal data upload.

thnkim: Oh, when I add torch.cuda.synchronize() at the end of a batch, one machine works properly, although the other machine still has the same issue.

apaszke: 

apaszke: Unfortunately even if we add these locs, doing that in two processes that use the same GPUs in DataParallel will deadlock too…

melody-rain: so…is it a bug of pytorch? I met the same issue.

apaszke: No, it’s a bug in NCCL (NVIDIA’s library). But you probably shouldn’t be using the same GPU in multiple data parallel jobs anyway.

octopusyo: 

smth: Please have a look and see if it applies to you.

Mohit_Chhabra: 

Mohit_Chhabra: It was some hardware issue, although the p2platency test passed, when i changed the CUDA_VISIBLE_DEVICES , and ran the code without DataParallel it gave illegal memory access error for the faulty GPU. Changed the GPU and now the code works fine.
Assertion failure when using Conv1d

Orpheon: Am I doing something wrong or is this a bug? The dimension of the input is [batchsize, channels, data] < 4, and this fits with the documentation if I can read the equation right. There is a previous post here [which I can’t link because newbie link limit] that mentions the same error for conv2d without batch_size, and a possible hack around it with unsqueeze, but this shouldn’t be necessary for conv1d with batch_size.

smth:
How to normalize embedding vectors?

maplewizard: Hi, I am using a network to embed some entity into vector space. As the length of the vector decrease during the training. I want to normalize it’s length to 1 in the end of each step. Is there any tool that I can use to normalize the embedding vectors?

apaszke: I think the best thing you can do is to save the embedded indices, and normalize their rows manually after the update (just index_select them, compute row-wise norm, divice, index_copy back into weights). We only support automatic max norm clipping.

samarth-robo: If you want to normalize a vector as a part of a model, this should do it: assume q is the tensor to be L2 normalized, along dim 1

asdass: Can I use it to normalise the embedding after each update in the training ?

samarth-robo: 

asdass: I see, thanks.  But I need to do the constraints on the norms of embeddings( not bigger than 1 ). Do you have any better suggestions to do it?

jdhao: <SCODE># suppose x is a Variable of size [4, 16], 4 is batch_size, 16 is feature dimension
x = Variable(torch.rand(4, 16), requires_grad=True)
norm = x.norm(p=2, dim=1, keepdim=True)
x_normalized = x.div(norm.expand_as(x))
<ECODE>

moi90: <SCODE>norm = x.norm(p=2, dim=1, keepdim=True)
x_normalized = x.div(norm)
<ECODE>

jdhao: <SCODE>import torch.nn.functional as F
x = F.normalize(x, p=2, dim=1)
<ECODE>

Liang: 

sssohrab: Then in this case I get the following error for any of the above options I try: TypeError: cannot assign ‘torch.autograd.variable.Variable’ as parameter ‘W’ (torch.nn.Parameter or None expected) <SCODE>class myUnit(nn.Module):
    def __init__(self,myParameter):
        super(myUnit, self).__init__()
        self.myParameter = Parameter(myParameter,requires_grad=True)
    def forward(self,input):
        """
        Whatever operation. Just an example:
        """
        self.myParameter = F.normalize(self.myParameter,p=2,dim=1)
        output = self.myParameter * input - 1
        return output<ECODE>

Guohai93: May be you can try: <SCODE>self.myParameter.weight.data = F.normalize(self.myParameter.weight.data, p=2, dim=1)
<ECODE>

Ritwick_Chaudhry: 

f10w: 

ptrblck: Your issue seems to be unrelated to the linked post, as your error points towards an error running a DDP model.

f10w: Sorry I copied the wrong error message. The above message was for the following code: <SCODE>self.embeddings = torch.nn.Parameter(F.normalize(self.embeddings, p=2, dim=-1))
<ECODE> What I would like to do is to constraint the weights of my model to always lie on a sphere. And I finally found the solution: <SCODE>self.embeddings.data.copy_(torch.nn.Parameter(F.normalize(self.embeddings.data, p=2, dim=-1)))
<ECODE> Hope that this will be useful for future readers.
How to experiment on test_torch.py?

dl4daniel: Hi Everyone, Basically, I want to run each test_function on each tensor operation individually, so I can see how the operation work. How I tried, but failed: <SCODE># inside jupyter notebook
%cd  /path to /pytorch-master/test

import torch
import test_torch
test1 = test_torch.TestTorch(methodName='runTest')
test1.test_dot()
<ECODE> <SCODE>def test_dot(self):
        types = {
            'torch.DoubleTensor': 1e-8,
            'torch.FloatTensor': 1e-4,
        }
        for tname, prec in types.items():
            v1 = torch.randn(100).type(tname)
            v2 = torch.randn(100).type(tname)
            res1 = torch.dot(v1, v2)
            print(res1)
            res2 = 0
            for i, j in zip(v1, v2):
                res2 += i * j
            print(res2)
            self.assertEqual(res1, res2)
<ECODE> Could anyone help me find a way to experiment on each test function individually? Thanks a lot! Daniel
Converting Image to Tensor Error
Category: vision

wwoodal1: I am getting this error “RuntimeError: can’t convert a given np.ndarray to a tensor - it has an invalid type. The only supported types are: double, float, int64, int32, and uint8.” even though the ndarray is an array of 249 ndarrays of shape (512x384x3) with dtype=uint8. One interesting thing is np.array(images).shape returns (249,) not the entire shape I am expecting. The images are of varying size so I read in each one and create a thumbnail of standard sizing using PIL thumbnail method. Then i create an ndarray and add it to the images list. From there is try and create a Tensor. <SCODE>def read_image_files(path):
size = 512, 512

# Read in image using imread
images = []
for typePath in tqdm_notebook(glob(path + "/*"), desc='Type'):
    count = 0
    for imgPath in tqdm_notebook(glob(typePath + "/*"), desc='Image'):
        img = Image.open(imgPath)
        img.thumbnail(size)
        images.append(np.array(img))

return torch.from_numpy(np.array(images))<ECODE>

smth: We dont support ndarray of ndarrays, which is why you are seeing this error. torch itself does not support a Tensor containing many variable length Tensors, and we do not plan to support it.
DataParallel, data may have different sizes

chengyangfu: Currently, the DataParallel forces the inputs having same size. But is it possible to support data with different sizes? I am using PyTorch to implement my detection and segmentation frameworks (computer vision). For example, in each image, the number of objects are different. My current solution is to add some dummy to make the annotations of different images in the same batch have same dimensions. I think this works bounding boxes but not a good solution for segmentation masks. I would like to hear the better solution for this kind of problems.

apaszke:
DataLoader vs convolution tensor shape

rracinskij: <SCODE>from torchvision.datasets import ImageFolder
from torchvision.transforms import ToTensor
data = ImageFolder(root='PytorchTestImgDir', transform=ToTensor()) 
print(data.classes) 
from torch.utils.data import DataLoader
train_loader = DataLoader(data, batch_size=1)
<ECODE> but the result is a: RuntimeError: Need input of dimension 4 and input.size[1] == 1 but got input to be of shape: [1 x 3 x 138 x 138] at /Users/soumith/anaconda/conda-bld/pytorch-0.1.10_1488750409207/work/torch/lib/THNN/generic/SpatialConvolutionMM.c:47

jekbradbury: Looks like the MNIST example’s model expects one color channel (MNIST is black and white) but the images you’re providing are being loaded with three (RGB) channels.

rracinskij: My blunder, thanks a lot for figuring it out!

Lee_Jim: nst as bx or not, no right wrong
How to transform classification into regression?

AjayTalati: Well here’s a pretty simple problem, how do you go from a loss = nn.CrossEntropyLoss(output, labels) y_pred = torch.normal( mu, sigma2.sqrt() ) and loss = F.smooth_l1_loss(y_pred, labels) Here’s the relevant bit of my code, <SCODE>    def forward(self, x):
        # Set initial states
        h0 = Variable(torch.zeros(self.num_layers*2, x.size(0), self.hidden_size)) # 2 for bidirection 
        c0 = Variable(torch.zeros(self.num_layers*2, x.size(0), self.hidden_size))
        
        # Forward propagate RNN
        out, _ = self.lstm(x, (h0, c0))
        
        # Decode hidden state of last time step
        mu = self.mu( out[:, -1, :] )
        sigma2 = self.sigma2( out[:, -1, :] )
        return mu, sigma2

rnn = BiRNN(input_size, hidden_size, num_layers, num_classes)

# Loss and Optimizer
optimizer = torch.optim.Adam(rnn.parameters(), lr=learning_rate)
    
# Train the Model 
for epoch in range(num_epochs):
    for i, (images, labels) in enumerate(train_loader):
        images = Variable(images.view(-1, sequence_length, input_size))
        labels = Variable( labels.float() )        
        # Forward + Backward + Optimize
        optimizer.zero_grad()
        #outputs = rnn(images)
        mu, sigma2 = rnn(images)
        sigma2 = (1 + sigma2.exp()).log() # ensure positivity
        y_pred = torch.normal( mu, sigma2.sqrt() ) 
        y_pred = y_pred.float()
        #y_pred = Variable( torch.normal(mu, sigma2.sqrt()).data.float() )       
        loss = F.smooth_l1_loss( y_pred , labels )            
        loss.backward()
        optimizer.step()

<ECODE> and the compile error, <SCODE>  File "main_v1.py", line 90, in <module>
    loss.backward()
  File "/home/ajay/anaconda3/envs/pyphi/lib/python3.6/site-packages/torch/autograd/variable.py", line 158, in backward
    self._execution_engine.run_backward((self,), (gradient,), retain_variables)
  File "/home/ajay/anaconda3/envs/pyphi/lib/python3.6/site-packages/torch/autograd/stochastic_function.py", line 13, in _do_backward
    raise RuntimeError("differentiating stochastic functions requires "
RuntimeError: differentiating stochastic functions requires providing a reward
<ECODE> It’s modified from yunjey/pytorch-tutorial/blob/master/tutorials/07%20-%20Bidirectional%20Recurrent%20Neural%20Network/main.py OR, perhaps I’m making it more complicated than it needs to be with the Gaussian thing? Should I just stick an encoder on the output of the LSTM ???

tom: 

AjayTalati: Cheers

apaszke: 

AjayTalati: It’s nothing to do with PyTorch, I just haven’t thought carefully enough about what I’m actually trying to do here - I was carrying over an idea from continuous action reinforcement learning, and it doesn’t seem to make sense in the context of regression?
Can we use pre-trained word embeddings for weight initialization in nn.Embedding?

wasiahmad: I want to use pre-trained word embeddings as initial weight vectors for embedding layer in a encoder model? How can I achieve this? For example, in Keras, we can pass a weight matrix as parameter to the embedding layer. Is there any similar way in PyTorch to do this?

ruotianluo: you can just assign the weight to the embedding layer. Like: <SCODE>embed = nn.Embedding(num_embeddings, embedding_dim)
# pretrained_weight is a numpy matrix of shape (num_embeddings, embedding_dim)
embed.weight.data.copy_(torch.from_numpy(pretrained_weight))<ECODE>

ShawnGuo: I usually use the following way, which is better? <SCODE>#embeddings is a torch tensor.
embedding = nn.Embedding(embeddings.size(0), embeddings.size(1))
embedding.weight = nn.Parameter(embeddings)
<ECODE>

apaszke: I’d say they’re both ok.

xinyadu: And how can we keep the embedding matrix fixed during training? I didn’t find that in the doc.

cyyyyc123: <SCODE>embed.weight.requires_grad = False
<ECODE> to freeze the parameter of it.

xinyadu: But when I initialize optimizer, i got “ValueError: optimizing a parameter that doesn’t require gradients”

xwgeng: 

Navneet_M_Kumar: How can we specifically use glove vectors and mainly in the encoder - decoder model ? Not able to understand.

tushargupta14: 

sritvik: What if I want to use sentence embedding as a whole and not word vectors. Suppose I have the sentence embeddings ready, do I create a (number of sentences X sentence embedding dimension) matrix and map it to each sentence id? And pass this matrix to the embedding layer and call the sentence ids in the forward function? Is this approach right? I’m trying to perform a type of sentence classification.

mfa: Regarding the use of pre-trained word embeddings, the indexes of words that are passed to the Embedding layer should be equivalent to their index in the pre-trained embedding (the numpy matrix), right?

shirdu: 

colesbury: Yes, it’s a matrix. Each row is the embedding for a word. You’ll also want a dictionary: mapping of words (strings) to integers 0, 1, …, N

shirdu: 

shirdu: 

tushargupta14: Yes, the indexes of the words in your vocabulary should be same as your indexes in your embedding numpy matrix. This helps the embedding layer to map the word tokens to vectors.

austin: 

sidedishes: Sorry for the bump - what might be the easiest way to have only parts of the embedding matrix frozen? For example, if I wanted to use pre-trained embeddings but for certain words assign a special custom token whose embedding I want to train?

jlquinn: Two ways I’d see doing it: First to have two separate embeddings.  One embedding learns, the other uses pre-trained weights.  Select the embedding to use depending on the value of the input. The other approach would be to overwrite the pretrained parts of the embedding at the beginning of each batch to undo the results of the previous optimizer step.
attributeError on `_rebuild_tensor` caused conda install avoided by pip install

dl4daniel: <SCODE>    ", ".join([t for t in dir(torch._utils)])
                                                            ### inside _utils we have the following function, but why not shown up???

                                                            # def _rebuild_tensor(storage, storage_offset, size, stride):
                                                            #     class_name = storage.__class__.__name__.replace('Storage', 'Tensor')
                                                            #     module = importlib.import_module(storage.__module__)
                                                            #     tensor_class = getattr(module, class_name)
                                                            #     return tensor_class().set_(storage, storage_offset, size, stride)

# output is the following
# '__builtins__, __cached__, __doc__, __file__, __loader__, __name__, __package__, __spec__, 
# _accumulate, _cuda, _import_dotted_name, _range, _type, torch'
<ECODE> <SCODE>", ".join([t for t in dir(torch.tensor)])

# output is the following
# '_TensorBase, __builtins__, __cached__, __doc__, __file__, __loader__, __name__, __package__, 
# __spec__, _cuda, _range, _tensor_str, _type, sys, torch'
<ECODE> Guesses <SCODE>ERROR: test_serialization_map_location (__main__.TestTorch)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_torch.py", line 2711, in test_serialization_map_location
    tensor = torch.load(test_file_path, map_location=map_location)
  File "/Users/Natsume/miniconda2/envs/dlnd-tf-lab/lib/python3.5/site-packages/torch/serialization.py", line 222, in load
    return _load(f, map_location, pickle_module)
  File "/Users/Natsume/miniconda2/envs/dlnd-tf-lab/lib/python3.5/site-packages/torch/serialization.py", line 370, in _load
    result = unpickler.load()
AttributeError: Can't get attribute '_rebuild_tensor' on <module 'torch._utils' from '/Users/Natsume/miniconda2/envs/dlnd-tf-lab/lib/python3.5/site-packages/torch/_utils.py'>
<ECODE> Solution worked this time <SCODE>Exception ignored in: <function WeakValueDictionary.__init__.<locals>.remove at 0x114c5da60>
Traceback (most recent call last):
  File "/Users/Natsume/miniconda2/envs/pytorch-experiment/lib/python3.5/weakref.py", line 117, in remove
TypeError: 'NoneType' object is not callable
<ECODE> should I be worried about this one? Thanks a lot!

cslxiao: I met the same problem. The reason may be the version of the pretrained model you load doesn’t match with your pytorch.