foldl/torch-forum huggingface dataset:
Convert int into one-hot format

Response777: Hi all. <SCODE>for batch_idx, (x, y) in enumerate(train_loader):
    y_onehot = y.numpy()
    y_onehot = (np.arange(num_labels) == y_onehot[:,None]).astype(np.float32)
    y_onehot = torch.from_numpy(y_onehot)
<ECODE> However, I notice that the it gets slower each iteration, and I doubt it’s these code which might request new memory each iteration that makes the code slower. Thanks!

moskomule: 

Response777: 

albanD: Hi, <SCODE>import torch

batch_size = 5
nb_digits = 10
# Dummy input that HAS to be 2D for the scatter (you can use view(-1,1) if needed)
y = torch.LongTensor(batch_size,1).random_() % nb_digits
# One hot encoding buffer that you create out of the loop and just keep reusing
y_onehot = torch.FloatTensor(batch_size, nb_digits)

# In your for loop
y_onehot.zero_()
y_onehot.scatter_(1, y, 1)

print(y)
print(y_onehot)
<ECODE>

Response777: Thanks, that is exactly what I need!

Nadav_Bhonker: 

Response777: 

apaszke: 

ncullen93: And also for future readers just to reiterate what user moskomule says- cross entropy and neg. log-likelihood losses in pytorch do NOT require one-hot encodings, so you can just use the normal target vector.

apaszke: 

rajarsheem: How about overriding the default nn.Embedding weights data with torch.eye ? <SCODE>emb = nn.Embedding(10, 10) 
emb.weight.data = torch.eye(10)
<ECODE>

apaszke: Additionaly, zero + scatter a few ones will be  much faster than copying whole rows, of which most values are 0 anyway.

rogetrullo: <SCODE>batch_size=10
y = torch.LongTensor(batch_size,5,5).random_() % 3#3 classes,5x5 img
y_onehot = torch.FloatTensor(batch_size,3, 5,5)#I want the one hot going through the chans dim
y_onehot.zero_()
ones=torch.ones(y.size())

y_onehot.scatter_(1,y,ones)
<ECODE> Could you help me with this? Thanks!

rogetrullo: oh never mind, I just found that it works if I add a singleton dimension so that y and y_onehot have the same NUMBER of dimensions…

zeng: 

albanD: 

zeng: Sorry, I write wrong, it is also not available for cuda.LongTensor.

albanD: 

zeng: Thank you, my problem. I ignored the y_onehot type is not a cuda tensor.

kakrafoon: Thank you! I got my cvae implemented with this tip.
Gradient computation with index_select (for recursive neural networks)

elPistolero: Hi all, My cell looks something like this: <SCODE>import torch
import torch.nn.functional as F
import torch.nn as nn

class ReNNCell(nn.Module):
def __init__(self, dim):
    super(ReNNCell, self).__init__()
    self.dim = dim
    self.W = nn.Linear(dim*2, dim)
    self.W_score = nn.Linear(dim, 1)

def forward(self, inputs):
    assert(inputs.size()[0] == 1), 'we expect batch size = 1'
    rep = F.relu(self.W(inputs))
    score = F.relu(self.W_score(rep))
    return rep, score
<ECODE> <SCODE>tensors = [Variable(torch.randn(1,3)), Variable(torch.randn(1,3)), Variable(torch.randn(1,3)), Variable(torch.randn(1,3))]

cats = []
for i in range(1,4,2):
    cats.append(torch.cat([tensors[i-1], tensors[i]],1))

cell = ReNNCell(3,2)
outputs = [cell(c) for c in cats]
reps = [o[0] for o in outputs]
scores = [o[1] for o in outputs]
scores = torch.cat(scores, 1)
reps = torch.cat(reps, 0)
max_score, max_index = torch.max(scores, 1)
max_index = torch.squeeze(max_index, 1)
max_score = torch.squeeze(max_score, 1)
<ECODE> Based on this selection I will compute some dummy loss. <SCODE>crit = torch.nn.MSELoss()
loss = crit(reps.index_select(0,max_index), Variable(torch.ones(1,3)))
<ECODE> <SCODE>RuntimeError: could not compute gradients for some functions (Threshold, Threshold)
<ECODE> Could anybody point me in the right direction? Is this a bug or do I perhaps have to somehow  compute the gradients myself?

albanD: Hi,

elPistolero: Yes it does! So index_select somehow discards the Variable state (or is it squeeze)?

albanD: 

elPistolero:
Masked Summation of Tensor

mattyd2: <SCODE>   99    99
  489  1171
  281  1317
     ⋮      
    0   435
    0  2741
    0   517
[torch.LongTensor of size 2406x2]```

POST EMBEDDING:
```context_output.size() -- (2406L, 2L, 1L)``` -- (word_sequence, batch_size, attention_score)

```Variable containing:
( 0  ,.,.) = 
1.00000e-02 *
  2.0804
  1.6674
  2.9782
   ⋮    
  4.8565
  4.8565
  4.8565
...
( 1  ,.,.) = 
1.00000e-02 *
  0.2246
  1.4224
  4.1816
   ⋮    
  4.4363
  3.0162
  3.3986
[torch.FloatTensor of size 2x2406x1]```


I need to sum the values in the POST EMBEDDING tensor if they have the same ```word_id``` from the PRE EMBEDDING tensor.

Example:

PRE EMBEDDING
```Variable containing:
   4    1
   4    2
   2    2
[torch.LongTensor of size 3x2]```

POST EMBEDDING:

```Variable containing:
( 0  ,.,.) =
  0.35
  0.35
  0.65
...
( 1  ,.,.) = 
  0.25
  0.65
  0.65
[torch.FloatTensor of size 2x3x1]```

DESIRED RESULT

```Variable containing:
( 0  ,.,.) =
   4.0    0.7
   2.0    0.65
   1.0    0.0
...
( 1  ,.,.) =
   4.0    0.0
   2.0    1.3
   1.0    0.25
[torch.LongTensor of size 2x3x2]```<ECODE>

apaszke:
Update PyTorch version on LINUX w/ CUDA

Kalamaya: I want to make sure that I do not break my installation on my LINUX machine with GPU (cuda 8.0) when attempting to update pytorch, so I am asking here to be sure. I would like to now upgrade my PyTorch version to whatever is on master, and the website has this to say about LINUX with CUDA8: conda install pytorch torchvision cuda80 -c soumith However, I do not have conda, and anyway I installed via a wheel with pip. Is there anything I should do (or not do) in particular here to make sure I do not break my system? I want to basically just update my pytorch on my LINUX machine. Thanks!

Veril: 

Kalamaya: Thanks!

Kalamaya: 

supakjk: Hope there is something like “luarocks install nn” in Torch7 so that we can update blood-edge versions quicky…

apaszke: All remaining question are about managing venvs, and there are probably lots of articles, that will explain it better, that I will in this short response.

Kalamaya: How do I do that?  I looked at sys.modules but… it’s just a huge amount of information dumped into the screen. I currently do not know how to control what version is being loaded… :-/ Help!

apaszke:
Usage of nn.ModuleList

Kalamaya: <SCODE> module = [nn.Conv2d(1,64, 3, stride = (1,1), padding = (1,1)), nn.BatchNorm2d(
 self.module_list = nn.ModuleList(module)
 self.module_list += module
<ECODE> Each “module”, is composed of a Conv2d, and a BatchNorm. What I would like self.module_list to be, is: Conv2d -> BatchNorm -> Conv2d -> BatchNorm Is this the right way to go about it? Thanks!!

apaszke: No, you’re adding the same modules, with the same weights twice. You need to create a new list, and append that after construction. <SCODE>def make_sequence():
    return [nn.Conv2d(1,64, 3, stride = (1,1), padding = (1,1)), ...]

self.module_list = nn.ModuleList()
for i in range(5):
    self.module_list += make_sequence()
<ECODE>
Stop gradients (for ST gumbel softmax)

eliabruni: Hi, Given the soft implementation of Gumbel Softmax: <SCODE>from torch.autograd import Variable
import torch.nn.functional as F

def sample_gumbel(input):
    noise = torch.rand(input.size())
    eps = 1e-20
    noise.add_(eps).log_().neg_()
    noise.add_(eps).log_().neg_()
    return Variable(noise)

def gumbel_softmax_sample(input):
    temperature = 1
    noise = sample_gumbel(input)
    x = (input + noise) / temperature
    x = F.log_softmax(x)
    return x.view_as(input)
<ECODE>

Ilya_Kostrikov: I did stop_gradient(x) in PyTorch as Variable(x.data). However, I’m looking for a better alternative.

apaszke: 

eliabruni: Thank you both, I’ll try it out!

Seungyoung_Park: <SCODE>def gumbel_softmax_sample(self, input):
        temperature = 1
        noise = self.sample_gumbel(input)
        x = (input + noise) / temperature
        x = F.softmax(x)

        if hard == True:
            max_val, _ = torch.max(x, x.dim()-1)
            x_hard = x == max_val.expand_as(x)
            tmp = (x_hard.float() - x)
            tmp2 = tmp.clone()
            tmp2.detach_()
            x = tmp2 + x

        return x.view_as(input)
<ECODE> But, I tried to modify it such that only one ‘1’ exists at each row in x_hard as below. <SCODE>def gumbel_softmax_sample(self, input):
    temperature = 1
    noise = self.sample_gumbel(input)
    x = (input + noise) / temperature
    x = F.softmax(x)

    if hard == True:
        _, max_inx = torch.max(x, x.dim()-1)
        x_hard = torch.cuda.FloatTensor(x.size()).zero_().scatter_(x.dim()-1, max_inx.data, 1.0)
        x2 = x.clone()
        tmp = Variable(x_hard-x2.data)
        tmp.detach_()

        x = tmp + x
        
    return x.view_as(input)
<ECODE> These two scripts work without error, but I am not sure they are equivalent to the original ST gumbel softmax. I would greatly appreciate if someone reviews them.

Seungyoung_Park: Is there anyone helps me to confirm that these two scripts are equivalent to the original tensorflow implementation ?

ethancaballero: 

Seungyoung_Park: Because I do not know how to make the parameters same as those of the original, I have not yet compared them.

Qi_Zhang: 

hughperkins: <SCODE>y = tf.stop_gradient(y_hard - y) + y
<ECODE> y is pure one-hot, in terms of value (since we add the soft y, and then subtract it again the gradients are those of soft y (since all the other terms in this expression have their gradient stripped)

hughperkins:
Implementing low rank approximation

madan-ram: below is simple code to test the performance part 1 <SCODE>import torch
import time
import numpy as np

img = torch.autograd.Variable(torch.rand(1, 3, 416, 416))
filters = torch.autograd.Variable(torch.rand(64, 3, 5, 5))

avg = 0
for x in xrange(10):
	num_ops = 0
	st = time.clock()
	conv = torch.nn.functional.conv2d(img, filters, padding=1)
	num_ops = np.prod(conv.size())*np.prod(filters.size())
	# print result.size()
	avg += time.clock() - st
print avg/10.0, 'Average conv operation time'
print 'Number of operation', num_ops
<ECODE> Part 2 <SCODE>print "================================================================="
filters = torch.autograd.Variable(torch.rand(8, 3, 5, 5))
A = torch.autograd.Variable(torch.rand(64, 8))
avg = 0
core_avg = 0
for x in xrange(10):
	num_ops = 0
	st = time.clock()

	core_st = time.clock()
	conv = torch.nn.functional.conv2d(img, filters, padding=1)
	num_ops = np.prod(conv.size())*np.prod(filters.size())
	core_avg += time.clock() - core_st

	conv = conv.view(8, -1)
	core_st = time.clock()
	num_ops += np.prod(conv.size()) * A.size()[0]
	conv = torch.mm(A, conv)
	core_avg += time.clock() - core_st

	# print conv.view(result.size()).size()
	avg += time.clock() - st
print 'Number of reduced operation', num_ops
print avg/10.0, 'Average reduced operation time'
print core_avg/10.0, 'Average reduced core operation time'
<ECODE> part 1 does the simple convolution and part 2 does low-rank approximation.

apaszke: The number of operations you computed is for the naive convolution algorithm, but there’s been a lot of research in this area, and modern algorithms perform much less operations. Additionally, the actual speed depends not only on floating point operations, but also on memory bandwidth. Doing conv + mm requires reloading some intermediate values multiple times, and this takes time. A single conv kernel can reuse values already loaded int registers.
Using your own dataset on PyTorch

Ismail_Elezi: I have a dataset in numpy format (actually, I just modified the CIFAR-10/MNIST datasets given on PyTorch). The dimensions of it consist with the dimensions a normal CNN expects. For example, if I write: print(our_dataset.shape, our_labels.shape) I get: (10000, 3, 32, 32) (10000,) which is fine. Now I cast the data info torch format using: <SCODE>train_data = torch.from_numpy(our_dataset)
our_labels = torch.from_numpy(our_labels)
<ECODE> encapsulate it into a TensorDataset: train = torch.utils.data.TensorDataset(train_data, our_labels) and finally into a DataLoader: trainloader = torch.utils.data.DataLoader(train, batch_size=128, shuffle=True) All fine here. Now I build the neural network, and then when I do the training, I get the error: TypeError: DoubleSpatialConvolutionMM_updateOutput received an invalid combination of arguments - got (int, torch.DoubleTensor, torch.DoubleTensor, torch.FloatTensor, torch.FloatTensor, torch.DoubleTensor, torch.DoubleTensor, long, long, int, int, int, int), but expected (int state, torch.DoubleTensor input, torch.DoubleTensor output, torch.DoubleTensor weight, [torch.DoubleTensor bias or None], torch.DoubleTensor finput, torch.DoubleTensor fgradInput, int kW, int kH, int dW, int dH, int padW, int padH) If I am working on cuda, then the error changes to: _cudnn_convolution_full_forward received an invalid combination of arguments - got (torch.cuda.DoubleTensor, torch.cuda.FloatTensor, torch.cuda.FloatTensor, torch.cuda.DoubleTensor, tuple, tuple, int, bool), but expected (torch.cuda.RealTensor input, torch.cuda.RealTensor weight, torch.cuda.RealTensor bias, torch.cuda.RealTensor output, std::vector<int> pad, std::vector<int> stride, int groups, bool benchmark) Anyone has seen these errors before? In addition, is the right way of using a new dataset by first casting it to torch format, then building a TensorDataset and finally a DataLoader?

sarthak1996: 

edgarriba: you need to cast your data to double in that case

Ismail_Elezi: If I type: print(our_dataset.dtype, our_labels.dtype) I get: float32 int64 I sent the data to cuda (and gave the Error in that case). However, if I don’t send the net (and data) to cuda, I get the other error that I posted in my original post. @edgarriba Well, the data is already in float.

edgarriba: right, but the error says that expect a DoubleTensor

sarthak1996: 

Ismail_Elezi: If I case the our_dataset to float32 and labels to int64, then the error changes to: multi-target not supported at /data/users/soumith/miniconda2/conda-bld/pytorch-0.1.7_1485444530918/work/torch/lib/THNN/generic/ClassNLLCriterion.c:20 but the shape of the labels is (10000,). Hmm, this is becoming weird.

sarthak1996: 

Ismail_Elezi: It seems that this works.

Shine_George: the version of 0.3.0 can well solve the problem of the problem of ‘received an invalid combination of arguments’
TypeError: addmm_, after transfering the net to GPU (works fine on CPU)

nelsonleung: Hi all, to Thanks a lot for your help.

smth: the input to your network is on the CPU, but your network is transferred to the GPU (or vice versa). Here’s the critical part of the error that helped me figure this out: <SCODE> got (int, int, torch.FloatTensor, torch.cuda.FloatTensor),
<ECODE>

nelsonleung: I see. Got it to work. Thanks a lot for your prompt reply!

Ke_Bai: Thanks in advance.

jupiter: 

Dami_Mi: <SCODE>def __len__(self):
    return self.npy_data.shape[0]

def __getitem__(self, index):
    fm_data = np.array(self.npy_data[:, :-1], dtype = np.float)
    label_data = np.array(self.npy_data[:, -1], dtype = np.float)

    return fm_data, label_data
<ECODE> <SCODE>def forward(self, x):
    out = self.fc1(x)
    out = self.relu(out)
    out = self.fc2(out)
    return out
<ECODE> <SCODE>        label_data = torch.from_numpy(label_data).float().cuda()
        label_data = Variable(label_data)

        optimizer.zero_grad()  # zero the gradient buffer
        outputs = net(fm_data)
        loss = criterion(outputs, label_data)
        loss.backward()
        optimizer.step()

        if (i + 1) % 100 == 0:
            print ('Epoch [%d/%d], Step [%d/%d], Loss: %.4f'
                   %(epoch+1, num_epochs, i+1, len(train_data)//batch_size, loss.data[0]))
<ECODE> could you give me an advice to solve this problem? Thank you very much!

SimonW: By the way, your question has nothing to do with the original post. So you should post a new thread instead replying in here. Further more, the error message is quite clear in your case, you might be able to solve such issues faster if you think about the message and consult the doc in future…

Dami_Mi: 

Borna_Ghotbi: Hi Nelson

xhran2010: Hey. Have you got it worked?
Clarification on simple network using pytorch

kirk86: Hi everyone, thanks for the great community. I was wondering whether could  someone provide some insights into how to construct a NN such as those in the examples on the github page but on its simplest form.  Let me explain with an example. For instance in the github page of examples one can find the following. <SCODE>import torch.nn as nn
import torch.nn.functional as F

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(1, 6, 5) # 1 input image channel, 6 output channels, 5x5 square convolution kernel
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1   = nn.Linear(16*5*5, 120) # an affine operation: y = Wx + b
        self.fc2   = nn.Linear(120, 84)
        self.fc3   = nn.Linear(84, 10)

    def forward(self, x):
        x = F.max_pool2d(F.relu(self.conv1(x)), (2, 2)) # Max pooling over a (2, 2) window
        x = F.max_pool2d(F.relu(self.conv2(x)), 2) # If the size is a square you can only specify a single number
        x = x.view(-1, self.num_flat_features(x))
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

net = Net()
<ECODE> There are a couple of things here that are being obfuscated and might create confusion for newcomers. Thanks.

apaszke: 

smth: From your post, I see that you want to avoid classes and inheritance. You could look at Sergey Zagoruyko’s purely functional implementation of Wide ResNets:

kirk86: 

isalirezag: I’m confused regarding the size of the max pooling window. Thanks

kirk86: Exactly as you said it. The example is just to show both ways are valid. If you have pooling size of 2x2 you can simply use one number (2) instead of (2, 2). In other cases you might wanna have pooling of over different window/kernel sizes e.g. (2, 3).
In what situation the network computation would be slow?

ruotianluo: I know when you run the first batch, it’s possible to be very slow. However, I met the following situation: When I spend a long time reading data, the time of network is also higher. The time is counted after synchronized. My dataloader includes multi-threaded loading. Anyone has any idea?

apaszke: That’s a tough one. There are many reasons why this can happen. You can have a background process that’s sleeping most of the time, but wakes up and consumes a lot of resources for some time. It might be Python’s garbage collection. It can be some kind of cleanup in the OS. It can be because of the hardware. Does this happen to you very often?

ruotianluo: It may be solved if I change my data loading strategy. (The readdata time is occasionally long because I’m periodically dumping tasks into the pool). But every time readdata is long, the network also becomes slow. So I’m curious what will cause the network to become slow. Could high IO slow down the network computation? I thought the network computation time is mostly determined by GPU?

apaszke: It is, but you still need to have your CPU share, so it can queue the kernels for the GPU. If the Python process is suspended, then it can’t give the GPU any work to do.
Allocate cuda tensor in subprocess

florin: <SCODE>import torch
import torch.multiprocessing as mp

def put_in_q():
    x = torch.IntTensor(2, 2).fill_(22)
    x = x.cuda()
    print(x)

p = mp.Process(target=put_in_q, args=())
p.start()
p.join()
<ECODE> But not this one: <SCODE>import torch
import multiprocessing as mp

class CudaProcess(mp.Process):
    def __init__(self):
        mp.Process.__init__(self)

    def run(self):
        x = torch.IntTensor(2, 2).fill_(22)
        x = x.cuda()
        print(x)

p = CudaProcess()
p.start()
p.join()
<ECODE> The error I’m getting is: <SCODE>terminate called after throwing an instance of 'THException'
  what():  cuda runtime error (3) : initialization error at .../pytorch/torch/lib/THC/THCGeneral.c:70
<ECODE>

apaszke: CUDA multiprocessing is quite complicated, and I wouldn’t recommend it, unless it would give you huge performance benefits. The two most important things to remember are:

kmichaelkills: I’m not familiar with all the architecture in pytorch, but I would like to share some thoughts if I may. Wouldnt multi-threading mult-gpu work if: I guess moving all c++ part to cffi would solve 1)  and would make it even possible to use pypy (lua torch  used JIT but in python its suddenly a bad idea?)

florin: 

florin: edit:

kmichaelkills: UB=undefined behaviour I guess

apaszke: Yeah, UB is undefined behaviour = anything could happen now. Regarding cffi, we didn’t want to add more dependencies, and having more fine grained control over GIL and some other aspects makes it easier for us to develop the C code. I’d love to support PyPy, but there are very few people using it, so it’s not really a top priority for us.
How to use mark_non_differentiable

mattyd2: Hi All,

apaszke: For example, comparison is non-differentiable, so you could implement a Function for it like that: <SCODE>class Gt(Function):

    def __init__(self, scalar=None):
        super(Gt, self).__init__()
        self.scalar = scalar

    def forward(self, tensor1, tensor2=None):
        other = tensor2 if tensor2 is not None else self.scalar
        mask = tensor1.gt(other)
        self.mark_non_differentiable(mask)
        return mask
<ECODE>
Did torch.gather change in new pytorch version?

Kalamaya: Before I upgraded to the latest pytorch, this command worked for me: ss = pp.gather(1, labels) <SCODE>Variable containing:
 0.0651  0.9349
 0.6208  0.3792
 0.3024  0.6976
 0.2226  0.7774
 0.0394  0.9606
 0.4197  0.5803
 0.1205  0.8795
 0.3774  0.6226
 0.1682  0.8318
 0.2281  0.7719
 0.3845  0.6155
 0.4658  0.5342
 0.4982  0.5018
 0.2653  0.7347
 0.2694  0.7306
 0.6550  0.3450
[torch.cuda.FloatTensor of size 16x2 (GPU 0)]
<ECODE> <SCODE>Variable containing:
 1
 1
 1
 1
 1
 1
 1
 1
 0
 0
 0
 0
 0
 0
 0
 0
[torch.cuda.LongTensor of size 16 (GPU 0)]
<ECODE> However after I upgraded to the latest version off of master, this same line gives me the following error: *** RuntimeError: Input tensor must have same dimensions as output tensor at /data/pytorch/torch/lib/THC/generic/THCTensorScatterGather.cu:16

Kalamaya: The answer, is that labels needs to be reshaped, into an explicit 16x1 vector, from the 16 it is at. Therefore, doing labels.view(-1,1) will make it work.

jdily: RuntimeError: Index tensor must have same dimensions as input tensor at /home/jdily/Desktop/project/lib/pytorch/torch/lib/THC/generic/THCTensorScatterGather.cu:111 And there is no error when running under cpu mode. thx!

smth: Maybe you are affected by this.
Why RNN needs two biases?

Sunnydreamrain: 

Ismail_Elezi: You mean in general? For the same reason that it needs two sets of weights, one for the input and one from the previous state.

Sunnydreamrain: 

apaszke: As you pointed out it doesn’t really change the definition of the model, but this is what cuDNN does, so we’ve made our RNNs consistent with this behaviour.

Sunnydreamrain: OKay. Good to know. Thanks.