foldl/torch-forum huggingface dataset:
Source compile error related to half precision?

supakjk: <SCODE>gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/homes/3/kimjook/pytorch -I/homes/3/kimjook/pytorch/torch/csrc -I/homes/3/kimjook/pytorch/torch/lib/tmp_install/include -I/homes/3/kimjook/pytorch/torch/lib/tmp_install/include/TH -I/homes/3/kimjook/pytorch/torch/lib/tmp_install/include/THPP -I/homes/3/kimjook/pytorch/torch/lib/tmp_install/include/THNN -I/u/drspeech/opt/anaconda2/lib/python2.7/site-packages/numpy/core/include -I/usr/local/cuda/include -I/homes/3/kimjook/pytorch/torch/lib/tmp_install/include/THCUNN -I/u/drspeech/opt/anaconda2/include/python2.7 -c torch/csrc/cuda/Storage.cpp -o build/temp.linux-x86_64-2.7/torch/csrc/cuda/Storage.o -D_THP_CORE -std=c++11 -Wno-write-strings -DWITH_NUMPY -DWITH_CUDA -DCUDA_LIB_PATH=/usr/local/cuda/lib64
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++ [enabled by default]
torch/csrc/nn/THNN_generic.cpp: In function ‘void torch::nn::Abs_updateOutput(thpp::Tensor*, thpp::Tensor*)’:
torch/csrc/nn/THNN_generic.cpp:45:14: error: ‘THCudaHalfTensor’ was not declared in this scope
             (THCudaHalfTensor*)input->cdata(),
              ^


In file included from /homes/3/kimjook/pytorch/torch/lib/tmp_install/include/THPP/tensors/../storages/THCStorage.hpp:11:0,
                 from /homes/3/kimjook/pytorch/torch/lib/tmp_install/include/THPP/tensors/THCTensor.hpp:22,
                 from torch/csrc/DynamicTypes.cpp:11:
/homes/3/kimjook/pytorch/torch/lib/tmp_install/include/THPP/tensors/../storages/../TraitsCuda.hpp:9:20: error: ‘half’ was not declared in this scope
 struct type_traits<half> {
                    ^
<ECODE> It seems that the compiler cannot correctly recognize “half” precision related data structures or classes. How would I reslove this problem? (I don’t need half precision operations so if possible it’s fine for me to disable it.) Thanks!

apaszke: 

supakjk: The full log is as follows. Thanks.

smth: are you using CUDA 8.0.27 by any chance, instead of the latest 8.0.44?

supakjk: I am using CUDA 8.0.61. Thanks.

apaszke: I have no idea what’s wrong. You have a fresh version of CUDA, so half should be automatically enabled everywhere.

supakjk: Tried it, but no differences.

apaszke: 

supakjk: Oh. That’s it. My system’s /usr/local/cuda is linked to cuda-6.5 since I am sharing the system with others. Would there be any other way I can explictly specify the directory path? Thanks!

apaszke:
Type Variable doesn’t implement stateless method cumprod

ypxie: <SCODE>import torch
from torch.autograd import Variable
inputs = Variable(torch.randn(4,5), requires_grad = True)
#inputs = torch.randn(4,5)
cum_out = torch.cumprod(inputs, dim =0)

print(cum_out)
<ECODE> <SCODE>RuntimeError                              Traceback (most recent call last)
<ipython-input-6-427be1ef1fd7> in <module>()
      4 #inputs = torch.randn(4,5)
      5 
----> 6 cum_out = torch.cumprod(inputs, dim =0)
      7 
      8 print(cum_out)

RuntimeError: Type Variable doesn't implement stateless method cumprod
<ECODE>

apaszke: 

Ja-Keoung_Koo:
Nan in backward() if prod() is used

OCY: In torch.autograd._functions.reduce When input entry is zero, this method returns ‘nan’ gradient. By replacing backward() function of Prod() class with <SCODE>    if self.dim is None:
        input, = self.saved_tensors
        zero_loc = (input==0).nonzero()
        if zero_loc.dim() == 0:
            grad_input = grad_output.new(self.input_size).fill_(self.result)
            return grad_input.div(input)
        elif zero_loc.size()[0] > 1:
            return grad_output.new(self.input_size).fill_(0)
        else:
            grad_input = grad_output.new(self.input_size).fill_(0)
            indexing_tuple = tuple(zero_loc[0].numpy())
            input_copy = grad_output.new(self.input_size).fill_(0)
            input_copy.copy_(input)
            input_copy[indexing_tuple] = 1.0
            grad_input[indexing_tuple] = input_copy.prod()
            return grad_input
    else:
        input, output = self.saved_tensors
        input_copy = grad_output.new(self.input_size).fill_(0)
        input_copy.copy_(input)
        input_copy[input == 0] = 1.0

        repeats = [1 for _ in self.input_size]
        repeats[self.dim] = self.input_size[self.dim]
        output_zero_cnt = (input == 0).sum(self.dim)
        output_one_zero_ind = (output_zero_cnt == 1).nonzero()
        grad_input = output.mul(grad_output)
        grad_input[output_zero_cnt > 0] = 0.0
        grad_input = grad_input.repeat(*repeats).div_(input_copy)
        if output_one_zero_ind.dim() == 0:
            return grad_input

        for i in range(output_one_zero_ind.size()[0]):
            if output_one_zero_ind.is_cuda:
                output_one_zero_vec_ind = tuple(output_one_zero_ind[i].cpu().numpy())
            else:
                output_one_zero_vec_ind = tuple(output_one_zero_ind[i].numpy())
            output_one_zero_vec_indexing = output_one_zero_vec_ind[:self.dim] + (slice(0, None),) + output_one_zero_vec_ind[self.dim+1:]
            output_one_zero_vec = input.new(self.input_size[self.dim]).fill_(0)
            output_one_zero_vec.copy_(input[output_one_zero_vec_indexing])
            output_one_zero_vec[(output_one_zero_vec==0).nonzero()[0, 0]] = 1.0
            grad_input[output_one_zero_vec_ind] = output_one_zero_vec.prod() if output_one_zero_vec.numel()>1 else 1.0
        return grad_input
<ECODE> It would be better this modification is reflected in the next release.

apaszke: Thanks, we’ll improve the formula.

OCY: Previous code only worked for 2 dim tensor. I modified it to make it work for n dim tensor.

OCY: This should be solved.
Graph not resetting between backward passes?

ematvey: I guess I’m missing something obvious here, but why running the model again doesn’t refill the buffers? I was going to implement training in a similar loop. Code to reproduce: <SCODE>import torch as th
import torch.nn as nn
from torch.autograd import Variable

# borrowed from `word_language_model`
class RNNModel(nn.Module):
    """Container module with an encoder, a recurrent module, and a decoder."""

    def __init__(self, rnn_type, ntoken, ninp, nhid, nlayers):
        super(RNNModel, self).__init__()
        self.encoder = nn.Embedding(ntoken, ninp)
        self.rnn = getattr(nn, rnn_type)(ninp, nhid, nlayers, bias=False)
        self.decoder = nn.Linear(nhid, ntoken)

        self.init_weights()

        self.rnn_type = rnn_type
        self.nhid = nhid
        self.nlayers = nlayers

    def init_weights(self):
        initrange = 0.1
        self.encoder.weight.data.uniform_(-initrange, initrange)
        self.decoder.bias.data.fill_(0)
        self.decoder.weight.data.uniform_(-initrange, initrange)

    def forward(self, input, hidden):
        emb = self.encoder(input)
        output, hidden = self.rnn(emb, hidden)
        decoded = self.decoder(output.view(output.size(0)*output.size(1), output.size(2)))
        return decoded.view(output.size(0), output.size(1), decoded.size(1)), hidden

    def init_hidden(self, bsz):
        weight = next(self.parameters()).data
        if self.rnn_type == 'LSTM':
            return (Variable(weight.new(self.nlayers, bsz, self.nhid).zero_()),
                    Variable(weight.new(self.nlayers, bsz, self.nhid).zero_()))
        else:
            return Variable(weight.new(self.nlayers, bsz, self.nhid).zero_())

vocab_size = 256
batch_size = 64

model = RNNModel('GRU', vocab_size, 100, 100, 3)
model.train()

optimizer = th.optim.Adam(model.parameters(), lr=1e-2)
criterion = nn.CrossEntropyLoss()

x = Variable(th.LongTensor(50, 64))
x[:] = 1

state = model.init_hidden(batch_size)

print('first pass')
optimizer.zero_grad()
logits, state = model(x, state)
loss = criterion(logits.view(-1, vocab_size), x.view(-1))
loss.backward()
optimizer.step()

print('second pass')
optimizer.zero_grad()
logits, state = model(x, state)
loss = criterion(logits.view(-1, vocab_size), x.view(-1))
loss.backward()   # <-- error occurs here
optimizer.step()<ECODE>

apaszke: 

apaszke: 

ematvey:
How to transfer an existing tensor to another device based on other tensor

ypxie: For example <SCODE>tensor_a is currently in cpu
I wanna transfer tensor_a to cpu or gpu based on 
tensor_b's residential state.
<ECODE>

apaszke: <SCODE>tensor_a = tensor_a.cuda(b.get_device()) if b.is_cuda else tensor_a
<ECODE>

ypxie: good to know that, thanks!
Loading videos from folders as a dataset object
Category: vision

Atcold: Now I have that <SCODE>next(DataLoaderIter)
<ECODE> will call <SCODE>DataLoaderIter._next_indeces()
<ECODE> <SCODE>list(range(t * batch_size, (t + 1) * batch_size))
<ECODE> <SCODE>dataset[i] for i in indeces
<ECODE> Can anyone provide feedback on this strategy? Does it sound reasonable, or am I missing something? <SCODE>   0     5     10    15    20    25
0  >xxxx|xxxxx|xxxxx|xxxxx|xxxxx|xxx
1  xxxxx|xxxxx|xxx>x|xxxxx|xxxxx|xxx
2  xxxxx|xx>xx|xxxxx|xxxxx|xxxxx|xxx
3  xxxxx|xxxxx|xxxxx|xxxxx|xxxx>|xxx
4  xxxxx|xxxxx|xxxxx|x>xxx|xxxx
<ECODE> <SCODE>   0     5     10    15    20    25
0  0oooo|ooooo|ooooo|ooooo|ooooo|ooo
1  ooooo|ooooo|ooo>x|xxxxx|xxxxx|xxx
2  xxxxx|xx>xx|xxxxx|xxxxx|xxxxx|xxx
3  xxxxx|xxxxx|xxxxx|xxxxx|xxxx>|xxx
4  xxxxx|xxxxx|xxxxx|x>xxx|xxxx0|ooo
<ECODE>

raivokoot:
How can I use legacy torch module?

Seungyoung_Park: Because I need to employ nn.Replicate from torch, I tried the below <SCODE>import torch
import torch.nn.functional as F
from torch.legacy import nn as nn
from torch.autograd import Variable

input = Variable(torch.rand(32,128,1,1))
replicate = nn.Replicate(nf = 32, dim = 3)
output = replicate(input)
<ECODE> But, I got the error message TypeError: ‘Replicate’ object is not callable Can you tell me how I can use nn.Replicate ?

ruotianluo: torch.expand can do the same thing, however the arguments are a little different from nn.Replicate.

Seungyoung_Park: Thanks for your reply.

apaszke:
Improve error message: .numpy() not supported on cuda

robromijnders: I was struggling to feed my predictions into an sklearn function <SCODE>M_ = M.data.numpy()
<ECODE> I got error <SCODE>RuntimeError: numpy conversion for LongTensor is not supported
<ECODE> <SCODE>M_ = M.data.cpu().numpy()
<ECODE> I suggest to improve this error message. Maybe something like <SCODE>RuntimeError: numpy conversion for LongTensor is not supported on CUDA
<ECODE> That way, novices like me will know what to do

smth: thanks, it’s a good suggestion. i’ll work on this.
Can a thread be assigned to each GPU?

Seungyoung_Park: In Torch, we can assign a thread to each GPU as like below. <SCODE>net = nn.DataParallelTable(1, true, true):add(net, {1, 2}):threads(function() require 'cudnn' cudnn.benchmark = true  cudnn.fastest = true end)
<ECODE>

smth: unlike LuaTorch, we dont dispatch DataParallel via python threads in PyTorch. No, we plan to (and do) dispatch multi-GPU differently at high performance, without the user needing to do anything. This is irrelevant to the user, we are working on improving the internals and multi-gpu perf. Depending on how many parameters you have in your model, this is possible.

Seungyoung_Park: Thanks for your reply.
Autogradable image resize

meijieru: 

smth: you can look at these Upsampling* modules to resize up: To resize down, you can use AvgPool2d

meijieru: 

thnkim: 

apaszke: 

jwin: Has this been implemented yet for float scale factors? Greatly appreciated!

flavio: Jwin, I had the same problem and I solved using a sampling grid: <SCODE>def downsampling(x, size=None, scale_factor=None, mode='nearest'):
	# define size if user has specified scale_factor
	if size is None: size = (int(scale_factor*x.size(2)), int(scale_factor*x.size(3)))
	# create coordinates
	h = torch.arange(0,size[0]) / (size[0]-1) * 2 - 1
	w = torch.arange(0,size[1]) / (size[1]-1) * 2 - 1
	# create grid
	grid = torch.zeros(size[0],size[1],2)
	grid[:,:,0] = w.unsqueeze(0).repeat(size[0],1)
	grid[:,:,1] = h.unsqueeze(0).repeat(size[1],1).transpose(0,1)
	# expand to match batch size
	grid = grid.unsqueeze(0).repeat(x.size(0),1,1,1)
	if x.is_cuda: grid = grid.cuda()
	# do sampling
	return F.grid_sample(x, grid, mode=mode)
<ECODE>

ndronen: 

mariosfourn: Does anyone have a solution for autogradable general nearest neighbours upsampling?

ndronen: 

ptrblck: <SCODE>x = torch.randn(1, 3, 24, 24)
up = nn.Upsample(size=47, mode='bilinear')
y = up(x)
print(y.shape)
> torch.Size([1, 3, 47, 47])
<ECODE>

chenchr: Is there any plan to support ‘nearest’ ? As in many dense estimation task, a edge preserving property is important, using bilinear  downsample will hurt that property…

solsol: 

ptrblck: 

solsol: Got it. Thanks a lot
Adaptive threshold possible?

Seungyoung_Park: I am trying to threshold Variables using the adaptive value as below. <SCODE>import torch
from torch.autograd import Variable
import torch.nn as nn

x = Variable(torch.rand(10))
maxx = torch.max(x)
m = nn.Threshold(0.6*maxx,0)
y = m(x)
<ECODE> Thanks in advance for your help.

albanD: Hi, I think you want to use directly the nn functional version of threshold: <SCODE>import torch
from torch.autograd import Variable
import torch.nn as nn

x = Variable(torch.rand(10))
maxx = torch.max(x)
y = nn.functional.threshold(x, 0.6*maxx.data[0], 0)
<ECODE>

Seungyoung_Park: 

albanD: Hi,

Seungyoung_Park: Thanks for your reply. It works as I want.

Seungyoung_Park: When I use this snippet in my code, I got an error as <SCODE>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation
<ECODE> My test code is as follows. <SCODE>import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torch.autograd import Variable

class AdaReLU(nn.Module):
def __init__(self):
    super(AdaReLU, self).__init__()
    self.relu = nn.ReLU()
    
def forward(self, x):
    y = x
    for i in range(x.size(0)):
        for ii in range(x.size(1)):
            maxx = torch.max(x[i,ii,0,:])
            y[i,ii,0,:] = F.threshold(x[i,ii,0,:], config.max_th*maxx.data[0], 0)

    x = self.relu(y)
    return x

class TestNet(nn.Module):
def __init__(self):
    super(TestNet, self).__init__()
    self.conv_1 = nn.Conv2d(len(config.alphabet), config.basic_frames, kernel_size = (1,3), stride = (1,2), padding = (0,0), bias = False)
    self.batchnorm_1 = nn.BatchNorm2d(config.basic_frames)
    self.conv_2 = nn.ConvTranspose2d(config.basic_frames, len(config.alphabet), kernel_size = (1,3), stride = (1,2), padding = (0,0), bias = False)
    self.batchnorm_2 = nn.BatchNorm2d(len(config.alphabet))
    self.adarelu = AdaReLU()

def forward(self, x):
    x = self.adarelu(self.batchnorm_1(self.conv_1(x)))
    x = F.sigmoid(self.batchnorm_2(self.conv_2(x)))
    return x

x = Variable(torch.rand(32,68,1,2283))
model = TestNet()
optimizer = optim.SGD(model.parameters(), lr =0.01, momentum=0.9)
optimizer.zero_grad()
output = model(data)
loss = F.binary_cross_entropy(output, data)
loss.backward()
optimizer.step()<ECODE>

albanD: 

Seungyoung_Park: 

albanD: Hi, I think you can replace the loop by: <SCODE>y = x * (1 - x.lt(maxx.expand_as(x))).float()
<ECODE>

apaszke: The whole function can be simplified to this: <SCODE>def AdaReLU(x):                    
    max_val, _ = torch.max(x, 3)   
    mask = x < max_val.expand_as(x)
    mask.detach_()                 
    x = x.clone()                  
    x[mask] = 0                    
    return F.relu(x)               
<ECODE>

Seungyoung_Park:
How to free the memory used by processed layers in the testing/eval phase?

dhm: Hi. In the testing/eval phase, after one layer is processed is there anyway to free its memory? Or can the output tensors of all layers share the same memory, since the outputs are not needed for calculating the gradients. Thanks.

apaszke: Training is already quite memory efficient. If it doesn’t need to hold onto some input/output because it can compute the gradient in another way, it won’t do it.

dhm: Thanks. It is so convenient to use!
Multi-dimensional Labels in pytorch

sarthak1996: 

apaszke: 

sarthak1996: I have some co-ordinates for bounding boxes in the train dataset and I want the net to train on those.

apaszke: What kind of loss function do you want to use then?

sarthak1996: 

apaszke: Sooo you’re predicting a 0 or 1 for each pixel? Why does your output have 3 channels then?

apaszke: L1 and L2 losses should be easy to write yourself using only autograd operations. It’s just a subtraction + abs or pow.
Cuda runtime error (2) : out of memory

mderakhshani: I would like to know how can I solve my problem. What is the source of problem? My Code:

apaszke: The problem is exactly what the error says, you ran out of memory on the GPU. You can try reducing the batch size or image sizes.

sarthak1996: 

Ismail_Elezi: To complement apaszke’s answer, in case you are using Jupyter and you ‘run’ multiple times the cell which creates the network, put always a delete net on that cell, otherwise you might be creating multiple neural networks which will cause similar problems. If not, just try to find the maximum batch size which works for your hardware.

afrozalm: 

afrozalm: 

PeterXiaoGuo: I face the same problem with you and I find the problem results from the upsampling layer. When I use nvidia-smi to monitor my GPU memory usage, I find that during some time, the memory requirement for transpose2d for upsampling will be doubled and it tells me out of memory… Maybe consider to reduce the batch size.
Help with svd approximation

gsp-27: Hi I am trying to implement “Exploiting linear structure for efficient evaluation” paper and it requires me to perform SVD on the weight tensor and find its rank 1 approximation and now these rank 1 filters perform the convolution operation. I wrote a simple test suite to verify this and I am getting good speed ups. Here is the code which does that: <SCODE># doing passes
res1 = np.zeros((oH, pw))
out = np.zeros((oH, oW))
t = time.time()
for i in xrange(oH):
    hst, hend = i*stride, i*stride+HH
    patch = padded_x[hst:hend, :]
    res1[i] = np.sum(patch*u, axis=0)
for i in xrange(oW):
    wst, wend = i*stride, i*stride+WW
    patch = res1[:, wst:wend]
    out[:, i] = np.sum(patch*v, axis=1)
elapsed = time.time() - t
print "time taken to do this awesome operation is {}".format(elapsed)
print out
<ECODE> I wanted to know if there is a way I can do this using pytorch and get the speed improvements?

apaszke: 

gsp-27: If you would guide me in the right direction, I would be grateful. <SCODE>import numpy as np
import torch
import torch.nn as nn
from torch.autograd import Variable
import torch.nn.functional as nnf
import time

image = torch.randn(512, 512)
image = Variable(image.view(1,1,512, 512))

u = torch.Tensor([1,2,1])
v = torch.Tensor([-1, 0, 1])
w = Variable(torch.ger(u, v))
w = w.view(1,1,3,3)

u = Variable(u.view(1,1,3,1))
v = Variable(v.view(1,1,1,3))

t = time.time()
res1 = nnf.conv2d(image, u, stride=1, padding=1)
out = nnf.conv2d(res1, v, stride=1)
print "time taken is {}".format(time.time()-t)
print out

t = time.time()
out1 = nnf.conv2d(image, w, stride=1, padding=1)
print "time taken by normal is {}".format(time.time()-t)
print out1
<ECODE>

apaszke: I don’t think it’s that surprising that a single convolution is faster. For each of them, you have to preprocess the image in some way, most libraries are optimized for 3x3 filters, since they’re one of the most commonly used sizes, and you benefit from data locality if you only do a single conv (no need to transfer the image back and forth).

gsp-27: So i would not get any improvements if I implement the forward and backward pass for this as per your previous suggestion of extending “autograd” Thanks

apaszke: Correct. I just wanted to tell you how could you do that.
Does torch.utils.data.DataLoader processing disk i/o?

yunjey: <SCODE>transform=transforms.Compose([transforms.ToTensor(),
                              transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5)),
                             ])
trainset = torchvision.datasets.CIFAR10(root='./data', train=True, download=True, transform=transform)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=4, 
                                          shuffle=True, num_workers=2)

testset = torchvision.datasets.CIFAR10(root='./data', train=False, download=True, transform=transform)
testloader = torch.utils.data.DataLoader(testset, batch_size=4, 
                                          shuffle=False, num_workers=2)

dataiter = iter(trainloader)
images, labels = dataiter.next()
<ECODE>

apaszke: 

isalirezag:
Train 3D Networks with Pytorch

ltnghia: Hi all, Thank you in advance.

apaszke: 

ltnghia: Thanks. Do you support Deconv3D?

apaszke: 

YunnFeng: The return 4D elements is (c,d,h,w) ?

Ismail_Elezi: The return is a tuple of 5 elements (N, c, d, h, w).

apaszke: You need to return 4D elements from the dataset, so they will be concatenated into 5D batches. Then, we only support batch mode in nn, so you’ll be using 5D tensors throught the network.

YunnFeng: 

apaszke:
Support for slicing with a step != 1?

Ajoo: Currently when attempting to slice a tensor or Variable object with a step different from 1 I get a Runtime Error: Is support for this opperation coming anytime soon and what is the best way to reverse a tensor along an axis currently? Best I can come up with is doing it via index_select which is clunky and copies the tensor… Doing it via numpy is also not possible as from_numpy() doesn’t seem to work with negative strides.

apaszke: 

acgtyrant: So how do I flip the image?

acgtyrant: 

bobiblazeski: Strange it sometimes work, and sometimes doesn’t: <SCODE>tensor([[-0.8948],
            [-0.3556],
            [ 1.2324],
            [ 0.1382],
            [-1.6822]])
<ECODE> <SCODE>_ ValueError                                Traceback (most recent call last)_
_ <ipython-input-42-67cae86c60db> in <module>_
_torch.mm(features, torch.reshape(weights, weights[::-1]))_
_ ValueError: negative step not yet supported_
<ECODE> <SCODE>tensor([[-0.8948],
            [-0.3556],
            [ 1.2324],
            [ 0.1382],
            [-1.6822]])
<ECODE>

Shriya_Kak: Hi guys, I am also facing the same issue. Could anyone please confirm if the issue is resolved?

albanD: Which issue? negative strides? I’m afraid we do not support it.

Shriya_Kak: <SCODE>detected_faces = face_detector.detect_from_image(data[..., ::-1].copy())
<ECODE> ValueError: negative step not yet supported This is what I am getting.

albanD: Hi,