foldl/torch-forum huggingface dataset:
Platform not supported - Python 2.7

schroeder-dewitt: Trying to install x86_64 version of Python 2.7 pytorch as advertised: Output of uname -a : For Python 3, similar procedure works without problem.

schroeder-dewitt: ok, simple mistake - forgot that now pip will be pip3 by default. installing with pip2.7 works. You may want to update install instructions to use pip3 / pip2.7 explicitely!
Can not download the linux pip cud 8.0 compiled pytorch

Gao_Peng: I use pip to install however run into this error. Storing debug log for failure in /home/gaop/.pip/pip.log

Gao_Peng: 

apaszke: It must be some kind of a network error at your side (maybe you don’t have some trusted SSL certificates installed)?

Mathieu-Seurin: Isemel answer : Worked for me
Combine multiple Variables to one

melody-rain: x is a Variable which has 3 dims. <SCODE>x.size()[0] is sequence length
x.size()[1] is batch size.
<ECODE> I did something below(for LSTM’s input): <SCODE>temp = []
for i in xrange(len(x)):
    temp.append(nn.Linear(512, 256)(x[i]))
<ECODE> As you can see here, I got many Variables stored in temp. But I what I want is to combine these new Variables into one Variable. How can I do it? Many thanks.

Lyken17: 

melody-rain: cat’s input should be tensors, but temp’s elements are Variables.

jekbradbury: torch.cat also works with Variables.
How can I print the shape of a tensor inside the forward function?

wasiahmad: I have an attention decoder whose forward function is as follows. <SCODE>def forward(self, input, hidden, encoder_outputs):
    embedded = self.embedding(input).view(1, 1, -1)
    embedded = self.drop(embedded)
    attn_weights = F.softmax(self.attn(torch.cat((embedded[0], hidden[0]), 1)))
<ECODE> Any suggestion?

jekbradbury: 

Nabarun_Goswami: 

Mithun_Paul: 

kingxueyuf: 

jekbradbury: 

eduamf: Use print(embedded) to see the shape, or embedded.eval() If you want to see the content, embedded.weight will show you the tensor and if it requires grad.
How to resolve the RuntimeError related to torch.cat() function?

wasiahmad: I have an attention decoder whose forward function is as follows. <SCODE>def forward(self, input, hidden, encoder_outputs):
    embedded = self.embedding(input).view(1, 1, -1)
    embedded = self.drop(embedded)
    attn_weights = F.softmax(self.attn(torch.cat((embedded[0], hidden[0]), 1)))
<ECODE> When ever the forward function gets called, I get the following error. How to resolve this problem? The problem is associated with torch.cat(). Please help.

smth: you can add prints to figure out the problematic tensor shape for torch.cat

wasiahmad: Thanks for your reply. I printed the shape and found the following. <SCODE>embedded = self.embedding(input).view(1, 1, -1)
embedded = self.drop(embedded)
print(embedded[0].size(), hidden[0].size())
<ECODE>

Spider101: 

wasiahmad: 

Spider101: 

wasiahmad: Now it makes sense! Thanks.
Is there any way to get second order derivative or Hessian Matrix?

LiuzcEECS: I am trying to get the Hessian Matrix of weights in a convolutional kernel. However,  there is no API which can do the job like Tensorflow.

smth: 

apaszke: AFAIK TensorFlow will return you a Hessian-vector product like most automatic differentiation software.

Ilya_Kostrikov: Any updates regarding second order derivatives in PyTorch?

jekbradbury: 

shijie-wu: 

adrianalbert: Has there been any update on this? That is, how to get the Hessian (even if just the diagonal) in Pytorch? Would something like this work:

vabh: 

adrianalbert: Thanks, will look into that!

adrianalbert: It looks like torch.autograd does not have a “grad” function? I’m using the latest pytorch (0.1.12_2) ImportError: cannot import name grad

vabh: 

adrianalbert: Got it, thanks! I installed Pytorch from source and now I have access to the grad and backward functions in torch.autograd. However when I try to use grad my kernel just crashes. I use like this: Is this not intended to be used with non-scalar (multiple-dimensional) Tensors? The examples on the github work fine. Thanks!

vabh: 

adrianalbert: I see, that makes sense, given that the model is a CNN with skip connections. Thanks!

Brando_Miranda: Are there any updates on this? Its been some time. Whats wrong with just doing w.grad.backward()?

fehiepsi: 

Albert_Chen: <SCODE>import torch
from torch import Tensor
from torch.autograd import Variable
from torch.autograd import grad
from torch import nn

torch.manual_seed(623)

x = Variable(torch.ones(2,1), requires_grad=True)
A = torch.FloatTensor([[1,2],[3,4]])

print(A)
print(x)

f = x.view(-1) @ A @ x
print(f)


x_1grad, = grad(f, x, create_graph=True)
print(x_1grad)
print(A @ x + A.t() @ x)

x_2grad0, = grad(x_1grad[0], x, create_graph=True)
x_2grad1, = grad(x_1grad[1], x, create_graph=True)

Hessian = torch.cat((x_2grad0, x_2grad1), dim=1)
print(Hessian)

print(A + A.t())
<ECODE>

JIANG_GUOQING: I’ve worked on this issue for some time, plz feel free to use it below.
Strangely slow weight loading [fixed]

tachim: I’m running into a strange issue where loading weights is quite slow. Specifically for the DC-GAN example in the repo, loading the weights for a DC-GAN with 10 latent variables takes 150 seconds which doesn’t seem right given the size of the model. The code to create/load the model is below; is anything obviously wrong here? Thanks! <SCODE>class _netG(nn.Module):
    def __init__(self, ngpu):
        super(_netG, self).__init__()
        self.ngpu = ngpu
        self.main = nn.Sequential(
            # input is Z, going into a convolution
            nn.ConvTranspose2d(     nz, ngf * 8, 4, 1, 0, bias=False),
            nn.BatchNorm2d(ngf * 8),
            nn.ReLU(True),
            # state size. (ngf*8) x 4 x 4
            nn.ConvTranspose2d(ngf * 8, ngf * 4, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ngf * 4),
            nn.ReLU(True),
            # state size. (ngf*4) x 8 x 8
            nn.ConvTranspose2d(ngf * 4, ngf * 2, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ngf * 2),
            nn.ReLU(True),
            # state size. (ngf*2) x 16 x 16
            nn.ConvTranspose2d(ngf * 2,     ngf, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ngf),
            nn.ReLU(True),
            # state size. (ngf) x 32 x 32
            nn.ConvTranspose2d(    ngf,      nc, 4, 2, 1, bias=False),
            nn.Tanh()
            # state size. (nc) x 64 x 64
        )
    def forward(self, input): # [omitted for previty]

netG = _netG(ngpu)
netG.apply(weights_init)
with rtk.timing.Logger('load weights'):
    if opt.netG != '':
        netG.load_state_dict(torch.load(opt.netG))
print(netG)
<ECODE>
What is the difference between view() and unsqueeze()?

wasiahmad: <SCODE>input = torch.Tensor(2, 4, 3) # input: 2 x 4 x 3
print(input.unsqueeze(0).size()) # prints - torch.size([1, 2, 4, 3])
<ECODE> <SCODE>input = torch.Tensor(2, 4, 3) # input: 2 x 4 x 3
print(input.view(1, -1, -1, -1).size()) # prints - torch.size([1, 2, 4, 3])
<ECODE> Any help with good explanation would be appreciated!

pranav: Also, in the latest versions of PyTorch you can add a new axis by indexing with None as: <SCODE>>>> input = torch.Tensor(2, 4, 3) # input: 2 x 4 x 3
>>> print(input[None].size())
torch.Size([1, 2, 4, 3])
>>> print(input[:, None].size())
torch.Size([2, 1, 4, 3])
<ECODE>

apaszke: 

ecolss: 

apaszke: Yeah, but it works too. You can repeat a tensor along new dimensions as well.

nicoliKim: 

alexis-jacq: 

Wei: In pytorch 0.4, I get the same result for the input input = torch.Tensor(2, 4, 3) and print(input.unsqueeze(0).size()) It seems to me that the two (unsqueeze, view) change only the representation of a tensor.

mimoralea: <SCODE>In [1]: import torch                                                                                                                                                                                                                           

In [2]: im = torch.Tensor(40,40)                                                                                                                                                                                                               

In [3]: im.size()                                                                                                                                                                                                                              
Out[3]: torch.Size([40, 40])

In [4]: im.view(1,1,-1).size()                                                                                                                                                                                                                 
Out[4]: torch.Size([1, 1, 1600])
<ECODE>

iacob: (1,12), (2,6), (3,4), (6,2), (12,1)
Support for tensordot

shijie-wu: 

neverfox: <SCODE>import torch
import numpy as np

M = torch.randn(3, 3, 4)
v = torch.randn(2, 2, 3)

out = torch.Tensor(np.tensordot(v.numpy(), M.numpy(), axes=[[2], [0]]))
<ECODE> Not as concise, but it’s there.

miguelvr: Not exactly true, because if you want to backprop through it, you will break the graph…

neverfox: Ah, yes, that’s true and that would be important under most circumstances.

ajeetk: So, is there any workaround for tensordot in pytorch?

deanmark: 

tom:
Schema for feeding RNN training data

d10genes: <SCODE>             t=0  t=1  t=2  t=3  t=4  t=5  ...  t=598  t=599
sample       |---------------------|
sample            |---------------------|
sample                 |-----------------
...
sample                                          ----|
sample                                           ----------|
<ECODE> <SCODE>             t=0  t=1  t=2  t=3  t=4  t=5  t=6  t=7  ...  t=598  t=599
sample       |-----------|
sample                      |-----------|
sample                                     |-------
...
sample                                                    -----------|
<ECODE> Thanks

jekbradbury: 

d10genes: Thanks for the answer. I’m using time series data rather than LM but it sounds like the 2nd schema is still preferred. Is it important that the sequences are trained in order so that the relevant hidden state is reused, or is it common to shuffle them? It seems the hidden state may not be used anyways if they’re fed in parallel batches.

jekbradbury: For truncated BPTT training, it’s important that batches be processed in order so that hidden states are preserved.

ecolss: 

jekbradbury:
How to build my own modules(change cnn network structure)?

hongyuan: <SCODE>    I want to build a model like that. But I don't know how to use PyTorch basic modules to do that. I plan to train the  horizontal  parameters(CNN) at first. Then,train the vertical parameters(RNN). But,I don't know how to combine them together and train this.<ECODE>

pranav: This model looks like a Stacked Convolutional Recurrent Neural Network. <SCODE>import torch.nn as nn
from torch.autograd import Variable

class RNN(nn.Module):
    def __init__(self, input_size, hidden_size, output_size):
        super(RNN, self).__init__()
        
        self.hidden_size = hidden_size
        
        self.i2h = nn.Linear(input_size + hidden_size, hidden_size)
        self.i2o = nn.Linear(input_size + hidden_size, output_size)
        self.softmax = nn.LogSoftmax()
    
    def forward(self, input, hidden):
        combined = torch.cat((input, hidden), 1)
        hidden = self.i2h(combined)
        output = self.i2o(combined)
        output = self.softmax(output)
        return output, hidden

    def initHidden(self):
        return Variable(torch.zeros(1, self.hidden_size))

n_hidden = 128
rnn = RNN(n_letters, n_hidden, n_categories)
<ECODE>

hongyuan: Thank you! I’ll try.
Cannot figure out how to resolve AssertionError

randy: I’ve been trying to figure out how to resolve this error for a couple of hours to no luck. I’ve combed through the documentation, but couldn’t find anything. Can someone explain to me why the AssertionError is being thrown? Here’s the code: <SCODE>import torch
import torch.nn as nn
import torchvision.transforms as transforms
from torch.autograd import Variable

num_epochs = 15
batch_size = 500
learning_rate = 0.003

class CNN(nn.Module):
    def __init__(self):
        super(CNN, self).__init__()
        self.layer1 = nn.Sequential(
            nn.Conv2d(1, 16, kernel_size=5),
            nn.BatchNorm2d(16),
            nn.ReLU(),
            nn.MaxPool2d(2))

	def forward(self, x):
        out = self.layer1(x)
        return out

cnn = CNN()

criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(cnn.parameters(), lr=learning_rate)

img = torch.Tensor(img)
#img & labl are numpy ndarrays
labl = torch.Tensor(labl)
image = Variable(img)
label = Variable(labl)

optimizer.zero_grad()
output = cnn(image)
<ECODE> Here’s the error message: <SCODE>Traceback (most recent call last):
  File "conv_net.py", line 84, in <module>
    output = cnn(image)
  File "/home/randy/.virtualenv/ml-pytorch/local/lib/python2.7/site-packages/torch/nn/modules/module.py", line 202, in __call__
    result = self.forward(*input, **kwargs)
  File "conv_net.py", line 50, in forward
    out = self.layer1(x)
  File "/home/randy/.virtualenv/ml-pytorch/local/lib/python2.7/site-packages/torch/nn/modules/module.py", line 202, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/randy/.virtualenv/ml-pytorch/local/lib/python2.7/site-packages/torch/nn/modules/container.py", line 64, in forward
    input = module(input)
  File "/home/randy/.virtualenv/ml-pytorch/local/lib/python2.7/site-packages/torch/nn/modules/module.py", line 202, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/randy/.virtualenv/ml-pytorch/local/lib/python2.7/site-packages/torch/nn/modules/conv.py", line 237, in forward
    self.padding, self.dilation, self.groups)
  File "/home/randy/.virtualenv/ml-pytorch/local/lib/python2.7/site-packages/torch/nn/functional.py", line 38, in conv2d
    return f(input, weight, bias) if bias is not None else f(input, weight)
  File "/home/randy/.virtualenv/ml-pytorch/local/lib/python2.7/site-packages/torch/nn/_functions/conv.py", line 35, in forward
    output = self._update_output(input, weight, bias)
  File "/home/randy/.virtualenv/ml-pytorch/local/lib/python2.7/site-packages/torch/nn/_functions/conv.py", line 99, in _update_output
    output = self._thnn('update_output', input, weight, bias)
  File "/home/randy/.virtualenv/ml-pytorch/local/lib/python2.7/site-packages/torch/nn/_functions/conv.py", line 159, in _thnn
    impl = _thnn_convs[self.thnn_class_name(input)]
  File "/home/randy/.virtualenv/ml-pytorch/local/lib/python2.7/site-packages/torch/nn/_functions/conv.py", line 140, in thnn_class_name
    assert input.dim() == 4 or input.dim() == 5
AssertionError<ECODE>

alexis-jacq: You can simply add a fake batch dimension with unsqueeze(): img = torch.Tensor(img).unsqueeze(0)
Customized dataloader distorts images

waitwaitforget: Hi, All. Here is my code: <SCODE> # training set
    transform = transforms.Compose([transforms.Scale(28), transforms.ToTensor(), ])
    trainset = USPS(root='./data', train=True, download=False, transform=transform)
# split data
    trainloader = torch.utils.data.DataLoader(trainset, batch_size=100, shuffle=False, num_workers=1)

# kernel code in USPS
def load(self):
        
        # process and save as torch files
        print('Processing...')
        data = sio.loadmat(os.path.join(self.root,'usps_train.mat'))
        traindata = torch.from_numpy(data['data'].transpose())
        
        traindata = traindata.view(16,16,-1).permute(2,1,0)
        trainlabel = torch.from_numpy(data['labels'])

        data  = sio.loadmat(os.path.join(self.root,'usps_test.mat'))
        testdata = torch.from_numpy(data['data'].transpose())

        testdata = testdata.view(16,16,-1).permute(2,1,0)
        testlabel = torch.from_numpy(data['labels'])
        training_set = (traindata, trainlabel)
        test_set = (testdata, testlabel)
        with open(os.path.join(self.root, self.training_file), 'wb') as f:
            torch.save(training_set, f)
        with open(os.path.join(self.root, self.test_file), 'wb') as f:
            torch.save(test_set, f)

        print('Done!')
<ECODE>
Single node multiGPU + multiCPU parallelism?

FuriouslyCurious: I got the DataParallel() example for multi-GPU working successfully for my model. However, I happen to have a 64-core Xeon Phi CPU, and I can’t stand looking at it sitting idle.  How can I saturate the CPU by assigning it some work?  In other words, can I split the workload on GPU0, GPU1, and CPU0-255 and sum the gradients on CPU? Thank you!

jekbradbury: I don’t know very much about Xeon Phi, but I believe that Intel has an MKL version that will parallelize BLAS calls over the whole chip, making it work like a single GPU device. I would do some experiments to compare speed for each part of your network, then use model parallelism to put submodules on the device they work best on. Or you could subclass/modify the code for DataParallel to allow the CPU (Phi) to be one of the included devices.

FuriouslyCurious: Thanks James. I had a subclassing script for Keras based on Kuza55’s script: it replicated models to /gpu0, /gpu1, and /cpu0-255. I will look into subclassing data_parallel.py to use both CPU and GPU.  What are the device IDs for CPUs inside PyTorch?

jekbradbury: CPUs don’t have device IDs; there’s just one kind of CPU tensor and operations with them are implemented in TH and farmed out to MKL, which would then have its own strategy for parallelizing over the Phi.
Where is the source code for `torch.Tensor.contiguous()`?

dl4daniel: data = data.view(bsz, -1).t().contiguous()

apaszke: 

dl4daniel: Thanks a lot!

apaszke: <SCODE>x = torch.randn(5, 4)
print(x.stride(), x.is_contiguous())
print(x.t().strinde(), x.t().is_contiguous())
x.view(4, 5) # ok
x.t().view(4, 5) # fails
<ECODE>

dl4daniel: 

kj7kunal: 

yassineAlouini: Thanks for the clarification. I guess there is a minor typo in the second print: print(x.t().strinde(), x.t().is_contiguous()) should be print(x.t().stride(), x.t().is_contiguous())
LSTM hidden & cell outputs and packed_sequence for variable-length sequence inputs

supakjk: I have three questions regarding LSTM. Thanks!

jekbradbury: For a forward RNN, the returned last hidden and cell values are e00 if you don’t use PackedSequence, but they’re ezw if you do. For the backward direction of a bidirectional RNN, they’re axv in both cases, but the RNN will have started at ezw in the PackedSequence case and e00 in the case without it.

supakjk: 

cerisara: 

jekbradbury: Yes, the padding is always assumed to be on the right. There are reasons to put the padding on the left if your model is performing computations on the padding tokens and you want to minimize the distortion, but if there are reasons to use left-padding rather than right-padding when the padding tokens won’t be used, we must have overlooked them.

cerisara: 

Nick_Young: I have a question. What does it mean “padding on the right”? In real situation, “a” is represented by a vector, right? so one sample “x y z 0 0” is represented by a matrix. So what you mean “padding on the right” is “padding several all-zero lines under the useful data”. Am I right?

jekbradbury: Yes. The padding is on the right if the timesteps of the sentence are left to right and you think of the vector of x or a as going into the screen.

JooSung_Yoon: I can not understand how it works. I saw doc in pytorch but I couldn’t figure it out. is there anyone who help me??

smth: providing initial hidden state is independent of whether you give packed sequences or not. If you dont give initial hidden state, it is initialized to zero.

yukw777:
Combine / concat dataset instances

bodokaiser: I came up with two ideas: <SCODE>class Concat(Dataset):

    def __init__(self, datasets):
        self.datasets = datasets
        self.lengths = [len(d) for d in datasets]
        self.offsets = np.cumsum(self.lengths)
        self.length = np.sum(self.lengths)

    def __getitem__(self, index):
        for i, offset in enumerate(self.offsets):
            if index < offset:
                if i > 0:
                    index -= self.offsets[i-1]
                return self.datasets[i][index]
        raise IndexError(f'{index} exceeds {self.length}')

    def __len__(self):
        return self.length
<ECODE> <SCODE>loader = itertools.chain(*[MyDataset(f'file{i}') for i in range(1, 4)])
<ECODE>

apaszke: 

bodokaiser: <SCODE>x = itertools.repeat(itertools.chain.from_iterable([dataset1, dataset2]), times=epochs)

next(next(iter(x))

# or 

for epoch in x:
    for (inputs, targets) in epoch:
       print(inputs)
<ECODE>

apaszke: <SCODE>for epoch in range(num_epochs):
    dset = itertools.chain(...)
    dloader = # create DataLoader
    for ... in dloader:
        ...
<ECODE>

acgtyrant: 

jnhwkim: 

kavyajeetbora: <SCODE>def return_all_items(dataset):
  all_items = []
  for i in range(len(dataset)):
    all_items.append(dataset[i])
  return all_items
<ECODE> <SCODE>list1 = return_all_items(original_data)
list2 = return_all_items(transformed_data)
list1.extend(list2)
<ECODE> then converting the list to dataset object <SCODE>class AugmentedDataset(Dataset):
  
  def __init__(self, combined_list, transform=None):
    self.combined_list = combined_list
    
  def __len__(self):
    return len(self.combined_list)
  
  def __getitem__(self, idx):
    sample = self.combined_list[idx]      
    return sample
  
augmented_dataset = AugmentedDataset(list1)
<ECODE>

jasha:
Cannot install PyTorch on Ubuntu14.04 via pip

Kalamaya: Hi, made a virtual environment via: virtualenv pytorch011 I then source it, and then run the command to install pytorch via pip with: pip install https://download.pytorch.org/whl/cu80/torch-0.1.10.post2-cp27-none-linux_x86_64.whl However I get this error: Not sure what to do here? Thanks

jekbradbury: 

Kalamaya: I figured as much, so I am in the process of updating python right now to 2.7.13…

melgor: Is sth wrong with my env?

smth: 

Kalamaya: I ended up just installed python 2.7.13 and now it works.

Adel_Saleh1: 

ginobilinie: This really works!!! Thanks.
How to declare a torch with unknown value in one dimension

wasiahmad: I have a list of sentences and I am convert the list to a 3d tensor. The first dimension represents number of sentences, second dimension represents number of words and third dimension represents word embedding size. The problem is, number of words can vary in sentences. I tried to create a 3d tensor as follows. <SCODE>all_sentences1 = torch.FloatTensor(len(instances), None, args.emsize)
<ECODE> But this gives me the following error. <SCODE>TypeError: torch.FloatTensor constructor received an invalid combination of arguments - got (int, NoneType, int), but expected one of:
 * no arguments
 * (int ...)
      didn't match because some of the arguments have invalid types: (int, NoneType, int)
 * (torch.FloatTensor viewed_tensor)
 * (torch.Size size)
 * (torch.FloatStorage data)
 * (Sequence data)
<ECODE> How can I declare a 3d tensor?

jekbradbury: 

wasiahmad: Edit

jekbradbury: 

apaszke: You need to understand that PyTorch works differently than static graph frameworks. You can’t define “placeholders” with unspecified sizes, but you can pass in tensors of different sizes to your model without any modifications. Instead of reusing a single placeholder, you always have to operate on real data.
CPU usage issue

xwgeng: 

xwgeng: And I attempt to install from source or binary, but no change. And the OS is CentOS Linux release 7.3.16.11

apaszke: What’s your PyTorch version? Also, did you try running e.g. with 4 OMP threads? I think the problem appears because when you’re using all the cores, they’re competing over cache space and can’t proceed as effectively.

xwgeng: My pytorch version is 0.1.10. And as you say, set the OMP_NUM_THREADS=4 and it works well. Thanks for your reply.