foldl/torch-forum huggingface dataset:
Autoencoders in Pytorch

dmadeka1: Would Pytorch support something like this? How does one go about implementing a simple Autoencoder? <SCODE>class Encoder(nn.Module):
    def __init__(self):
        super(Encoder, self).__init__()
        self.fc1 = nn.Linear(784, 32)

    def forward(self, x):
        return F.sigmoid(self.fc1(x))

class Decoder(nn.Module):
    def __init__(self):
        super(Decoder, self).__init__()
        self.fc1 = nn.Linear(32, 784)
        
    def forward(self, x):
        return F.sigmoid(self.fc1(x))

class AutoEncoder(nn.Module):
    def __init__(self):
        super(AutoEncoder, self).__init__()
        self.fc1 = Encoder()
        self.fc2 = Decoder()

    def forward(self, x):
        return self.fc2(self.fc1(x))

model = AutoEncoder()
optimizer = optim.Adam(model.parameters(), lr=0.5)
for epoch in range(1, 201):
    train(epoch)
    test(epoch, validation)
<ECODE>

dmadeka1: Just looking for the simplest possible implementation of an AutoEncoder here

apaszke: Yes, that should work.

alexis-jacq: If you really want to do the simplest, I would suggest: <SCODE>class Autoencoder(nn.Module):
    def __init__(self, ):
        super(Autoencoder, self).__init__()
        self.fc1 = nn.Linear(784, 32)
        self.fc2 = nn.Linear(32, 784)
        self.sigmoid = nn.Sigmoid()

    def forward(self, x):
        x = self.sigmoid(self.fc1(x))
        x = self.sigmoid(self.fc2(x))
        return x<ECODE>

dmadeka1: <SCODE>matrices expected, got 4D, 2D tensors at /data/users/soumith/miniconda2/conda-bld/pytorch-0.1.7_1485444530918/work/torch/lib/TH/generic/THTensorMath.c:857
<ECODE>

alexis-jacq: In that case your approach seems simpler. You can even do: <SCODE>encoder = nn.Sequential(nn.Linear(782,32), nn.Sigmoid())
decoder = nn.Sequential(nn.Linear(32,732), nn.Sigmoid())
autoencoder = nn.Sequential(encoder, decoder)<ECODE>

Wong_Fungtion: 

alexis-jacq: So you want a kind of balanced autoencoder, where Encoder = Transpose(Decoder)? In that case, I would do something like this: <SCODE>class BalancedAE(nn.Module):
    def __init__(self, ):
        super(BalancedAE, self).__init__()
        self.encoder = nn.Parameter(torch.rand(size_input, size_output))

    def forward(self, x):
        x = torch.sigmoid(torch.mm(self.encoder, x))
        x = torch.sigmoid(torch.mm(x, torch.transpose(self.encoder, 0, 1))
        return x
<ECODE>
PyTorch apply transform

dmadeka1: If we have a dataloader <SCODE>train_loader = torch.utils.data.DataLoader(trainset_imoprt, batch_size=200, shuffle=True)
semi_loader = torch.utils.data.DataLoader(trainunl_imoprt, batch_size=200, shuffle=True)
valid_loader = torch.utils.data.DataLoader(validset_import, batch_size=200, shuffle=True)
<ECODE>

apaszke: DataLoaders don’t support transforms, you need to make your dataset apply them.

smth:
Strange bug with numpy conversion and back

dmadeka1: Kind of weird bug. if i do: <SCODE>train_loader = torch.utils.data.DataLoader(trainset_imoprt, batch_size=200, shuffle=True)
semi_loader = torch.utils.data.DataLoader(trainunl_imoprt, batch_size=200, shuffle=True)
valid_loader = torch.utils.data.DataLoader(validset_import, batch_size=200, shuffle=True)

features = train_loader.dataset.train_data.numpy()
labels = train_loader.dataset.train_labels.numpy()

img = features
img = img.astype('float32')
lab = labels

img, lab = torch.from_numpy(img), torch.from_numpy(lab)

train = torch.utils.data.TensorDataset(img.unsqueeze(1), lab)
train_loader = torch.utils.data.DataLoader(train, batch_size=64, shuffle=False)
<ECODE>

apaszke: Maybe it’s because of the different batch size or shuffle? Also, I don’t understand why you have to unsqueeze a dimension that wasn’t used by the first train_loader.
[Solved]Confusing HTTP error when installing pytorch

squirrel: <SCODE>Fetching package metadata ...........
Solving package specifications: .

Package plan for installation in environment /home/zhanglu/anaconda3:

The following NEW packages will be INSTALLED:

pytorch:     0.1.9-py36_2 soumith
torchvision: 0.1.7-py36_1 soumith


The following packages will be UPDATED:

conda:       4.3.8-py36_0         --> 4.3.13-py36_0


Proceed ([y]/n)? y

pytorch-0.1.9- 100% |################################| Time: 0:02:49   1.44 MB/s
pytorch-0.1.9- 100% |################################| Time: 0:05:52 692.97 kB/s
pytorch-0.1.9- 100% |################################| Time: 0:01:15   3.23 MB/s

CondaError: CondaHTTPError: HTTP None None for url 
Elapsed: None

An HTTP error occurred when trying to retrieve this URL.
ConnectionError(ReadTimeoutError("HTTPSConnectionPool(host='binstar-cio-packages-prod.s3.amazonaws.com',
 port=443): Read timed out.",),)

CondaError: CondaHTTPError: HTTP None None for url 
Elapsed: None

An HTTP error occurred when trying to retrieve this URL.
ConnectionError(ReadTimeoutError("HTTPSConnectionPool(host='binstar-cio-packages-prod.s3.amazonaws.com',
 port=443): Read timed out.",),)

CondaError: CondaHTTPError: HTTP None None for url 
Elapsed: None

An HTTP error occurred when trying to retrieve this URL.
ConnectionError(ReadTimeoutError("HTTPSConnectionPool(host='binstar-cio-packages-prod.s3.amazonaws.com',
 port=443): Read timed out.",),)
<ECODE>

squirrel: 

wangg12: I am getting this error when I use conda to install magma-cuda80. Reboot does not help me. Is there any alternative way to install magma-cuda80?

chenyuntc: <SCODE>conda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/free/
conda config --set show_channel_urls yes
<ECODE>

wangg12: 

qichaotang: what do you to slove the problem about magma-cuda80 package?

1113: Though i reboot,can not solve the problem.

SarveshTS: Please update to the latest version of Navigator. $ conda update anaconda-navigator
Odd segmentation fault if torch is imported

Atcold: <SCODE>from skimage.data import astronaut
from matplotlib import pyplot as plt
import torch

f = plt.figure()
plt.imshow(astronaut())
f.savefig('0.pdf')
<ECODE> I have no idea what’s going on.

apaszke: 

apaszke: Reverse the order of imports for now

Atcold:
Sparse Tensors in PyTorch

Ilya_Kostrikov: What is the current state of sparse tensors in PyTorch?

smth: right now, the description / summary of this PR that was merged 11 hours ago gives a good idea of the current state of things:

apaszke: But we’re not documenting them on purpose, because they might undergo some more changes in the future. The first step was to implement sprase updates for Embedding. Can I ask what’s your use case?

Ilya_Kostrikov: I need just basic sparse matrix multiplication in order to implement a Graph ConvNet model. For example: and The Laplacian matrix is extremely sparse is this case.

apaszke: 

Ilya_Kostrikov: I need sparse x dense -> dense. So I can use PyTorch in this case. Thanks a lot!

apaszke: 

Ilya_Kostrikov: But it’s not in autograd yet? Is there a fast way to add it?

smth: a fast and local way is for you to write an autograd function for yourself.

hconroy: That’s my question too (now on 21st Sept).  Can anyone comment on the current state of sparse tensors in PyTorch? Thank you

11131: I would like to update variable with sparse gradients. and it’s normal.  I know that wasn’t support by tensorflow. so how about pytorch/. Thank you!

pradeepgopalakrishna: What is the status of Sparse Support in PyTorch? I’d like to contribute to the module. What are the current design strategies in place? There seems to be no detail or comprehensive discussion on this aspect.

rajeshpiryani:
Is it possible to check each ouput of a layer in a nn.Sequential or nn.Module?

maplewizard: I want to check the output value of each layer I build. but I didn’t find any element in layers. Is there any way to do this? For example, I have a model: <SCODE>model = nn.Sequential(nn.Linear(128, 64), nn.Linear(64, 32), nn.Linear(32, 1))
av = Variable(torch.rand(7, 128), requires_grad = True)
model.forward(av)
<ECODE> I want to see the output of each linear layer.

alexis-jacq:
Missing torch.nn.utils.clip_grad_norm in Anaconda-based pytorch Installation!

mderakhshani: So, I would like to know how can I repair this error? I think somehow the installation procedure, above mentioned command, is a little incomplete! Thanks

yunjey: Below is an example for installing PyTorch from source. <SCODE>$ git clone https://github.com/pytorch/pytorch
$ export CMAKE_PREFIX_PATH=/home/yunjey/anaconda3   # your anaconda path
$ conda install numpy mkl setuptools cmake gcc cffi
$ conda install -c soumith magma-cuda80
$ export MACOSX_DEPLOYMENT_TARGET=10.9   # if OSX
$ pip install -r requirements.txt
$ python setup.py install<ECODE>

mderakhshani: Thanks, problem was solved.

apaszke: We’ll be publishing new binaries soon and they will include that function.

mderakhshani: 

mderakhshani: 

apaszke: We’re working on fixing some issues with compilers over optimizing the binaries with AVX2 vector instructions, that are not available in lots of older CPUs. We’re going to publish them once we’re done with that.
[resolved] Is there MNIST dataset in torchvision.datasets?

yunjey: <SCODE>from tensorflow.examples.tutorials.mnist import input_data
mnist = input_data.read_data_sets("./mnist/data/")
x_train = mnist.train.images    # numpy array 
y_train = mnist.train.labels
x_test = mnist.test.images
y_test = mnist.test.labels
<ECODE> Is there any simple way to handle this in PyTorch?

yunjey: 

AjayTalati: Yes it already there - see here and here, The code looks something like this, <SCODE>train_loader = torch.utils.data.DataLoader(
    datasets.MNIST('../data', train=True, download=True,
                   transform=transforms.Compose([
                       transforms.ToTensor(),
                       transforms.Normalize((0.1307,), (0.3081,))
                   ])),
    batch_size=args.batch_size, shuffle=True, **kwargs)
test_loader = torch.utils.data.DataLoader(
    datasets.MNIST('../data', train=False, transform=transforms.Compose([
                       transforms.ToTensor(),
                       transforms.Normalize((0.1307,), (0.3081,))
                   ])),
    batch_size=args.batch_size, shuffle=True, **kwargs)
<ECODE>

AjayTalati: How do you subset the MNIST training data? It’s 60,000 images, how can you reduce it to say 2000? Here’s the code <SCODE>>>> from torchvision import datasets, transforms


>>> 
>>> 
>>> train_all_mnist = datasets.MNIST('../data', train=True, download=True,
...                    transform=transforms.Compose([
...                        transforms.ToTensor(),
...                        transforms.Normalize((0.1307,), (0.3081,))
...                    ]))
Files already downloaded
>>> train_all_mnist
<torchvision.datasets.mnist.MNIST object at 0x7f89a150cfd0>
<ECODE> It’s a bit ugly - anyone know a neater way to do this?

yunjey: what is your purpose of subsetting the training dataset?

AjayTalati: I’m interested in Omniglot, which is like an inverse, MNIST, lots of classes, each with a small number of examples. Take a look, here By the way - thank you for your tutorials - they are very clear and helpful to learn from. Best regards, Ajay

smth: 

AjayTalati: Spent a hour hacking together my own loader -  but this looks better! Seems to be the easiest data set for experimenting with one-shot learning?

AjayTalati: Whats the current best methodology for Omniglot? Who or what’s doing the best at the moment?

smth: 

AjayTalati: I though you might like this, implementation of a GAN on Omniglot,

ritchieng: Have you found a better way to do this?

AjayTalati: Nope sorry - been totally snowed under the past couple of months - not had any time to work on it. If you’re referring to the alternative cost functions for GANs I don’t think they make much difference? Cheers, Aj

pranav: It includes Omniglot data downloading and iterating scripts along with all the models proposed in the paper (the nets are written and trained with Theano).
Giving multiple parameters in optimizer

yunjey: How can i give multiple parameters to the optimizer? <SCODE>fc1 = nn.Linear(784, 500)
fc2 = nn.Linear(500, 10)
optimizer = torch.optim.SGD([fc1.parameters(), fc2.parameters()], lr=0.01)  # This causes an error.
<ECODE> In this case, for simplicity, i don’t want to use a class with nn.Module.

smth: you have to concatenate python lists: <SCODE>params = list(fc1.parameters()) + list(fc2.parameters())

torch.optim.SGD(params, lr=0.01)
<ECODE>

yunjey: Thanks, it works well.

Zichun_Zhang: 

MrPositron: Dear Soumith, While executing your approach, it says: TypeError: add() received an invalid combination of arguments - got (list), but expected one of: (Tensor other, Number alpha) (Number other, Number alpha) Can you help me?) Is there something wrong?

justusschock: Probably you set a bracket to the wrong place. You have to convert the parameters to a list separately and add the lists afterwards.

MrPositron: [SOLVED] <SCODE>params = self.net.state_dict()
pas = list(params['net.0.weight']) + list(params['net.0.bias']) + list(params['net.3.weight'] + list(params['net.3.bias'])  + list(params['net.6.weight']) + list(params['net.6.bias']))
self.optimizer1 = optim.Adam(pas, lr = 0.01)
<ECODE> Here is my code. I think everything is ok

huangbiubiu: <SCODE>import itertools

params = [fc1.parameters(), fc2.parameters()]

torch.optim.SGD(itertools.chain(*params), lr=0.01)
<ECODE>

riley: How is this different from just putting all of the tensors in a list directly as OP did?

NeverConvex: If your models are in a list or tuple somewhere already, you can also use a nested list comprehension: <SCODE>models = [nn.Linear(784, 500),
          nn.Linear(500, 10)
          ]
optimizer = torch.optim.SGD((par for model in models for par in model.parameters()),
                             lr=0.01)
<ECODE>

Shubajit_Saha: OP is passing a list of lists.

ncuxomun: Thank you! This helped me a lot!
Is there pretrained CNN (e.g. ResNet) for CIFAR-10 or CIFAR-100?

yunjey: 

smth: We dont offer pre-trained resnet with cifar. You might have to train one yourself.

skrish13: Maybe NIN with CIFAR? :3

skrish13: 

jekbradbury: 

skrish13: Oh okay! I’ll see if I can train it. Thanks again!

skrish13: Didn’t we have a repo of links to PyTorch implementation of various papers? (Or I’m confusing that with Chainer’s similar repo :S)

jekbradbury: 

skrish13: 

prlz77: 

Ismail_Elezi: Sorry for the dumb question, but how do you load a .pytorch file. Is it the same extension as .pth? This doesn’t seem to work:

prlz77: 

Ismail_Elezi: What would be TheModelClass for your model?

Ismail_Elezi: @jekbradbury Did you manage to train/load the model successfully? I managed to train it, but I am having a sizes do not match error, when I am trying to load it. I am trying to load it with the following code: but getting:

karlTUM: 

Ismail_Elezi: Yes, I managed to load ResNets that I trained on CIFAR datasets. The code for that is: The parameters for the model and for the net you are loading should agree. For what is worth, the accuracy I got was: with these hyperparameters: Would be interesting to see what happens if I use some more advanced optimizer like Adam. Anyway, in case you don’t have time to train them, I can upload the models today, during the afternoon.

prlz77: 

raaj043: Hi, Then on which dataset all the models are pretrained?

smth: they’re all pre-trained on Imagenet-12

raaj043: Thank you very much. Its helpfulu
Binary label: Tensor type?

lg-zhang: Hello, It seems to be a minor issue. I’m training a toy siamese network. My label is either -1 or 1, so I was using a LongTensor to store the label. It seems to me that torch complains because target is supposed to be a FloatTensor? (torch.cuda.LongTensor tensor, int value)
didn’t match because some of the arguments have invalid types: (torch.cuda.LongTensor, float) (torch.cuda.LongTensor tensor, torch.cuda.LongTensor other)
didn’t match because some of the arguments have invalid types: (torch.cuda.LongTensor, float) (torch.cuda.LongTensor tensor, int value)
didn’t match because some of the arguments have invalid types: (torch.cuda.LongTensor, float) (torch.cuda.LongTensor tensor, torch.cuda.LongTensor other)
didn’t match because some of the arguments have invalid types: (torch.cuda.LongTensor, float) Thanks!

apaszke: What loss are you using? I think casting the target to a FloatTensor should fix it.

lg-zhang: I assumed that a binary label should be an integer until it complains. Thanks!
How to creating a new layer with cuDNN?

Sunnydreamrain: HI, I have changed the RNN module into another one RNN1. Everything is working fine if I disable cuDNN. But once cuDNN is on, there is an error. It seems that I have to define it in cuDNN file? How to do that?

jekbradbury: 

Sunnydreamrain: 

jekbradbury: 

Sunnydreamrain: 

MatthijsVk: Any updates on this?
Gradient checking

alexis-jacq: Is there any simple and common gradient checking method, when extending an autograd function ?

jekbradbury: 

alexis-jacq: 

apaszke: It’s been added only recently and we forgot about that. Can you send a PR please?

alexis-jacq: 

yinhao: Hello, <SCODE>class SolveTrianguler(Function):
    # sloves A * x = b
    def __init__(self, trans=0, lower=True):
        super(SolveTrianguler, self).__init__()
        # trans=1, transpose the matrix A.T * x = b
        self.trans = trans
        # lower=False, use data contained in the upper triangular, the default is lower
        self.lower = lower
        # self.needs_input_grad = (True, False)

    def forward(self, matrix, rhs):
        x = torch.from_numpy(
            solve_triangular(matrix.numpy(), rhs.numpy(),
                             trans=self.trans,  lower=self.lower))

        self.save_for_backward(matrix, x)
        return x

    def backward(self, grad_output):
        # grad_matrix = grad_rhs = None
        matrix, x = self.saved_tensors
        # formula from Giles 2008, 2.3.1
        return -matrix.inverse().t().mm(grad_output).mm(torch.t(x)), \
               matrix.inverse().t().mm(grad_output)
<ECODE>
How to stop weight-decay from affectnig PReLU?

Kalamaya: Hi, so I would like to use the PReLU function, however as can be seen here, it warns us that we should not use weight-decay with it. Now in my code, the weight-decay is being used, and it is called like this: <SCODE># Optimizer
optimizer = optim.Adam(net.parameters(), 
                      lr = float(args.learningRate), 
                      weight_decay = float(args.l2_weightDecay)
                      ) 
<ECODE> What I do not understand though, is how can I specify explicitly, that I do not want my PReLU layers to be affected by weight decay?

apaszke: 

xuanzhangyang: 

falmasri:
Selecting entries of matrix based on on 1D tensor of indices

bilibili: Hi, My question is: if I have a 2D tensor and a 1D LongTensor which stores a list of indices, then I want to select an entry from each row of the 2D tensor based on the 1D longTensor. How can I achieve that in PyTorch? For example, a = [[1,2,3],[4,5,6],[7,8,9]], b = [2,1,0], then I would like to get [3, 5, 7] Also, say if I torch.sum([3,5,7]) and then take the derivative of it, is it doable in the sense that the partial derivatives will be successfully calculated? Thanks a lot!

apaszke: This should do it: <SCODE>x.gather(1, b.unsqueeze(1))
<ECODE>

jekbradbury: The values in b shouldn’t have to be unique, since they’re indexing into separate rows?

apaszke: Yes, of course. You’re right. They should be unique within a row, but it’s not a problem here.

bilibili: Thanks a lot for your answer! It works now.
CPU usage issue when using pytorch compiled from source

yikang-li: Hi, When I use the pytorch compiled from the src(pytorch-src), the CPU usage is much higher than the pytorch installed directly through conda(pytorch-conda). In addition, when using pytorch-src, gpu usage is lower than pytorch-conda. I just followed the instructions on github to install from src. I’m using python 2.7. Is there anything with my installation?

jekbradbury: What OS are you on? And by higher CPU usage do you mean that PyTorch is able to use more cores effectively and run faster?

yikang-li: I’m on Ubuntu 14.04. It uses more cores, but not faster nor efficient, I’m afraid. When I changed back to the pytorch installed through conda, it seems OK.

fmassa: Maybe your pytorch that was compiled from source is not using cudnn? And maybe the compiled version is linking against OpenBLAS instead of MKL?

yikang-li: Hi Massa, Thank you for your help. I’m not very familiar with Linux. If you don’t mind, could you tell me how to check them.

fmassa: 

florin:
Example DCGAN: when updateing netD, noise can be volatile

chenyuntc: <SCODE>noise.data.resize_(batch_size, nz, 1, 1)
noise.data.normal_(0, 1)
fake = netG(noise)
<ECODE> I think it would be better if we change it to: <SCODE>noise = Variable(torch.Tensor(batch_size, nz, 1, 1).normal_(0, 1),volatile = True)
fake = Variable(netG(noise).data)
<ECODE> because when training netD, we won’t need the buffer and gradient of netG. this may acclerate training step and reduce memory usage.[details=Summary]This text will be hidden[/details]

smth: You are correct, changing it like this is better. Even better, you can do: <SCODE>fake = netG(noise)
fake.detach()
<ECODE> If you are interested, please send a pull request to fix it, I will merge.

Veril: detach according to the docs: What does it mean for the variable to be detached and how does it affect performance?

smth: 

Veril: Right, but it will also copy the volatile attribute of its predecessor, which means that you must either recast the variable anyway or not use volatile for the generator. So does detach serve a purpose here? I assume volatile in the generator is good, or why else have that flag option at all?

apaszke: Because you don’t need to use volatile if you’re going to use detach later. It will be nearly equivalent, with volatile possibly being a bit faster and more memory efficient.

chenyuntc: <SCODE>-------------------------------------------------------------------
RuntimeError                      Traceback (most recent call last)
<ipython-input-25-5b95d5373ece> in <module>()
     19         fake_pic = netg(noise_).detach()
     20         output2 = netd(fake_pic)
---> 21         output2.backward(mone) #change for wgan
     22         D_x2 = output2.data.mean()
     23         optimizerD.step()

/usr/local/lib/python2.7/dist-packages/torch/autograd/variable.pyc in backward(self, gradient, retain_variables)
    149         """
    150         if self.volatile:
--> 151             raise RuntimeError('calling backward on a volatile variable')
    152         if gradient is None and self.requires_grad:
    153             if self.data.numel() != 1:

RuntimeError: calling backward on a volatile variable
<ECODE> <SCODE>> /usr/local/lib/python2.7/dist-packages/torch/autograd/variable.py(151)backward()
    149         """
    150         if self.volatile:
--> 151             raise RuntimeError('calling backward on a volatile variable')
    152         if gradient is None and self.requires_grad:
    153             if self.data.numel() != 1:

ipdb> u
> <ipython-input-27-5b95d5373ece>(21)<module>()
     19         fake_pic = netg(noise_).detach()
     20         output2 = netd(fake_pic)
---> 21         output2.backward(mone) #for wgan
     22         D_x2 = output2.data.mean()
     23         optimizerD.step()

ipdb> output2.volatile
True
ipdb> fake_pic.volatile
True
<ECODE>

chenyuntc: 

apaszke: Yes, volatile will be propagated even if you use detach(). I meant that you could remove the volatile flag and use detach with Variables that don’t require grad.