foldl/torch-forum huggingface dataset:
About the Site Feedback category
Category: Site Feedback

system: Discussion about this site, its organization, how it works, and how we can improve it.

shaun: Is it possible to have more categories other than those specified? Similar to tags on stackoverflow where people can create their own tags that then appear as suggestions.

Royi: For instant, a question of the first kind would be something like: “How could one change the learning rate of of the optimizer as a function of epoch number?” A question of the second kind would be something like: “How would you chose architecture for a network for CIFAR10 which is small, namely less than 100,000 parameters?”
About the vision category
Category: vision

smth: Topics related to either pytorch/vision or vision research related topics

bunnisiva: I created my model from scratch . I tried with batch_size 32 , it threw me this error. I tried with batch_size 1 , I got 320% accuracy (something wrong !). The error went away. so there is some dimensional mismatch  some where, I am lost! thanks a million !!! for your help and advise

JoBot_CoBot: can we see your Model ?
About the reinforcement-learning category
Category: reinforcement-learning

smth: A section to discuss RL implementations, research, problems

lakehanne: 

elynn: I had the same problem, it only reached at 199 at my env, then back and force…

howard50b: I got it work the first few times I ran it, but later without any changes the same situation happened as you mentioned. Very weird. Thought it’s using the same random seed thought out.

dpernes: Dear all, One interesting thing that I am observing is that, after a few training iterations (one match is enough), my Q-network starts outputting zeros regardless of the input state! Initially, I thought there was a bug in my code, but now I think that it somehow makes sense. In Pong, the obtained reward is almost always zero (except in the frames where we score or concede a goal) and the Bellman equation is: Q(s,a) = reward + GAMMA * max_a’ (Q(s’,a’)) so, every time we get a zero reward, the Bellman equation is easily satisfied if Q(s,a) = max_a’ (Q(s’,a’)) = 0. That’s why I think my Q-network is basically learning to output zeros regardless of the input… Any hints on how I can overcome this issue? I am following the exact same methodology as in DeepMind’s paper, including the network architecture and the preprocessing.

mathematics: 

cruzas: Hi everyone, If any of you could help to clarify this uncertainty, it would be much appreciated. Thank you.
Request: Tutorial for deploying on cloud based virtual machine

rcmckee: I don’t have a GPU computer. Is there a tutorial or best practices for using PyTorch on a cloud-based virtual machine with GPU capabilities?

smth: PyTorch does not NEED GPUs to function. It works great on CPUs as well. That said, if you want to use a cloud based VM with GPUs, checkout Amazon EC2, Nimbix or Azure which all provide decent GPU instances.

colesbury: I just installed PyTorch on AWS g2.2xlarge machine with the default Ubuntu AWS image (ami-e13739f6). Here’s what I did: <SCODE>sudo apt-get update
sudo apt-get install -y gcc make python3-pip linux-image-extra-`uname -r` linux-headers-`uname -r` linux-image-`uname -r`
wget http://us.download.nvidia.com/XFree86/Linux-x86_64/375.39/NVIDIA-Linux-x86_64-375.39.run
chmod 755 NVIDIA-Linux-x86_64-375.39.run
sudo ./NVIDIA-Linux-x86_64-375.39.run -a
sudo nvidia-smi -pm 1  # enable persistence mode for faster CUDA start-up
<ECODE> And then install NumPy and PyTorch <SCODE>pip3 install numpy ipython
pip3 install https://download.pytorch.org/whl/cu75/torch-0.1.10.post2-cp35-cp35m-linux_x86_64.whl 
pip3 install torchvision
<ECODE> Now PyTorch works with CUDA <SCODE>ipython3
>>> import torch
>>> torch.randn(5, 5).cuda()
 0.8154  0.9884 -0.7032  0.8225  0.5738
-1.0872  1.0991  0.5105 -1.2160  0.3384
-0.0405  0.2946  0.3753 -1.9461  0.0952
 1.6247 -0.8727 -0.6441 -0.8109  1.7622
 1.2141  1.3939 -1.2827 -0.3837 -0.0731
[torch.cuda.FloatTensor of size 5x5 (GPU 0)]
<ECODE>

rcmckee: Thank you! I especially like your warning about the error message! Very helpful!

mjdietzx: 

QianXuX: Thanks, it is great help!
Roadmap for torch and pytorch

trypag: why did you choose to create this interface for python now, it did not seem a priority for the community (however I understand this will probably increase the adoption rate) ? as a torch user, I invested some time to learn how lua and torch operate, I guess it will operate the same way in the near future, can you give us some insights about this ? again as a torch user, would you see some big pros to move from torch to pytorch, without sacrifying performance for example ? maybe sharing your vision of how pytorch will be used at Facebook or Twitter ?

smth: hi pierre, Continue to develop things in whichever frontend you want (Torch or PyTorch) and you can be assured that the Lua side of  things will be maintained. we have done some thorough benchmarks and found this to be not the case, especially for the Torch ecosystem where we cross the C boundaries all the time (LuaJIT traces stop everytime you cross the C boundary). recurrent nets, weight sharing and memory usage will be big positives with PyTorch compared to Torch, while retaining the flexibility of interfacing with C and the current speed of Torch.

farrajota: Hi all, Also, I’ve tried to find a roadmap for the future (2017+) of any of these implementations on top of the torch computing framework and got not much. I do see some of the ideas proposed in the previous roadmap for torch coming to life (like tutorials, standardized datasets, a nice forum to hang out), so I must point out to the big elephant in the room: which one will take the focus on now? Pytorch or Torch7? Don’t get me wrong, I much prefer python’s libraries because they are more standardized and mature than Lua’s, and Lua lacks many of the key functionality for some of these tasks (scipy, matplotlib, etc.). But this wasn’t that big of a deal due to some libraries like “fb.python” that allowed the use of some functionalities from python to be used with Lua, but I realise that python is a better choice for a research/developing platform compared to Lua. Another thing I would like to clarify from the devs is this: what is the big advantage of pytorch regarding tensorflow? Or shall I say, what makes pytorch stand out of the rest of the “competition”?

rouniuyizu: Hi, could you further elaborate on “memory usage will be big positives with PyTorch compared to Torch”, any benchmark/example scenario?

ruotianluo: Hi, You say retaining the current speed of Torch. From my takeaway of minpy, the dynamic definition of the graph in python would hurt speed, is that also true in pytorch. And when would you do start working on benchmarks.

apaszke: There is a slight impact on the perf (within 5% in most cases), but we know of a couple things that can speed things up further. Remember it’s still beta.

apaszke: You can try running the examples. We’re usually seeing 30-50% memory usage improvements. Benchmarks coming soon.

trypag: 

apaszke: That’s an old README, it has changed a bit since then.
CFFI for interfacing C functions

robert.zaremba: There are many benefits of CFFI, just to mention few: Abstract the Python version (CPython2, CPython3, PyPy). Better control over when and why the C compilation occurs, and more standard ways to write setuptools-based setup.py 3 files. Keep all the Python-related logic in Python so that you don’t need to write much C code.

apaszke: Are you asking about our library (TH, THC, etc.) wrappers or about the extensions?

robert.zaremba: 

smth: 

jdily:
Import/Export models

edgarriba: Meanwhile, do you have already any internal data structure to read the models definition?

apaszke: We’re open to discussion about formats for sharing parameters and model definitions between libraries. If there’s going to be a larger initiative with some of the frameworks adopting a new format, we’ll likely join and try to help. Right now our serialization is very torch specific and we didn’t design it having in mind interoperability with other software. If you need to export model weights I encourage using state_dict and some more common formats like HDF5 or protobuf.

Eugenio_Culurciello: is there a suggested new format to export in pytorch for use in other framework and custom compilers?
Is new serialization format documented?

jsenellart-systran: 

apaszke: I can write up the specs, but they’re much more complicated now. Serialized files are actually tars containing 4 other files - one listing tensors, one listing storages, one with storage data and one with system info (long size, endianness, etc.). I think for this use case it would by much simpler to take advantage of some more standardized format, with a C library supporting loading from it (such as HDF5 or protobuf). Do you think that would be ok?

edgarriba: 

jsenellart-systran: Thanks for your answer. yes I think this would be great - but cannot you use torch native serialization since you have them at hand in the TH* libraries - the 4 containers you are talking about could be simply put in a lua table - and this would avoid dependencies with other libraries? I understand that the pytorch objects (variables) would not be compatible with lua modules objects but at least would be readable.

apaszke: No we can’t. These are not continers, these four are binary blobs. In Lua torch serialization has been written from scratch, in PyTorch we depend on pickle to save the objects, and while it allows us to have very robust serialization that conforms to python standards, it gives us less control, and is unreadable by Lua. I don’t think it’s worth changing that. It’s a complex matter, and conforming even to some of the old Lua standards would hurt usability for all Python users. HDF5 is quite widespread and will provide you with a way to save weights in Python and load them in Lua.

jsenellart-systran: 

apaszke: 

apaszke: I think in the future we’ll also have to figure out some kind of format that allows for exporting graphs. This will be needed for Caffe2 and TF model exports.

jsenellart-systran: Let me rephrase that to be sure to correctly understand :).

apaszke: Yes, because in Lua torch nngraph objects are the same all the time. PyTorch builds the graph dynamically, so every iteration uses fresh Python objects to represent the graph, and the recipe for the graph construction is the model’s code.
2017 and we still can’t support AMD hardware, why?

yaxattax: Not all of us own NV hardware. Some of us want to use these tools at home without having to make a new purchase or purchase some AWS time. Not all of us are research scientists with fat grants allowing us to buy the latest NV hardware. When will the vendor lock-in stop? Surely “free” tools should be open to all.

ClementPinard: Lots of work need to be done to reach Nvidia’s performance. Cudnn is a huge step for performance that AMD does not have yet. There was OpenCL support for torch, but pragmatically everyone just turned to CUDA devices. If someone is willing to make a cudnn equivalent maybe it would change.

smth: Please see this thread for more details. We are waiting for HIP to be released from AMD.

Felix_Lessange: 

gvskalyan: 

Madegomez: Did you have a good experience using ROCm?
Package manager choice: conda vs. pip

Atcold: 

smth: This is because conda manages dependencies that are not just python dependencies. For example with conda, we get tighter control over which BLAS is installed.

phenixcx: How can I deal with the problem? Thank you beforehand.

apaszke: It looks as if your script ran out of memory.

phenixcx: I guessed the problem is caused by the imcompatibility of OpenBLAS with Pytorch, because the problem is totally solved by reinstalling Pytorch with the command ‘conda install pytorch torchvision -c soumith’. It is heard that Pytorch compiled from the sources of github cannot well control OpenBLAS.

apaszke: I haven’t heard about problems with OpenBLAS before, but I don’t think it’s really a PyTorch bug, since we’re only calling its functions. If it can’t manage its threads properly, there’s nothing we can do about it. I’d recommend using MKL.

phenixcx:
How to do backward() for a net with multiple outputs?

pengsun: e.g., in Torch 7 I have a net with the last module an nn.ConcatTable, then I make the gradOutputs a table of tensors and do the net:backward(inputs, gradOutputs) How to do similar things with pytorch? I tried to backward() for each output, but it complained that backward should not be called multiple times?

colesbury: It takes a list of variables and a list of gradients (one for each variable).

pengsun: 

pengsun: 

apaszke: Thanks for the report, we’ll fix that!
Input preparation

gabrieldlm: Hello all, For instance, the fallowing code results in error: <SCODE>(class model above)
    self.affine1(10, 100)
(...)

x = np.random.random(10)
ipt = torch.from_numpy(x)
probs = model(Variable(ipt))
<ECODE> Then <SCODE>TypeError: addmm_ received an invalid combination of arguments - got (int, int, torch.DoubleTensor, torch.FloatTensor), but expected one of:
 * (torch.DoubleTensor mat1, torch.DoubleTensor mat2)
(...)
<ECODE> How is the proper data preparation in pytorch? Thanks in advance and looking forward to use pytorch (the performance of the cart pole in openai gym was better than with other frameworks). Obs: As I didn’t saw any other mechanic-ish question topic, nor a question flag, hope that is not off topic for the forum.

smth: hi there. In this particular case, thanks for your feedback. We will improve our onboarding process, by making more newbie friendly tutorials. <SCODE>ipt = torch.from_numpy(x)
ipt = ipt.float()
<ECODE>

Atcold: Please, let me know of anything you think it should be included, so that I can better plan the structure of my lessons.

gabrieldlm: Thanks for the reply, but after changing the tensor type to float I get the fallowing error: <SCODE>RuntimeError: matrices expected, got 1D, 2D tensors at /Users/soumith/anaconda/conda-bld/pytorch0.1.6_1484755992574/work/torch/lib/TH/generic/THTensorMath.c:857
<ECODE> <SCODE>print ipt
(...)
[torch.FloatTensor of size 10]<ECODE>

smth: 

Atcold: 

gabrieldlm: 

Atcold: 

henrye:
Help core dumped problem!

Chun_Li: 

smth: Thank you. It seems to be specific to AMD Athlon CPU. I will take a look.

Chun_Li: Thanks for the response. BTW, I forgot mention to OS of my computer, which is Linux Ubuntu 16.10.

Chun_Li: 

smth: what is the result of the command: <SCODE>which gcc<ECODE>

Chun_Li: This gcc comes from anaconda2, not the gcc comes from ubuntu 16.10.

Chun_Li: 

Chun_Li: Summary of the solution on my last post. I think this case can be closed if the above solution is verified by the pytorch team.

apaszke: Great to hear that it works for you now! We’re aware that building from source will solve the problem, yet it would be good to have binary packages that work on any CPU architecture. We’ll keep the issue open util it’s fixed.

Vijay_Dubey: 

smth: 

buildabeast: I built from source and problem is gone

buildabeast: processor	: 1
How to transform Variable into numpy?

cumttang: When I used pytorch, I met a problem that I can not transform Variable into numpy. When I try to use torch.Tensor or transform it into torch.FloatTensor, but it’s fail. So how can I to solve this problem?

edgarriba: 

apaszke: 

cumttang: 

Morpheus_Hsieh: 

Morpheus_Hsieh: 

Robi: Yes, it’s working: (Variable(x).data).cpu().numpy()

blitu12345: 

Royi: Is there a function which handles both CUDA and CPU at once?

Nitin286roxs: what will type if cuda tenso is fp16 than after converting it will numpy(fp32) or numpy(fp16)?
Load a saved model

Eugenio_Culurciello: and how do you forward with this model file?

Atcold: <SCODE>x = torch.rand(1, 3, 224, 224)
xVar = torch.autograd.Variable(x)
res18(xVar)

--> 

Variable containing:
-0.4374 -0.3994 -0.5249  ...  -0.5333  1.4113  0.9452
[torch.FloatTensor of size 1x1000]
<ECODE> Let me know if it works.

phenixcx:
Copying nn.Modules without shared memory

lenny: 

smth: no, what you need to do is to send them model to the new process, and then do: <SCODE>import copy
model = copy.deepcopy(model)
<ECODE> share_memory() only shares the memory ahead of time (in case you want to reuse the shared memory for example)

apaszke: 

Juna: Then could you give me an example of using load_state_dict() to copy a module?

JosueCom: He is referring to saving the state_dict (for instance into a file), sending it over to another process, then loading it with load_state_dict
Why cant I change the padding, and strides in Conv2d?

Kalamaya: Hi, perhaps I am missing something, but I cannot seem to figure out how to change the padding and stride amounts in the Conv2d method. Thanks

apaszke: You can pass them as arguments to the Module constructor e.g. <SCODE>nn.Conv2d(16, 32, kernel_size=3, padding=(5, 3))
<ECODE> Alternatively, if you need to change them at runtime, I’d suggest using the functional interface: <SCODE>import torch.nn.functonal as F

...

F.conv2d(input, self.weight, self.bias, kernel_size=3, padding=(x, y))
<ECODE>

Kalamaya: The change at run-time is intriguing … I didn’t even know people did that. How could you guarantee the same dimensionality if you changed padding amounts during run time though?

apaszke: I don’t know if people do that, but it might be reasonable if you have variably sized inputs, or are doing something completely new.

Kalamaya: Very interesting - I mean I know that in the context of RNNs we can have variable length inputs etc, since the back-prop can be unrolled N times, but I didnt realize that we could have variable length weights/parameters etc… I’ll read more about it, (and any sources on this are appreciated), but the top question in my mind is how do you guarantee that in a dynamic graph, when we change the dimensionality of weights, that those are new weights are even trained? Does that make sense?

apaszke: For example conv layers can be applied to variably sized inputs, with a fixed set of weights.

Kalamaya: Thank you will take a look!
Broadcasting? Or alternative solutions

rasbt: Hi, I really like the library so far. However, I was wondering if broadcasting is on the roadmap, and what would be your current suggestion for work-arounds? <SCODE>x = Variable(torch.Tensor([[1.0, 1.0], 
                           [2.0, 2.0], 
                           [3.0, 3.0], 
                           [4.0, 4.0], 
                           [5.0, 5.0], 
                           [6.0, 6.0]]))

weights = Variable(torch.zeros(2, 1))
bias = Variable(torch.zeros(1))

net_input = x.mm(weights) + bias
<ECODE> A workaround would be to add 1s to the input tensor, I guess: <SCODE>x = Variable(torch.Tensor([[1.0, 1.0, 1.0], 
                           [1.0, 2.0, 2.0], 
                           [1.0, 3.0, 3.0], 
                           [1.0, 4.0, 4.0], 
                           [1.0, 5.0, 5.0], 
                           [1.0, 6.0, 6.0]]))

weights = Variable(torch.zeros(3, 1))

net_input = x.mm(weights)
<ECODE> What would be your thoughts on that?

apaszke: Adding broadcasting to most operations is definitely on our roadmap, and will be hopefully ready quite soon. Since it’s so often asked for we’ll probably reprioritize that and have it implemented soonish. For now there are two solutions - one that works already, another one that will be implemented very soon (presented in the same order): You can do broadcasting by manually adding singleton dimensions and expanding along them. This doesn’t do any memory copy, and only does some stride tricks: <SCODE>net_input = x.mm(weights)
net_input += bias.unsqueeze(0).expand_as(net_input)
<ECODE> <SCODE>import torch.nn.functional as F
output = F.linear(input, weights, bias) # bias is optional
<ECODE>

rasbt: 

ecolss: 

apaszke: <SCODE>x = torch.randn(1)
y = torch.randn(2, 3, 4, 5, 6)
print(y + x.expand_as(y))
<ECODE>

jphoward: 

apaszke: 

jphoward: Let me know if you need more info - I’m not sure I’ve done a great job of explaining!

apaszke:
Concatenate two tensor for each timestep

NgPDat: <SCODE>def forward(...):
        ...
        z = Variable(torch.zeros(y.size(0), y.size(1), 1024))
        for i in range(z.size(0)):
            z[i] = torch.cat((y[i], x), 1)

        z = self.decoderGRU(z, init_decoder_GRU)
        decoded = self.classLinear(z.view(z.size(0)*z.size(1), z.size(2))) #error here.
        ...
<ECODE> Error message: <SCODE>AttributeError: 'tuple' object has no attribute 'view'
<ECODE> What is the best way to do this?

apaszke: If so, this should do the trick for you and will be much more efficient: <SCODE># Replace the for loop with this line. It will expand x to be of size seqLen x batch x feature1
# and concatenate it with y in one go. 
z = torch.cat([y, x.unsqueeze(0).expand(seqLen, *x.size())], 2)
<ECODE> <SCODE>z = self.decoderGRU(z, init_decoder_GRU)[0]
<ECODE>

NgPDat: Thank you. Both solve my problem. Never thought of using expand that way.

chifish: Hi, I wonder if it’s convenient for you to share your encoder-decoder code with me, as I am trying to implement a similar network?

amy: Hi,
Model zoo categories

Eugenio_Culurciello: 

smth: i’m not sure i understand your question

Eugenio_Culurciello: the list of category names that correspond to each of the output neurons of CNN. Even for ImageNet, they need to specified.

rasbt: 

fmassa: 

Eugenio_Culurciello: 

fmassa: 

Eugenio_Culurciello:
Are tables like concattable or paralleltable present in torch

gsp-27: Are there utilities like concattable, paralleltable in pytorch??

apaszke: No, but you can easily achieve that using autograd. e.g. for concat table: <SCODE>class MyModel(nn.Module):
    def __init__(self):
        self.submodule1 = MySubmodule1()
        self.submodule2 = MySubmodule2()

    def forward(self, x):
        # this is equivalent to ConcatTable containing both submodules,
        # but is much more readable
        concat = [self.submodule1(x), self.submodule2(x)]
        # do some other processing on concat...
<ECODE>

thongnguyen: I tried the following <SCODE>class MyModel(nn.Module):
    def __init__(self, submod_num,submod_var):
        self.submod_pool = []
        for i in range(submod_num):
            self.submod_pool += [MySubmodule(submod_var[i])]

    def forward(self,X):
        outPool = []
        for submod in self.submod_pool:
            outPool += [submod(X)]

        return torch.cat(outPool,dim)
<ECODE>

Amir_Ghodrati: 

thongnguyen: I solved it by this: <SCODE> model= nn.Sequential(*layer_list)
<ECODE>
Non-legacy View module?

lebedov: There doesn’t currently appear to be any non-legacy View module in pytorch’s torch.nn module. Any reason for this? While obviously not essential, it’s convenient when porting over existing Torch networks. Would be happy to submit a PR.

gsp-27: 

apaszke: 

lebedov: That method directly resizes the associated tensor instance; having a View module would enable one to directly add it to the sequence of modules in a network rather than having to explicitly call it. For example: <SCODE>class MyNetwork(torch.nn.Module):
    def __init__(self):
        super(MyNetwork, self).__init__()
        modules = [nn.Conv2d(1, 10, 3),
                   nn.MaxPool2d(2, 2),
                   ...,
                   View(100),
                   nn.Linear(100, 2)]
        for i, m in enumerate(modules):
            self.add_module(str(i), m)
    def forward(self, x):
        for child in self.children():
            x = child(x)
        return x<ECODE>

apaszke: 

lebedov: 

apaszke: 

lebedov: Duly noted. Thanks for the feedback.

juliohm: Sorry to come back to this discussion after a month or so, but I indeed miss some convenient way of flattening out a layer inside of Sequential. I am breaking this operation into two pieces as suggested, a Sequential model that outputs a image shape, and then in the forward() method, I flatten the results.

ptrblck: <SCODE>class Flatten(nn.Module):
    def __init__(self):
        super(Flatten, self).__init__()

    def forward(self, x):
        x = x.view(x.size(0), -1)
        return x

model = nn.Sequential(
            nn.Conv2d(3, 1, 3, 1, 1),
            Flatten(),
            nn.Linear(24*24, 1)
        )

x = Variable(torch.randn(10, 3, 24, 24))
model(x)
<ECODE>

juliohm: