foldl/torch-forum huggingface dataset:
Sampled softmax loss

beegii: Hi, Does sampled softmax loss exist in pytorch? I cound not find it. Thanks

smth: 

beegii: 

apaszke: We don’t have any such thing in the core. We’ll need to add them. Thanks for the pointer!

ngimel: 

beegii: 

lopuhin: EDIT: sorry, I see that original link is to a page with a number of different softmax approximations, and NCE is one of them. I personally would be more interested in sampled softmax, as it tends to work better for me.

vince62s: You may also be interested in this implementation: Giving very good results for the LM task.

beegii: 

windweller: Any updates on this?? It probably isn’t a priority…but I secretly wish PyTorch has a big development team like TensorFlow and can add these functionalities easily!

smth: 

windweller:
Updating PyTorch

Atcold: I just wanted to pin this topic, so that it can be used for future reference. <SCODE>conda config --add channels soumith
<ECODE> <SCODE>conda update pytorch torchvision
<ECODE> <SCODE>$HOME/anaconda3/lib/python3.5/site-packages/torch
<ECODE> <SCODE>$HOME/anaconda3/lib/python3.5/site-packages/torchvision-0.1.6-py3.5.egg/torchvision/__init__.py
<ECODE> but such location does not exist. Nevertheless, my IDE can find those files in that location, when exploring the source code. <SCODE>/data/users/soumith/miniconda2/conda-bld/pytorch-0.1.7_1485445763020/work/torch/lib/THNN/generic/SpatialConvolutionMM.c
<ECODE>

apaszke: 

Atcold: About 2, that’s kind of not what I see here. <SCODE>>>> f = open(torchvision.__file__)
Traceback (most recent call last):
  File "/home/atcold/anaconda3/lib/python3.5/site-packages/IPython/core/interactiveshell.py", line 2881, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-95-16f43d841fa3>", line 1, in <module>
    f = open(torchvision.__file__)
NotADirectoryError: [Errno 20] Not a directory: '/home/atcold/anaconda3/lib/python3.5/site-packages/torchvision-0.1.6-py3.5.egg/torchvision/__init__.py'
>>> f2 = open(torch.__file__)
# succeed
<ECODE>

apaszke: 

Atcold: Oh, maybe I’ve figured it out… <SCODE>atcold@GPU0 ~/anaconda3/lib/python3.5/site-packages $ ls -daF torch*
torch/  torch-0.1-py3.5.egg-info/  torchvision-0.1.6-py3.5.egg  torchvision.pth
<ECODE>

apaszke: Oh yeah, I think it’s zipped in an egg.

Hubert: this is no longer working. PackageNotFoundError: Package not found: ‘pytorch’

antspy: Hi,

jdhao: <SCODE>conda install pytorch torchvision -c pytorch
<ECODE>

omair-kg: For me this did not work. It always tried to downgrade my pytorch and torchvision to some rather ancient version. I installed it again in a separate environment using pip then which worked.

jdhao: 

omair-kg: Tried that aswell. So far only Pip seems to work for me.

manozzm: This worked for me! The following NEW packages will be INSTALLED: <SCODE>cudatoolkit: 8.0-3
<ECODE> The following packages will be UPDATED: <SCODE>pytorch:     0.2.0-py36h53baedd_4cu80 soumith [cuda80] --> 0.3.0-py36_cuda8.0.61_cudnn7.0.3h37a80b5_4 pytorch
torchvision: 0.1.9-py36h7584368_1     soumith --> 0.2.0-py36h17b6947_1                       pytorch<ECODE>

shehabk: If the dependencies are not met, conda will not update to recent pytorch. What I did was the following.

Atcold: <SCODE>conda config --add channels pytorch
<ECODE>

Atcold: 

rasbt: <SCODE>>>> import torch
>>> import os
>>> os.path.isfile(torch.__file__)
True
>>> os.path.abspath(torch.__file__)
'/Users/sebastian/miniconda3/lib/python3.6/site-packages/torch/__init__.py'
<ECODE>

stoddur: Hey guys. What is the recommended way of upgrading to PyTorch V1.0 using conda? After adding the pytorch channel and trying to update this message appears: # All requested packages already installed. Any suggestions?

ptrblck: 

stoddur: Thanks for your reply. How does one go about installing the 1.0 preview? After installing the nightly version from source I end up with version 0.4.1
How to get the df_do as we used in torch7

dafang_He: <SCODE>     optimizer.zero_grad()
     output = model(data)
     loss = criterion(output, target)
     loss.backward()
     optimizer.step()
<ECODE> So all the network is back-propagated when I call loss.backward() ?

fmassa: 

dafang_He: Thanks!

dafang_He: Thanks!

apaszke: You need to make the hook a closure - just redefine it at every step, and use the data in its body: <SCODE>optimizer.zero_grad()

output = model(data)
def hook(d_output):
    return d_output * data.mean() # you can use the data here
output.register_hook(hook)

loss = criterion(output, target)
loss.backward()
optimizer.step()
<ECODE>
Model.zero_grad only fill the grad of parameters to 0

ypxie: 

smth: 

apaszke: 

ypxie: Thanks for the explanations, but if I don’t fill other variable.grad.data to 0, will the grad of parameters that depended on those variables be wrongly estimated?  Keeping tracking of all the variables and setting their grad to 0 properly seems quite error prone.

apaszke: No, why would it matter? Gradient of parameters is not a function of the gradient w.r.t. some other Variable.

ypxie: Do you mean whatever the module is, as long as it’s inherited from base nn.Module class, simply calling the default model.zero_grad() is enough? I mean the error can pass from other Variable to the parameter,  if some Variable’s grad is unintentionally accumulated from the last run, the gradient of parameters may also be wrong.

apaszke: 

ypxie: <SCODE>    def __init__(self):
        self.out = torch.autograd.Variable(torch.zeros(timestep, batchsize, 
                                                       self.W_decode.size()[1]) , requires_grad=True)

    def forward(self, input, state=None):
        # input is timesetep*N
        batchsize, timestep = input.size()[1], input.size()[0]
        vec_input = input.view(-1)
        emb = torch.index_select(self.W_emb, 0, vec_input).view(timestep,batchsize, -1) # emb = N*ninp
        inp = matmul(emb, self.W_rnn)
        state = torch.autograd.Variable( torch.zeros(inp.size()[1:])) if state is None else state
        for step in range(inp.size()[0]):
            this_input = inp[step]  # N * nhid
            this_input = torch.addmm(this_input, state, self.U_rnn)
            state = F.tanh(this_input  + self.b_rnn.expand_as(this_input) )
            self.out[step] = torch.addmm(self.out[step], state, self.W_decode)
            self.out[step] = F.softmax(self.out[step]  + self.b_decode.expand_as(out[step]) )
        return self.out
<ECODE>

apaszke: <SCODE>def __init__(self):
    self.out = get_new_out()

def forward(self, input):
    ...
    next_out = get_new_out()
    for step in range(inp.size(0)):
        next_out[step] = ... # an expression containing self.out
    self.out = next_out
    return next_out
<ECODE>

ypxie: 

apaszke: No, you’re not going to feel any difference in speed. CPU allocations are fast and we have a custom CUDA allocator that caches the memory, so it’s also very fast. I don’t know how you define the safety of Module.zero_grad - it does what it’s meant to do, i.e. zero the grad of parameters.

apaszke: Also, as I said, if you don’t reallocate the output your graphs will never be freed and that will blow up the memory. Don’t cache intermediate things too long, PyTorch philosophy is quite different from Lua torch.

ypxie: <SCODE>from torch.autograd import Variable
x = Variable(torch.ones(2, 2), requires_grad = True)
y = x  + 2 
z = y * y * 3
out = z.mean()
out.backward(retain_variables=True)
print(x.grad)
<ECODE> I get <SCODE>Variable containing:
 4.5000  4.5000
 4.5000  4.5000
[torch.FloatTensor of size 2x2]
<ECODE> <SCODE>Variable containing:
 0  0
 0  0
[torch.FloatTensor of size 2x2]
<ECODE>

fmassa: 

ypxie: 

smth: 

ypxie: Many thanks, very helpful!

apaszke:
Unrolling adversarial networks

Jordan_Campbell: Hi, Is the point that we simulate optimisation of the discriminator for K steps, then use the final error at D to optimise the parameters of D from before we started unrolling? Does this mean we have to store a copy of D’s parameters before we start the unrolling procedure? Does this mean that we end up discarding all the optimisation changes that occurred during unrolling? (Excluding the fact that they are implicitly bundled into the final output from the unrolling). Also, do we unroll for both real and fake data through D, or just fake data? Any help with this would be very much appreciated, cheers!

apaszke: I don’t really know how unrolled GANs work, but as far as I remember they require taking gradient of functions of another gradient, and we don’t support that yet. It’s on our roadmap and we’re actively working on that, but it might take us some time.

Jordan_Campbell: 

prolearner: this is supported now right? How can it be done?

tom: Best regards Thomas

lewiskit: Does it support recurrent modules using Pytorch 1.0 now ?
Model parameter changes on every load

Vijay_Rengarajan: Hi, I saved a model of a simple convnet created without using any loops in the class using torch.save(). When I load the same saved model using torch.load() and print parameters param.data iterating through model.parameters(), it prints different values each time while running the code. I used  torch.cuda.manual_seed(1234) while training the model. But I guess it should not affect loading the saved model. It might be a silly mistake on my part, but I could not figure it out.

Vijay_Rengarajan: Edit: There was a mistake in my code that I overwrote the loading with a model = ConvNet() definition. Please close this thread.
Adaptive learning rate

davidenitti: How do I change the learning rate of an optimizer during the training phase? thanks

trypag: 

apaszke: If you want to change the LR we recommend reconstructing the optimizer with new parameters.

trypag: Oh, ok sorry ! things move fast <SCODE>def adjust_learning_rate(optimizer, epoch):
    """Sets the learning rate to the initial LR decayed by 10 every 30 epochs"""
    lr = args.lr * (0.1 ** (epoch // 30))
    for param_group in optimizer.param_groups:
        param_group['lr'] = lr

<ECODE>

ecolss: 

apaszke: 

ruotianluo: 

lysuhin: 

mderakhshani: 

FuriouslyCurious: Keras provides two functions which are fairly straightforward to implement, and everyone loves them: This one reduces LR when gradient is stuck on a plateau for past “X=patience” epochs: This one stops you from burning up your Amazon AWS $$$ credits if your model is not learning anything after “X=patience” epochs: Would PyTorch be open to adding something like this?

ecolss: Any update on this thread about learning rate decay?

will: In the models I’m training right now I see an increase in the loss when I construct a new optimizer to decay the learning rate.

Rongzhao_Zhan: 

chsasank: Hi, I hope this clears things up.

apaszke: 

will: 

lliu25: How about more general situations where it is desired to adaptively adjust any parameters defined in the model? Is it practical to reconstruct optim in every loop? Maybe a mechanism like placeholder? Thanks!

smth: 

lliu25: 

ncullen93:
Return activations for all the layers

lcelona: Hello, Given the following code: <SCODE>def forward(self, x):
    x = x.transpose(2,1)
    x = self.max_pool(x)
    x = x.view(x.size(0), -1)
    x = self.classifier(x)
<ECODE> Does exist an elegant way to get back the results of each layer? Or I should use a dictionary containing a key for each layer result? Thank you

fmassa:
Sample multivariate normal with per-example standard deviation

solidor: Hi, I want to do something similar to this: <SCODE>mu = torch.zeros(5, 2)
sd = torch.ones(5)
torch.normal(mu, sd)
<ECODE> I noticed in the 1d case it works: <SCODE>mu = torch.zeros(5, 1)
sd = torch.ones(5)
torch.normal(mu, sd)
<ECODE>

solidor: 

apaszke: 

solidor: <SCODE>mu = Variable(torch.zeros(5, 2))
sd = Variable(torch.rand(5, 1))
torch.normal(mu, sd.expand_as(mu))
<ECODE>

apaszke: Yes, expand will be much better! There will be no memory copy in this case.
Indexing a Variable with a mask generated from another Variable

yusuf_isik: x[y[:,0] > 0] But I assume there should be a much easier way. Thanks.

apaszke: Yeah, we’ve added the ability to select based on the long tensor a few days ago. I think it’s in the new binaries, so once you reinstall pytorch this should work: <SCODE>x[(y[:, 0] > 0).nonzero().squeeze()]
<ECODE>

apaszke: And in the future, we’re going to support automatic broadcasting so the numpy way should work in some time too.

yusuf_isik: That’s very good news. Thanks.

yusuf_isik: I just reinstalled from scratch , but it cannot find nonzero() function. I am getting the following message: I also have a version that I built from source, and I get the same error from it, too. How can I get the right version? Thanks.

apaszke: 

acgtyrant: Four months passed, we still can not use nonzero() for Variables…

apaszke: You can always open an issue in the main repo
Fast Tensor access in python?

yzhu: In lua torch, we can access a Tensor using luajit-FFI pointer as fast as in C.  Do we have similar thing in pytorch?

smth: no. Python doesn’t have JITting.

yzhu: another maybe more noob question: since we are talking about the access speed here, how’s the overhead of .numpy() operation?

smth: 

apaszke:
PyTorch tutorial for Neural transfert of artistic style

alexis-jacq: Hi, If someones are interested, I’ve realized this PyTorch tutorial to implement the neural transfer of artistic style developed by Leon Gatys and AL: Any feedback is welcome!

apaszke: 

alexis-jacq: Yes, the code works and I give the whole script in the “.py”. I show the wrong code to explain the global idea, then I give the correct version just below (maybe not a good pedagogy…). But I didn’t know that just a list of Variable works, this is much simpler that what I did (I constructed a module with the variable as a parameter). Thanks !

phenixcx: If using torch.optim.SGD instead of Adam, the codes work. However, I guessed Adam would give better performance in some cases, and thus tried to use Adam algorithm. Would you mind give an example of using the Adam optimizer? Thank you Adam O(∩_∩)O

smth: 

apaszke: 

phenixcx: 

ecolss: <SCODE> Traceback (most recent call last):
  File "Neural_Style.py", line 205, in <module>
    style_score += sl.backward()
  File "Neural_Style.py", line 105, in backward
    self.loss.backward(retain_variables=retain_variables)
  File "/usr/local/lib/python3.6/site-packages/torch/autograd/variable.py", line 146, in backward
    self._execution_engine.run_backward((self,), (gradient,), retain_variables)
  File "/usr/local/lib/python3.6/site-packages/torch/nn/_functions/conv.py", line 48, in backward
    if self.needs_input_grad[0] else None)
  File "/usr/local/lib/python3.6/site-packages/torch/nn/_functions/conv.py", line 119, in _grad_input
    return self._thnn('grad_input', input, weight, grad_output)
  File "/usr/local/lib/python3.6/site-packages/torch/nn/_functions/conv.py", line 161, in _thnn
    return impl[fn_name](self, self._bufs[0], input, weight, *args)
  File "/usr/local/lib/python3.6/site-packages/torch/nn/_functions/conv.py", line 251, in call_grad_input
    grad_input, weight, *args)
RuntimeError: Need gradOutput of dimension 4 and gradOutput.size[1] == 64 but got gradOutput to be of shape: [64 x 2401] at /Users/soumith/code/pytorch-builder/wheel/pytorch-src/torch/lib/THNN/generic/SpatialConvolutionMM.c:50<ECODE>

apaszke: Did you update to 0.1.10?

alexis-jacq: 

ecolss: Yes, I did. I thought the problem was that, the input is cloned and resized in GramMatrix module, and the style loss is then computed on it, and as the error occurs at the stage of style loss backward(), so would it be that, the grad of style loss over GramMatrix is a 2 dim tensor, and further the grad over the cloned and resized input is not properly computed?

ecolss: 

ecolss: Debug for a while, found the root cause of the error: Variable.data.resize_() -> Variable.resize().

alexis-jacq: Thanks for having reported this issue.

mehdi-shiba: 

apaszke: Yes, it’s better to use .view

ecolss: 

apaszke: 

ecolss: 

leongatys:
In place randomization

eulerreich: 

smth: you can do: <SCODE>a = torch.zeros(3, 4) # or whatever shape
torch.randn(3, 4, out=a)
<ECODE> But yea we didn’t implement some methods. We just didn’t get around to it.

apaszke: 

jnhwkim: FYI, this isn’t implemented yet: <SCODE>>>> torch.cuda.FloatTensor(3,2).random_()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'torch.cuda.FloatTensor' object has no attribute 'random_'
<ECODE>
How does autograd handle multiple objectives?

Veril: Using the pytorch framework. Suppose you have 4 NN modules of which 2 share weights such that one objective relies on the computation of 3 NN modules (including the 2 that share weights) and the other objective relies on the computation of 2 NN modules of which only 1 belongs to the weight sharing pair, the other module is not used for the first objective. What would the optimisation step in this scenario entail? With efficiency in mind.

smth: In this case, you only have 3 NN modules, and one of them is simply reused. You give it the list of losses and grads. The optimization step is pretty standard, you give the all the modules’ parameters to a single optimizer.

Veril: So just to be clear, specify a single objective that merges (concat) all the sub-objectives and backward() on it? There won’t be any issue regarding going over the same variables twice through different pathways? Thanks.

smth: Yes. No issues.
How to define a new layer with autograd?

big_tree: Hi all,

apaszke: Sure, this will be handled for you. For example: <SCODE>import torch.nn as nn
from torch.autograd import Variable

class Gaussian(nn.Module):
    def __init__(self):
        self.a = nn.Parameter(torch.zeros(1))
        self.b = nn.Parameter(torch.zeros(1))
        self.c = nn.Parameter(torch.zeros(1))

    def forward(self, x):
        # unfortunately we don't have automatic broadcasting yet
        a = self.a.expand_as(x)
        b = self.b.expand_as(x)
        c = self.c.expand_as(x)
        return a * torch.exp((x - b)^2 / c)

module = Gaussian()
x = Variable(torch.randn(20))
out = module(x)
loss = loss_fn(out)
loss.backward()

# Now module.a.grad should be non-zero.
<ECODE>

big_tree: to update all the parameters?

apaszke: 

big_tree: Thank you so much for your help!

jdhao: Now we have defined the Gaussian layer, I want to know how to use it in a network of several layers. For example, <SCODE>class MyNet(nn.Module):
    def __init__(self):
        super(MyNet, self).__init__()
        self.fc1 = nn.Linear(100, 20)
        self.gauss = Gaussian()
    def forward(self, x):
        x = F.relu(self.fc1(x))
        x = self.gauss(x)

net = MyNet()
<ECODE> Is the above code correct? Or do we have to set up other things in order to use the newly-defined layer in other network?

smth: the above is correct.

Soumava_Roy: Can I define my own layer with a forward/backward function or I will have to define the layer as a Function so that I can define the backward function too?

Hao_Chu: 

herleeyandi: 

jdhao: 

herleeyandi: 

jdhao: Most of the time we do not need to extend PyTorch using numpy. As long as you use the builtin method of Variable, you can only write forward method and backward gradient computation is handled by autograd. So using a composition of builtin Variable method to achieve what you want is more time-saving.

colesbury: 

herleeyandi: 

colesbury: Yes, you can do that. Writing new CUDA kernels usually requires a lot of effort. If you can express your layer in terms of existing Tensor operations, then that’s usually a better way to get started. If you can’t do that, then you might have to write new kernels.

MahdiNazemi: What if the forward function is not differentiable? How should I update the parameters and return the gradient w.r.t inputs?

jwang9: I tried the piece of code posted by apaszke, but I got the following error. <SCODE>RuntimeError: bitwise_xor(): functions with out=... arguments don't support automatic differentiation, but one of the arguments requires grad.
<ECODE>

jwang9: I figured that the problem arises from the power calculation in python. I changed ^2 to **2 and it works now.
Reducing LSTM Hidden State Output to 1 Dimension

ritchieng: <SCODE>>>> rnn = nn.LSTM(1, 100, 4)
>>> input = Variable(torch.randn(1, 1, 1))
>>> h0 = Variable(torch.randn(4, 1, 100))
>>> c0 = Variable(torch.randn(4, 1, 100))
>>> output, hn = rnn(input, (h0, c0))
<ECODE> Any ideas?

apaszke: Not sure what 1 in 1 out LSTM is, you want to have 1 of input and output features, while using 100 for hidden size? I think the standard practice is to add a Linear layer on top of the LSTM. If the network learns with hidden states of size 1, but doesn’t with that of size 100, it might mean that the optimization task becomes too complex for a simple task that you’re training the network to perform. Can’t really help you a lot, it depends on a lot of factors.
LSTM Output Output Range

ritchieng: 

apaszke: Can’t you just multiply the output by 10?

Andre_Holzner: <SCODE>from sklearn.preprocessing import MinMaxScaler

# build a transformation to have the transformation result in the range 0..1
scaler = MinMaxScaler(feature_range=(0, 1)) 

# apply the transformation to the target values (note that fit_transform() 
# expects a 2D tensor)
transformedTarget = scaler.fit_transform(target)

# perform training
...

# apply inverse transform
origOutput = network.forward(...)
output = scaler.inverse_transform(origOutput.data.numpy())
<ECODE>
Multi-GPU error

Xia_Yandi: After I wrapped my model with DataParallel, this error happened: RuntimeError: Assertion `THCTensor_(checkGPU)(state, 5, input, gradOutput, gradWeight, sorted, indices)’ failed. Some of weight/gradient/input tensors are located on different GPUs. Please move them to a single one. at /home/soumith/local/builder/wheel/pytorch-src/torch/lib/THCUNN/generic/LookupTable.cu:17 My model includes an embedding() layer. Is this caused by embedding()? If so, any suggestion on how to do multi-gpu properly with embedding() layers inside the model? Thanks

apaszke: 

peak: 

apaszke: Not yet, the PR still needs some fixes.

apaszke: This is merged now. But you need to build from source if you need this change right away.
What’s the purpose of “retain_variables” in Variable backward function

zhengyun: How to use “retain_variables” in Variable backward function. I tried the following code:

albanD: Hi, <SCODE>import torch
from torch.autograd import Variable
x = Variable(torch.ones(2, 2), requires_grad = True)
y = x ** 2
y.backward(torch.ones(2, 2), retain_variables=False)
print "first backward of x is:"
print x.grad
y.backward(2*torch.ones(2, 2), retain_variables=False)
print "second backward of x is:"
print x.grad<ECODE>

zhengyun: I see. Thanks very much.
Converting a Variable to a Parameter
Category: vision

Quilby: (Also notice that I’ve changed batch size to 1, is there a way to do this with bigger batches?) Thanks a lot.

apaszke: 

Quilby: Thanks a lot apaszke, I was not aware of these “functionals”. Is this the correct way to do this convolution in a batch manner? (it runs very slow…) <SCODE>    y = self.pool(F.relu(self.conv1(y)))
    z = Variable(torch.Tensor(x.size()[0], 16, 10, 10))

    for i in range(x.size()[0]):
        z[i,:]= F.conv2d(y[i,:].unsqueeze(0), x[i,:]).squeeze(0)

    z = self.pool(F.relu(z))
    z = z.view(-1, 16*5*5)
<ECODE> (x contains the convolutional weights, y contains the image I want to convolve over, and z is where I put the result) Thanks

apaszke: Why can’t you compute the convolution with all filters in one go?

fmassa: 

Quilby: Exactly, that is the point of the dynamic filter network. The conv filters for a certain picture are a function of that picture.

apaszke: You could still separate the convolutions using groups. It’s not going to be super fast, but you could give that a try.

Quilby: The sizes are: Thanks

apaszke: