foldl/torch-forum huggingface dataset:
Question about Batch Normalization

dmadeka1: The same model without the two self.bn.forward statements gives an accuracy drop to random guessing… does that make sense? <SCODE>class WideNet(nn.Module):
    def __init__(self):
        super(WideNet, self).__init__()
        self.conv1 = nn.Conv2d(1, 10, kernel_size=5)
        self.conv2 = nn.Conv2d(10, 20, kernel_size=5)
#         self.conv3 = nn.Conv2d(10, 20, kernel_size=2)
        self.conv2_drop = nn.Dropout2d()
        self.fc1 = nn.Linear(5120, 500)
        self.fcmid = nn.Linear(500, 50)
        self.fc2 = nn.Linear(50, 10)
        self.bn1 = nn.BatchNorm2d(10)
        self.bn2 = nn.BatchNorm2d(20)

    def forward(self, x):
        x = F.leaky_relu(F.max_pool2d(self.conv1(x), 2))
        x = self.bn1.forward(x)
        x = F.upsample_bilinear(x, size=(16, 16))
        x = F.leaky_relu(F.max_pool2d(self.conv2_drop(self.conv2(x)), 2))
        x = self.bn2.forward(x)
        x = F.upsample_bilinear(x, size=(16, 16))
        x = x.view(-1, 5120)
        x = F.leaky_relu(self.fc1(x))
        x = F.dropout(x, training=self.training)
        x = F.leaky_relu(self.fcmid(x))
        x = F.dropout(x, training=self.training)
        x = F.leaky_relu(self.fc2(x))
        return F.log_softmax(x)
<ECODE>

chenyuntc: 

dmadeka1: Not sure, the leaky_relu doesnt seem to have that problem. The upsample is just local replication, so I’m not sure that’s the problem. The issue is that it drops to 10%. The way I fixed it is to have: <SCODE>    def forward(self, x):
        x = self.prelu_ac[0](self.bn[0](F.max_pool2d(self.conv1(x), 2)))
        #x = self.bn[0](x)
        x = F.upsample_bilinear(x, size=(16, 16))
        x = self.prelu_ac[1](self.bn[1](F.max_pool2d(self.conv2_drop(self.conv2(x)), 2)))
        #x = self.bn[1](x)
        x = F.upsample_bilinear(x, size=(16, 16))
        x = x.view(-1, 5120)
        x = self.prelu_ac[2](self.bn[2](self.drop1(self.fc1(x))))
        #x = self.bn[2](x)
        x = F.dropout(x, training=self.training)
        x = self.prelu_ac[3](self.bn[3](self.drop2(self.fcmid(x))))
        #x = self.bn[3](x)
        x = F.dropout(x, training=self.training)
        x = self.prelu_ac[4](self.bn[4](self.fc2(x)))
        #x = self.bn[4](x)
        return F.log_softmax(x)
<ECODE> Any tips for improvement? Should I drop the upsample? I think it helps the second conv layer
Whether Variable requires_grad

Tepp: Here inputs requires_grad default is True, and labels must be False, and my question is whether tmp_conv and h_init requires_grad True in the forward. Many thx <SCODE> import torch.nn as nn
 import torch.nn.functional as F

 class Net(nn.Module):
     def __init__(self):
         super(Net, self).__init__()
         #alexnet
         self.conv1 = nn.Conv2d(3, 20, 5, stride=1)
         self.conv1_bn = nn.BatchNorm2d(20)
         #for initial
         self.fc_h2l = nn.Linear(hidden_dim, out_dim)

     def forward(self, inputs):
         #alexnet
         inputs = F.max_pool2d(F.relu(self.conv1_bn(self.conv1(inputs))), (3, 3), stride = 2)
         #variable to store inputs
         tmp_conv = Variable(torch.zeros(2,batch_size,inputs.size()[1],inputs.size()[2],inputs.size()[3]))
         tmp_conv[0,:,:,:,:] = inputs[:,:,:,:].clone()
         ......
         #for initial
         h_init = Variable(torch.randn(batch_size,hidden_dim))
         step_init = F.sigmoid(self.fc_h2l(h_init))
         .....
 alexnet = Net()
 alexnet.cuda()
 #####train
  inputs= Variable(inpt.cuda())
  labels =  Variable(labels.cuda(), requires_grad=False)
<ECODE>

apaszke: 

Tepp: Yep, thans, that is exactly what I want. However, in my classification task, tmp_conv and step_init are incorporated to form the final feature representation as below. tmp_conv and h_init should requires_grad True or False?? I am newbie here, hope i don’t bother you. Here func is some sort of function

apaszke:
Multi Label Classification in pytorch

SpandanMadan: Hi Everyone, I’m trying to use pytorch for a multilabel classification, has anyone done this yet? I have a total of 505 target labels, and samples have multiple labels (varying number per sample). I tried to solve this by banalizing my labels by making the output for each sample a 505 length vector with 1 at position i, if it maps to label i, and 0 if it doesn’t map to label i. Then, I tried to use the multilabelmarginloss(). Problem I’m facing - with more iterations, the output of the model should move towards a sparse vector with most elements 0 and a few elements 1, but it is actually moving towards a vector with often very large negative values. Can some one tell me how to go about this? Thanks!

smth: Do you have a smaller dataset? 505 target labels with very few samples might just not train well. Maybe add weight decay?

SpandanMadan: 

jlee: Can you tell us what kind of loss function you used?

AjayTalati: I’d like to try doing the same. Can you suggest any small multi-label datasets, (i.e. samples with multiple labels), to start experimenting with? I’m a novice with this type of problem, so not sure where’s the simplest place to start? Best regards, Ajay

dablyo: i’m also learning pytorch, and take it as an exercise, the input is BCHW, using multilabelmarginloss()

AjayTalati: Kind regards, Ajay

dablyo: I’ve execute train loop in python console by this way: then

sshuair: I got simiar result, I got zero loss from the second epoch. Do you find the reason?

AjayTalati: Hi there, hope that helps? Aj

bartolsthoorn: 

AjayTalati: Kind regards, Ajay

mratsim: @AjayTalati Now once you have your prediction, you need to threshold. 0.5 is the default naive way but it’s probably not optimal. In any case, once you get there, great !

nabergh: Does anyone understand how MultiMarginLoss is calculated exactly? I’m not sure I understand completely.

mratsim: 

ahkarami: Hi Everyone, However, my obtained result isn’t good at all. As a result, my precise questions are as follows:

mratsim: Your loss function is correct btw.

ahkarami: 

dablyo: the MultiLabelMarginLoss doesn’t work, loss become 0 in 2nd minibatch. I’m whole confused

SpandanMadan: Hi, I used BCELoss. It’s the standard for multi label classification in many ways. Give it a shot.
How the backward() works for torch Variable?

david-leon: For torch.autograd Variable, there’s the ‘gradient’ input param for the .backward() function. I don’t quite understand what this ‘gradient’ input param stands for, and why this param is needed?

smth: 

jekbradbury: Note that if you’re calling it on a loss/cost variable, you don’t need to provide an argument since autograd will assume a tensor of ones.

apaszke: 

Adoni: For example, I have an Embedding variable which is 10,000*100. When I used 10th and 20th embedding, will backward function calculate the gradient only for this two embeddings or will it calculate the gradient for all 10,000 embeddings?

smth: 

automatisch: I also have trouble with this. It says that the argument of backward should be the gradients w.r.t. the output. But consistently across the documentation it is never mentioned of what the gradient w.r.t. the output (must be a conspiracy to drive me crazy). Clearly you can define the gradient as an operator but then you don’t get numbers (like 1). Can the thing of which you take the gradient w.r.t. the output not be stated concisely? Does it depend on some usecase? If I just have y=f(x) for example how do I derive the 1 that is assumed?

automatisch: Am I getting there?

apaszke:
How to pass a 3D tensor to Linear layer?

maplewizard: I have a 3D tensor (5x9x12) I want to cast it to a (5x9x1) tensor through the linear layer. But I found that the nn.LinearLayer require that the input should be a matrix instead of a 3d tensor. How can I achieve my task?

kim.seonghyeon: You can do that with Tensor.view() <SCODE>x = x.view(-1, 12)
b_size = x.size(0)
x = linear(x)
x = x.view(b_size, -1, 1)<ECODE>

maplewizard: Thanks, it is likely to solve my problem. If there is a single function to to the trick, it would be more clear.

unnatjain: 

Dimitrisl: Could you explain how to do this in a non iterative manner? I think it would be very helpful

mattheus_lee:
How to convert a variable to a numpy?

maplewizard: I found that a tensor can be converted into a numpy array. but how to cast a variable into a numpy array?

maplewizard: 

albanD: 

maplewizard: Thanks. It can fulfill my requirement.

Tepp: what about replacing the var by such sort of var.data <-- np.array, it won’t autograd ???

albanD: 

deepak_mangla: var.data.numpy() will convert Variable to numpy array.
How to loop over all the variable in a nn.module

ypxie: and register hooks to the gradient?

ypxie: Thanks for your reply. Do you know how to get name  or meaningful identifier of variable?  I did not find a name field in variable.

ypxie: Thanks for your reply. But this seems to only work on layer level? <SCODE>def inves(name=''):
    def f(tensor):
        if np.isnan(torch.mean(tensor).data.cpu().numpy() ):
            print('gradient of {} is'.format(name))
            print(tensor)
            assert 0, 'nan gradient'
    return f

for key, var in Model.varibles().iteritems():
    var.register_hook(inves(key))
<ECODE>

ypxie: update:  I fond using locals() can be a solution.

apaszke: You could try that: <SCODE>def register_nan_checks(model):
    def check_grad(module, grad_input, grad_output):
        if np.isnan(grad_input.data.numpy()):
            print('NaN gradient in ' + type(module).__name__)
    model.apply(lambda module: module.register_backward_hook(check_grad))
<ECODE>

ypxie: 

apaszke: <SCODE>def register_nan_checks(model):
    def check_grad(module, grad_input, grad_output):
        # print(module) you can add this to see that the hook is called
        if any(np.all(np.isnan(gi.data.numpy())) for gi in grad_input if gi is not None):
            print('NaN gradient in ' + type(module).__name__)
    model.apply(lambda module: module.register_backward_hook(check_grad))
<ECODE> I’ve just checked and it works for me. If I add an additional print, it will show all modules.

ypxie: 

apaszke: Can you show me the code? It works for me

ypxie: 

apaszke: 

ypxie: Thanks, it works now!

ypxie: 

fmassa: 

ypxie: 

fmassa: 

ypxie: Thanks for your comments. Currently, I have no clue which part goes wrong.

ruotianluo: Why it’s not suggested to directly use forward, isn’t it just building a graph?

apaszke: 

Brando_Miranda: I dont understand what the issue is, isn’t <SCODE>    for W in mdl_sgd.parameters():
        W.data = W.data - eta*W.grad.data
<ECODE> what u do to loop over? (also it seems ur title doesn’t reflect ur real question I suggest to update it, its confusing and unnecessary).
Division of tensors that don’t have the same size

Veril: When you divide a tensor of size (64,128,32,32) by a tensor of size (64,1,32,32) you get an error because the size must match, yet in situations like these expand() is implicit and numpy does it this way. torch’s div() doesn’t operate that way, am I missing something or is this intentional? The solution is of course expand_as() but the fact that it’s not default leads me to believe that perhaps I missed something.

apaszke: We want to add broadcasting, but didn’t have time to implement that yet.

maplewizard: 

greed2411: if you are still wondering, it’s been implemented but it should be done properly. <SCODE>
>>> a = torch.rand(64, 20)
>>> b = torch.rand(64, 1)
>>> a/b
tensor([[ 5.0057e-01,  3.5622e-01,  3.1053e-01,  ...,  3.2856e-01,
          1.0888e+00,  9.7678e-01],
        [ 4.7883e+00,  5.7695e+00,  2.8125e+00,  ...,  1.6500e+01,
          3.5257e+00,  1.5637e+01],
        [ 6.8112e-01,  1.8881e+00,  2.0702e+00,  ...,  4.2512e-01,
          1.4803e+00,  8.5795e-01],
        ...,
        [ 4.1682e-01,  9.6458e-01,  1.1828e+00,  ...,  9.9901e-01,
          1.0716e+00,  1.4875e+00],
        [ 1.2503e-01,  1.2347e+00,  6.0802e-01,  ...,  5.0439e-01,
          1.2536e+00,  1.3501e+00],
        [ 9.1765e-01,  6.7741e-01,  1.0928e+00,  ...,  8.1460e-01,
          9.7924e-01,  3.8059e-01]])
>>> torch.div(a,b)
tensor([[ 5.0057e-01,  3.5622e-01,  3.1053e-01,  ...,  3.2856e-01,
          1.0888e+00,  9.7678e-01],
        [ 4.7883e+00,  5.7695e+00,  2.8125e+00,  ...,  1.6500e+01,
          3.5257e+00,  1.5637e+01],
        [ 6.8112e-01,  1.8881e+00,  2.0702e+00,  ...,  4.2512e-01,
          1.4803e+00,  8.5795e-01],
        ...,
        [ 4.1682e-01,  9.6458e-01,  1.1828e+00,  ...,  9.9901e-01,
          1.0716e+00,  1.4875e+00],
        [ 1.2503e-01,  1.2347e+00,  6.0802e-01,  ...,  5.0439e-01,
          1.2536e+00,  1.3501e+00],
        [ 9.1765e-01,  6.7741e-01,  1.0928e+00,  ...,  8.1460e-01,
          9.7924e-01,  3.8059e-01]])

<ECODE>
Question about Dropout

YongyiTang92: Hi, I would like to know how you guys deal with the dropout in testing since the dropout rate should be set to 0 while testing. I directly set model.training = False while testing. Or is there other ways to handle this?

ypxie: 

YongyiTang92: That’s cool. Thanks.

micklexqg: 

YongyiTang92: Yes, it’s. You have to set to 0 when you evaluate your model.
Anything like transformer for TensorDataset?

siberiamark: So the second scheme is to read in a bunch of data into the memory at a time into tensors: Anything like the transformer in the first approach for TensorDataset？

apaszke: No, we don’t support that at the moment but it would make sense to add that. We’re going to be rewriting some transforms soon and we’ll do that then. Thanks for a suggestion!

adelaide: Is transform available for TensorDataset now？

zym1010: this looks like something you can integrate into.

RylanSchaeffer:
RuntimeError: input is not contiguous

yunjey: When i reshape a tensor (if the rank is changed), i get the following error. <SCODE>x = ...  # Tensor of shape (100, 20)
x.view(-1)   # expect a tensor of shape (2000)

RuntimeError: input is not contiguous
<ECODE> What does ‘contiguous’ mean and why does this error occur?

apaszke: 

wasiahmad: Can you tell when we get this error? I am facing this problem but not always. I am just wondering why this is happening occasionally?

parth126: 

colesbury: 

alexis-jacq: A classical way to obtain this bug is to use transposition. If you do <SCODE>x = torch.Tensor(5,2)
y = x.t()
<ECODE> Then, the storage of y is still the same than the one of x. You can check: <SCODE>x.fill_(0)
 0  0
 0  0
 0  0
 0  0
 0  0
[torch.FloatTensor of size 5x2]

y
 0  0  0  0  0
 0  0  0  0  0
[torch.FloatTensor of size 2x5]
<ECODE> <SCODE>y.view(-1)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
RuntimeError: input is not contiguous at /b/wheel/pytorch-src/torch/lib/TH/generic/THTensor.c:231
<ECODE>

parth126: Thanks. That makes complete sense now.

ducha-aiki: Why is it not good idea to add automatic  tensor.contiguous() to transpose function?

alexis-jacq: I guess, because it saves a lot of memory to keep one storage for each tensor, while you don’t apply any reshaping transformation.

zakaria_laskar: So, in the following case z=y.contiguous().view(-1) , will the gradients of some error with respect to z backpropgate to x?.

colesbury: 

alphadl: thanks for your clear and useful reply ~~~

Brando_Miranda: <SCODE>        xn_lstm = torch.cat((loss_prep, err_prep, grad_prep), 1).unsqueeze(0) # [n_learner_params, 6]
        # normal lstm([loss, grad_prep, train_err]) = lstm(xn)
        n_learner_params = xn_lstm.size(1)
        (lstmh, lstmc) = hs[0] # previous hx from first (standard) lstm i.e. lstm_hx = (lstmh, lstmc) = hs[0]
        if lstmh.size(1) != xn_lstm.size(1): # only true when prev lstm_hx is equal to decoder/controllers hx
            # make sure that h, c from decoder/controller has the right size to go into the meta-optimizer
            expand_size = torch.Size([1,n_learner_params,self.lstm.hidden_size])
            lstmh, lstmc = lstmh.squeeze(0).expand(expand_size), lstmc.squeeze(0).expand(expand_size)
        lstm_out, (lstmh, lstmc) = self.lstm(input=xn_lstm, hx=(lstmh, lstmc))
<ECODE> error: <SCODE>RuntimeError: rnn: hx is not contiguous
<ECODE>
[resolved] How to implement k-max pooling for CNN text classification?

WarBean: A possible implmentation is <SCODE>def kmax(self, x, k):
    return x.sort(dim = 3)[0][:, :, :, -k:]
<ECODE> However, this cannot keep the information of relative position.

apaszke: 

WarBean: 

apaszke: Then I think I don’t understand the problem. What’s the exact formula for k-pooling and what’s the problem with your implementation?

WarBean: <SCODE>import torch
def kmax_pooling(x, dim, k):
    index = x.topk(k, dim = dim)[1].sort(dim = dim)[0]
    return x.gather(dim, index)
x = torch.rand(4, 5, 6, 10)
y = kmax_pooling(x, 3, 5)
print(x[0, 0])
print(y[0, 0])
<ECODE> Output: <SCODE> 0.2762  0.3788  0.5708  0.3251  0.0568  0.2483  0.3930  0.1249  0.1874  0.1113
 0.9230  0.7428  0.0957  0.2301  0.6187  0.8898  0.3007  0.2653  0.5313  0.1032
 0.6376  0.9639  0.6584  0.1502  0.0250  0.5792  0.9283  0.1783  0.9545  0.1681
 0.8456  0.6135  0.2860  0.9366  0.5178  0.0113  0.4864  0.9308  0.3005  0.5403
 0.3280  0.8755  0.2290  0.0899  0.9093  0.6971  0.1557  0.2412  0.7991  0.9169
 0.5389  0.4603  0.7291  0.4070  0.0113  0.3571  0.3860  0.3354  0.4081  0.0209
[torch.FloatTensor of size 6x10]


 0.2762  0.3788  0.5708  0.3251  0.3930
 0.9230  0.7428  0.6187  0.8898  0.5313
 0.6376  0.9639  0.6584  0.9283  0.9545
 0.8456  0.6135  0.9366  0.9308  0.5403
 0.8755  0.9093  0.6971  0.7991  0.9169
 0.5389  0.4603  0.7291  0.4070  0.4081
[torch.FloatTensor of size 6x5]
<ECODE>
How to compute a multiplication between a matrixand a number

JunshengShen: 

apaszke: What kind of product do you need? You want to scale each matrix element by the same scalar? Try this: <SCODE>a = ...
b = ...
b * a.expand_as(b)
<ECODE>

IssamLaradji: I am wondering if using expand_as is efficient in this case ? does it store only the original value and the repetitions across each axis or something similar ?  it would be cheaper to divide each matrix element by the same value without having to create the whole matrix containing the same values.
Sampling new text from Pytorch language model example

jatinganhotra: Hi, <SCODE>        output, hidden = model(input, hidden)
        word_weights = output.squeeze().data.div(args.temperature).exp().cpu()
        word_idx = torch.multinomial(word_weights, 1)[0]
        input.data.fill_(word_idx)
        word = corpus.dictionary.idx2word[word_idx]
<ECODE> I understand that this is not necessarily a Pytorch question, but would appreciate if someone could share details behind the sampling.

apaszke: A guess would be that it makes the output richer. Sticking to the most probable words would restrict the model to always use the most commonly used words, while if you use softmax, it should end up using the words approximately as often as they appear in natural language (so it will sometimes insert some more complex ones too).
Bug or am I missing something?

rguthrie3: Please check out the following short program that reproduces the bug I was facing. Imagine in each module that self.params is just some transition matrix.  The goal is to sum up the scores of each transition, where params[i][j] is the score of transitioning to i from j. Both WorkingModule and BuggyModule have a forward() function that correctly computes this score. WorkingModule does what you would expect.  If you check its gradient after calling backward(), you will see 1’s in the places where there was a transition, and 0’s elsewhere.  BuggyModule though, doesn’t backpropagate to self.params!  The difference is that in this case, the sequence was wrapped in an autograd.Variable, and the transition indices are accessed with .data. I understand the dangers of .data and how it might cut off your backprop, but how is it cutting off the backprop from score to params?  The only way sequence is ever involved is just providing an index.  In principle, score should not be cut off from params in the computation graph unless I am missing something.  In addition, I think that sequence[i].data should be evaluated before being passed to the indexing function, so I am not sure how there is any difference at all as far as constructing the “score” computation graph is concerned.

apaszke: Updating to 0.1.10 should fix it.
Pytorch converges much slower than Theano

david-leon: I’m trying out Pytorch by comparing model with its Theano version equivalence, and I noticed: The Pytorch model is defined by: <SCODE>class model_5_1(nn.Module):
    def __init__(self, batchsize=None, channel=1, imsize=(256, 256), Nclass=16, kernel_size=3, border_mode='same'):
        super(model_5_1, self).__init__()
        self.batchsize   = batchsize
        self.channel     = channel
        self.imsize      = imsize
        self.Nclass      = Nclass
        self.kernel_size = kernel_size
        self.border_mode = border_mode
        if border_mode == 'same':
            pad = kernel_size // 2
        else:
            pad = 0
        self.conv0 = nn.Conv2d(channel, 32, kernel_size, padding=pad)
        self.conv1 = nn.Conv2d(32, 64, kernel_size, padding=pad)
        self.conv2 = nn.Conv2d(96, 128, kernel_size, padding=pad)
        self.conv3 = nn.Conv2d(128, 128, kernel_size, padding=pad)
        self.conv4 = nn.Conv2d(128, 128, kernel_size, padding=pad)
        self.conv5 = nn.Conv2d(256, 512, kernel_size, padding=pad)
        self.bn0   = nn.BatchNorm2d(128)
        self.bn1   = nn.BatchNorm2d(256)
        self.bn2   = nn.BatchNorm2d(512)
        self.rnn0  = nn.LSTM(input_size=512, hidden_size=100, batch_first=True, bidirectional=True)
        self.rnn1  = nn.LSTM(input_size=200, hidden_size=100, batch_first=True, bidirectional=True)
        self.fc0   = nn.Linear(200, Nclass)

    def forward(self, x):
        x  = F.max_pool2d(F.relu(self.conv0(x)), (2, 2))
        x1 = F.relu(self.conv1(x))
        x  = tr.cat((x, x1), 1)
        x  = F.max_pool2d(x, (2,2))
        x  = F.relu(self.bn0(self.conv2(x)))

        x  = F.max_pool2d(F.relu(self.conv3(x)), (2,2))
        x1 = F.relu(self.conv4(x))
        x  = tr.cat((x, x1), 1)
        x  = self.bn1(x)

        x  = F.max_pool2d(F.relu(self.conv5(x)), (4,4))
        x  = self.bn2(x)
        x  = x.view(x.size(0), x.size(1), x.size(2) * x.size(3))
        x  = tr.transpose(x, 1, 2)
        x, _ = self.rnn0(x)
        x  = F.tanh(x)
        x, _ = self.rnn1(x)
        x  = x[:,-1,:]
        x  = F.tanh(x)
        x  = F.softmax(self.fc0(x))
        return x
<ECODE> And I use cross_entropy for loss, Adadelta for optimizer.  For Theano version I use categorical_crossentropy for loss, Adadelta with same parameters for optimizer. Anyone has any thought on this problem?

kim.seonghyeon: You don’t need to use softmax in the forward operation. Maybe it is the cause of the problem.

fmassa: 

david-leon: