akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Multilayer RNN using RNNCell

username: harshildarji
cooked: <p>Hey all,</p>
<p>I am trying to implement a fully connected multilayer RNN using <code>torch.nn.RNNCell</code>.</p>
<p>I have implemented it, but it looks like it is not working.</p>
<p>Here is the code for reference:</p>
<pre><code class="lang-python">class MultilayerRNN(nn.Module):
    def __init__(self, input_size, output_size, hidden_layers = [100, 100, 100]):
        super(NormalDeepRNN, self).__init__()
        assert len(hidden_layers) &gt; 0

        self.hidden_layers = hidden_layers
        self.num_layers = len(hidden_layers)
        self.input_size = input_size

        self.embedding = nn.Embedding(self.input_size, self.hidden_layers[0])
        self.layer_out = nn.Linear(hidden_layers[-1], num_classes)
        
    def forward(self, input):
        # input.shape: [BATCH_SIZE, INPUT_SIZE], e.g. [16, 128]

        # After trnaspose,
        # input.shape: [INPUT_SIZE, BATCH_SIZE], e.g. [128, 16]
        input = input.t()

        # After embedding,
        # input.shape: [INPUT_SIZE, BATCH_SIZE, EMBEDDING_DIM], e.g. [128, 16, 100]
        input = self.embedding(input)

        for i, out_size in enumerate(self.hidden_layers):
            if i == 0:
               # For first hidden layer, input_size == embedding_dim
                in_size = input.size(2)
            else:
                in_size = self.hidden_layers[i-1]

            rnn = nn.RNNCell(in_size, out_size, 'tanh')
            hx = self.__init_hidden(input.size(1), in_size)

            input = self.__cell(rnn, input, hx)

        output = self.layer_out(input[-1])
        return output, input[-1]
            

    def __cell(self, cell, input, hx):
        inputs = input.unbind(0)
        outputs= []

        for i in range(len(inputs)):
            hx = cell(inputs[i], hx)
            outputs += [hx]

        return torch.stack(outputs)

    def __init_hidden(self, batch_size, hidden_size):
        hidden = torch.zeros(batch_size, hidden_size)
        return Variable(hidden)
</code></pre>
<p>It always gives accuracy around <strong>50%</strong>, which is baseline considering I am trying binary classification.</p>
<p>The same dataset gives around <strong>85%</strong> accuracy with standard RNN modules (e.g. <code>nn.RNNBase</code> or <code>nn.RNN</code>).</p>
<p>I am pretty sure there is something wrong with my <code>forward</code> method, but I am unable to point it out.</p>
<p>Any help is appreciated.<br>
Thank you.</p>
<p>Best,<br>
Harshil</p>

username: harshildarji
cooked: <p>Hey all,<br>
Just as an update, I solved this problem by changing <strong>the initialization</strong> of <code>nn.RNNCell</code> from <code>forward</code> to <code>__init__</code> method.<br>
Thank you.</p>
PyTorch debug strange behavior

username: whoab
cooked: <pre><code class="lang-auto">class MainModule(nn.Module):
...
self.proj = Mlp()
&lt;end MainModule&gt;


output = self.main_module(output)
</code></pre>
<pre><code class="lang-auto">class MainModule(nn.Module):
...
&lt;end MainModule&gt;

output = self.main_module(output)
output = self.proj(output)
</code></pre>
<p>These two bits of code should be exactly the same, yet they consistently produce very different results for every run. How can I debug this?</p>
<p>If I somehow can ensure that self.proj has the exact same weights (I don’t think setting the random seed works when the code itself is different from run to run) will let me at least see if the forward activations and gradient updates are different… how do I do that?</p>

username: ptrblck
cooked: <p>I don’t fully understand the code.<br>
In the first example <code>self.proj</code> seems to be a part of your <code>MainModule</code> class, while in the second you are calling two different functions?</p>
<p>If both models have the same layers, I would suggest do load the <code>state_dict</code> from one model to the other model, thus making sure all parameters are exactly the same. Then use a constant input tensor (or set the seed before creating a random one), and check the difference in the outputs.<br>
Note that an absolute error of ~1e-6 is most likely due to limited floating point precision.</p>
<p>Also, if one or both models contains e.g. <code>nn.Dropout</code> layers, you should disable it by calling <code>model.eval()</code> in them first.</p>
Pytorch load checkpoints to continue training - zero_grad() or not?

username: John_J_Watson
cooked: <p>I have saved and loaded the checkpoints as per the pytorch manual, and it all seems OK. Now, usually, when I want to start training, I have something like this in pytorch:</p>
<pre><code>    for itr in range(1, args.niters + 1):
        optimizer.zero_grad() # should I or should I not?
</code></pre>
<p>I am unsure if I should do <code>zero_grad()</code> here (which I use when I start training from scratch), since I am reloading all my weights and bias.</p>
<p>Apologies if this is a daft question.</p>

username: beaupreda
cooked: <p>Hello,</p>
<p>Yes, you should still use <code>optimizer.zero_grad()</code> since  the only thing it does is clearing the gradients. When you load a checkpoint, you load the weights of your model (parameters), you do not load any gradients. So if you keep training from this checkpoint, you still want to reset your gradients after each step.</p>
<p>And I don’t believe there are any <em>daft</em> questions (in fact, I just learned a new word <img src="https://discuss.pytorch.org/images/emoji/apple/upside_down_face.png?v=9" title=":upside_down_face:" class="emoji" alt=":upside_down_face:">)!</p>
Implement forward hook that has different behavior during train and eval mode

username: MrRobot
cooked: <h1>Problem</h1>
<p>I am trying to record some statistics in a list for both training and validation step. However, doing something like following will mix <code>stat</code> from training and validation into the same list.</p>
<pre><code class="lang-python">stat_list = []
def hook(self, input, output):
  # compute stat
  stat_list.append(stat)

model.&lt;layer&gt;.register_forward_hook(hook)
</code></pre>
<p>Since I know PyTorch has <code>model.train()</code>  and <code>model.eval()</code>, I am wondering if I could specify behavior of hook using certain flag returned by <code>model.train()</code> and <code>model.eval()</code>. In my use case, it would be like</p>
<pre><code class="lang-python">train_stat_list = []
val_stat_list = []
def hook(self, input, output):
  if train_flag:
   # compute stat
   train_stat_list.append(stat)
  if val_flag:
   # compute stat
   val_stat_list.append(stat)
</code></pre>

username: MrRobot
cooked: <p>Just figured this out by myself :-D.</p>
<p>It turns out there is a variable in <code>nn.Module</code> called <code>training</code> and <code>model.train()</code>  and <code>model.eval()</code> essentially makes it <code>True/False</code> (default <code>True</code>)</p>
<pre><code class="lang-python">from torch import nn

model  = nn.Module()
model.training
# True
model.eval()
model.training
# False
</code></pre>

username: albanD
cooked: <p>Yes, the <code>.training</code> flag on the module is what should be used to know whether you are in training or eval mode.</p>
Pytorch customize weight

username: wangyizhou30
cooked: <p>I have a network</p>
<pre><code class="lang-auto">class Net(nn.Module)
</code></pre>
<p>and two different weights <code>w0</code> and <code>w1</code> (concatenate weights of all layers into a vector). Now I want to optimize the network on the line connecting <code>w0</code> and <code>w1</code>, which means that the weight will have the form <code>theta  * w0 + (1-theta) * w1</code>. So now the parameter I want to optimize is no long the weight itself, but the <code>theta</code>.</p>
<p>How can I implement this? In Pytorch, how can I define the parameter to be <code>theta</code>, and set the weight to be form I want. To be specific, if I create a new class</p>
<pre><code class="lang-auto">NetOnLine(nn.Module)
</code></pre>
<p>how should I write the <code>forward(self, X)</code> function?</p>
How can I add different regularization to different layer

username: Fly
cooked: <p>The default regularization not suit me, how can add different regularization to different layer type.</p>

username: ptrblck
cooked: <p>You could add an own regularization term to each layer. <a href="https://discuss.pytorch.org/t/simple-l2-regularization/139/2?u=ptrblck">This post</a> shows a similar example for L1Loss but on each parameter.</p>

username: sksenthilkumar
cooked: <p>I have a similar need to have different regularization factor for different layers.</p>
<p>The post you have referred here calculates the regularization loss using the L1 norm of each parameter. But they add all the regularization_loss to the final loss. And during back propagation, this loss is common to all the parameters and they are not different for different parameters.  in my case how to add different regularization factor for different layers?</p>

username: ptrblck
cooked: <p>You could adapt the for loop and add the desired regularization to the corresponding layer.<br>
In the current example, the code just iterates all parameters, so you could add some if conditions etc. (depends on your use case).</p>

username: sksenthilkumar
cooked: <p>Thanks for that. But it is still tough for me to grasp the idea. Can you show me an example?</p>
<p>Maybe for this one:<br>
I have a LSTM model with 3 layers</p>
<p>lstm1 =  nn.LSTM(150, 100, num_layers=1, batch_first=True)<br>
lstm2 = nn.LSTM(100, 110, num_layers=1, batch_first=True)<br>
lstm3 = nn.LSTM(110, 200, num_layers=1, batch_first=True)</p>
<p>and during backpropagation,<br>
I want to apply the norm1(lstm1 weights) only on lstm1 and similarly for other layers. How should my loss function look like?</p>
Fourier-CNN Model Architecture Implementation

username: Joy
cooked: <p>How to implement this model architecture in PyTorch?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/d/2dad967698931fc58091a16c7d0d2726a1ba852f.png" data-download-href="https://discuss.pytorch.org/uploads/default/2dad967698931fc58091a16c7d0d2726a1ba852f" title="A0PWf"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/d/2dad967698931fc58091a16c7d0d2726a1ba852f_2_408x500.png" alt="A0PWf" data-base62-sha1="6w5oz0jGFj6qaxe873k77xcNGhh" width="408" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/d/2dad967698931fc58091a16c7d0d2726a1ba852f_2_408x500.png, https://discuss.pytorch.org/uploads/default/original/3X/2/d/2dad967698931fc58091a16c7d0d2726a1ba852f.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/2/d/2dad967698931fc58091a16c7d0d2726a1ba852f.png 2x" data-dominant-color="EFEFEF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">A0PWf</span><span class="informations">529×647 64.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Paper: <a href="https://link.springer.com/chapter/10.1007/978-3-319-71249-9_47" class="inline-onebox" rel="noopener nofollow ugc">FCNN: Fourier Convolutional Neural Networks | SpringerLink</a></p>

username: SoucheChapich
cooked: <p>Hello,</p>
<p>I am not exactly sure about what they are doing on that paper (could not view it) but seems that they are taking the input and they took the Fourrier transform of it. I suppose you should compute the Fourrier transform after the input layer (see torch.fft) and then apply the convolution and max pooling operations. Then taking the inverse Fourier transform to getting back to the spatial domain.<br>
But again, I am not sure about that because I could not read the paper.</p>
How to set different weight initialization parameters for each layers?

username: DeepLearner17
cooked: <p>Hello,</p>
<p>I define the following function to initialize the weights of my network of different layers.</p>
<p>l have 5 different convolutional layers of the same dimensions.<br>
And 3 different linear layers of the same dimensions.</p>
<p>Is this way of initializing the network ensures that all the layers have different weights initially ?</p>
<p><strong>Weight(conv1)  not equal Weight(conv2)  not equal Weight(conv3)   not equal Weight(conv4)  not equal Weight(conv5)</strong></p>
<p>and</p>
<p><strong>Weight(linear1)  not equal Weight(linear2)  not equal Weight(linear3)</strong></p>
<pre><code class="lang-auto">seed = 42
torch.manual_seed(seed)
np.random.seed(seed)
torch.cuda.manual_seed(seed)

def weights_init(m):
    classname = m.__class__.__name__
    if classname.find('Conv') != -1:
        m.weight.data.normal_(0.0, 0.02)
    if classname.find('Linear') != -1:
        # get the number of the inputs
        n = m.in_features
        y = 1.0 / np.sqrt(n)
        m.weight.data.uniform_(-y, y)
        m.bias.data.fill_(0)
    elif classname.find('BatchNorm') != -1:
        m.normal_(m.weight.data, mean=1, std=0.02)
        m.constant_(m.bias.data, 0)

model=ConvNet()
model.apply(weights_init)
</code></pre>

username: albanD
cooked: <p>You can remove all the <code>.data</code> and replace them with:</p>
<pre><code class="lang-python">@torch.no_grad()
def weights_init(m):
  # Your code
</code></pre>
<p>And yes this will reinitialize all the weights with random values.<br>
You might be interested by the <a href="https://pytorch.org/docs/stable/nn.init.html">torch.nn.init</a> package that gives you many common initialization methods.</p>

username: DeepLearner17
cooked: <p>Thank you for your answer <a class="mention" href="/u/alband">@albanD</a>,</p>
<p>Is it right ?</p>
<pre><code class="lang-auto">@torch.no_grad()
def weights_init(m):
    classname = m.__class__.__name__
    if classname.find('Conv') != -1:
        m.weight.normal_(0.0, 0.02)
    if classname.find('Linear') != -1:
        # get the number of the inputs
        n = m.in_features
        y = 1.0 / np.sqrt(n)
        m.weight.uniform_(-y, y)
        m.bias.fill_(0)

    elif classname.find('BatchNorm') != -1:
        m.normal_(m.weight, mean=1, std=0.02)
        m.constant_(m.bias, 0)
</code></pre>
<p>is <code>@torch.no_grad()</code> different from      <code>torch.no_grad()</code>  ?</p>
<p>What is wrong with <code>.data</code> ?</p>

username: albanD
cooked: <p>The two are the same, one is a context manager, the other is a function decorator. It is the same as running your whole function in <code>with torch.no_grad()</code>. So it is quite convenient.</p>
<p>The problem is that <code>.data</code> can hide some errors and give you wrong gradients.<br>
For example, this issue poped up today: <a href="https://github.com/pytorch/pytorch/issues/30073">https://github.com/pytorch/pytorch/issues/30073</a> which is caused by the use of <code>.data</code> in the old codebase.</p>
Gradient calculated in custom autograd function does not assigned to .grad field

username: MariosOreo
cooked: <p>Hello,</p>
<p>I try to run will the custom autograd function as follows, which is used in <code>mixed_operation</code> of ProxylessNAS.</p>
<pre><code class="lang-auto">class ArchGradientFunction(torch.autograd.Function):

    @staticmethod
    def forward(ctx, x, binary_gates, run_func, backward_func):
        ctx.run_func = run_func
        ctx.backward_func = backward_func
        detached_x = detach_variable(x)
        with torch.enable_grad():
            output = run_func(detached_x)
        ctx.save_for_backward(detached_x, output)
        return output.data

    @staticmethod
    def backward(ctx, grad_output):
        detached_x, output = ctx.saved_tensors
        grad_x = torch.autograd.grad(output, detached_x, grad_output, only_inputs=True)
        # compute gradients w.r.t. binary_gates
        binary_grads = ctx.backward_func(detached_x.data, output.data, grad_output.data)
        return grad_x[0], binary_grads, None, None
</code></pre>
<p>As shown that the gradients w.r.t. binary_gates is calculated in backward. But <code>binary_gates.grad</code> is <strong>None</strong> when I track it by backward hooks. I know that asking questions for some specific project is not unwise choice. Since the authors close the issue block on Github, I have to ask you for some help, what could be the error of this case?</p>
<p>Thanks in advance!</p>

username: albanD
cooked: <p>Hi,</p>
<p>Asking question for specific projects on the forum is fine.</p>
<p>You should be careful and never use <code>.data</code>. You should use <code>.detach()</code> or <code>with torch.no_grad()</code>.</p>
<p><code>binary_gates.grad</code> will only be populated if it’s a leaf variable and it requires gradients.<br>
Also, the backward hook on nn.Modules (that should not be used, see the doc), run before the <code>.grad</code> field for its inputs is populated. So it is expected even for a leaf Tensor that you don’t see it in the hook.</p>
Pin_memory and num_workers in pytorch data loaders

username: Luca_Pamparana
cooked: <p>I cannot find the link anywhere but I read once that if we set <code>pin_memory = True</code> in the pytorch data loaders then the <code>num_workers</code> threads should be set to 1.</p>
<p>Is this true? I am not sure if I am misremembering. If yes, what  is the reason behind not being able to use multiple threads when using <code>pin_memory</code>. It seems very counter intuitive that this would play a part as every thread would be copying data to a separate memory area.</p>

username: albanD
cooked: <p>Hi,</p>
<p>I think you’re misremembering. The two options are independent. And the <code>num_workers</code> is a number of processes, not threads.</p>
Recurrent net for a sequence similarity of floats

username: Chame_call
cooked: <p>Is that possible to classify similarity via rnn?<br>
For instance I pass to the network input two sequences(by batches consisting of one element of every sequence).<br>
And what I’d like is to find out whether there’s a correlation between them or there’s not.<br>
In addition to that:</p>
<ul>
<li>sequences may be shifted w.r.t. each other;</li>
<li>similarity in here doesn’t imply exact match; the most important thing is matching of according ups and down.</li>
</ul>
<p>for example here’s two plots of such similar sequences are the following:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/6/06c1192957a06665914ea4eee7276966265999dc.png" data-download-href="https://discuss.pytorch.org/uploads/default/06c1192957a06665914ea4eee7276966265999dc" title="Selection_030"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/6/06c1192957a06665914ea4eee7276966265999dc_2_370x500.png" alt="Selection_030" data-base62-sha1="XKzI7G1YamwKmaXLcaCLS7RELi" width="370" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/6/06c1192957a06665914ea4eee7276966265999dc_2_370x500.png, https://discuss.pytorch.org/uploads/default/original/3X/0/6/06c1192957a06665914ea4eee7276966265999dc.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/0/6/06c1192957a06665914ea4eee7276966265999dc.png 2x" data-dominant-color="F8F7F6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Selection_030</span><span class="informations">435×587 56.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: Chame_call
cooked: <p>Maybe somebody had similar experience…</p>
How can I free GPU memory for a specific tensor?

username: seankala
cooked: <p>Hello. I’m currently running a deep learning program using PyTorch and wanted to free the GPU memory for a specific tensor.</p>
<p>I’ve thought of methods like <code>del</code> and <code>torch.cuda.empty_cache()</code>, but <code>del</code> doesn’t seem to work properly (I’m not even sure if it frees memory at all) and <code>torch.cuda.empty_cache()</code> seems to free all unused memory, but I want to free memory for just a specific tensor.</p>
<p>Is there any functionality in PyTorch that provides this?</p>
<p>Thanks in advance.</p>

username: albanD
cooked: <p>Hi,</p>
<p>Why do you want to free the memory associated with a single specific Tensor?<br>
<code>del foo</code> will remove the link between the variable <code>foo</code> and the Tensor it contains. If nothing else uses the Tensor, it will be freed. But if other stuff use it (other views, or grad computation), it won’t be deleted right away.</p>

username: seankala
cooked: <p>Hello, thanks for the reply. There’s no particular grandiose reason, but I just wanted to try and make a project as memory-efficient as possible. I noticed that there were certain variables that were only used once and not used afterwards, and thought that freeing up the memory for those tensors would be good. I tried the <code>del foo</code>, but I also realized that this simply “frees” the memory and doesn’t return it to the device.</p>
<p>I was initially confused because I wasn’t aware of the difference between freeing memory and returning it to the device. Perhaps my knowledge is a bit short, but “freeing and not returning the device” is essentially allowing for more memory usage, right?</p>

username: albanD
cooked: <p>Yes.<br>
Because the allocator from cuda is a bit slow, we have a custom one in pytorch that is much faster.<br>
So when you free a Tensor, its memory is returned to the custom allocator that can then reuse it for other Tensors.</p>
Error on torch.Tensor.scatter

username: david-leon
cooked: <p>For the document example:</p>
<pre><code class="lang-auto">x = torch.rand(2, 5)
torch.zeros(3, 5).scatter_(0, torch.tensor([[0, 1, 2, 0, 0], [2, 0, 0, 1, 2]]), x)
</code></pre>
<p>there is no problem, however if the code snippet is changed to</p>
<pre><code class="lang-auto">x = torch.rand(2, 5)
torch.Tensor.scatter(0, torch.tensor([[0, 1, 2, 0, 0], [2, 0, 0, 1, 2]]), x)
</code></pre>
<p>Error raised as <code>TypeError: descriptor 'scatter' requires a 'torch._C._TensorBase' object but received a 'int'</code></p>

username: albanD
cooked: <p>You need to give it <code>x</code> as the Tensor to scatter the values into <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: david-leon
cooked: <p>Sorry I don’t get it. According to the documentation, <a href="https://pytorch.org/docs/stable/tensors.html#torch.Tensor.scatter" rel="nofollow noopener">https://pytorch.org/docs/stable/tensors.html#torch.Tensor.scatter</a>, the syntax is<br>
<code>scatter(dim, index, source) → Tensor</code>, so where should I put the <code>Tensor to scatter the values into</code>?</p>

username: david-leon
cooked: <p>Are you saying <code>z.scatter(0, torch.tensor([[0, 1, 2, 0, 0], [2, 0, 0, 1, 2]]), x)</code>?</p>

username: albanD
cooked: <p>The <code>torch.Tensor</code> elements in the docs are methods on Tensors.<br>
So they should be called on a tensor: <code>x.scatter(xxx)</code>.</p>
<p>If you want to use the version in <code>torch.</code>, then you need to provide the tensor as first argument: <code>torch.scatter(x, foo)</code>.</p>
Very creative maxpooling, don't konw how to implement

username: 111188
cooked: <p>suppost I have a tensor H * 3, and it’s cooresponding ids: H * 1,  ids[i] means tensor[i,:] belong to one block,  the numbers in ids can have same value, for example, ids = [0,0,0,1,2,3,5,5], now I want do maxpooling for those tensor[i,:] belong to same id.</p>
<p>For example:<br>
tensor :  4 * 3<br>
1 1 1<br>
2 2 2<br>
3 3 3<br>
4 4 4</p>
<p>ids:<br>
0<br>
0<br>
0<br>
1</p>
<p>now, do maxpooling,  1 1 1,  2 2 2, 3 3 3 do maxpooling , get 3 3 3 , 4 4 4 do maxpooling, get 4 4 4</p>
<p>use for loop is too slow, my H * 3, H is generally 20000</p>
Cant plot image of tensors

username: dato_nefaridze
cooked: <p>i am converting numpy matrix to tensors and plotting them to google colab.as a numpy array it works fine but as a tensor it does not:<br>
i have a array of tensors x:</p>
<pre><code class="lang-auto">X = torch.Tensor([i[0] for i in train_data]).view(-1, 3, 50, 50)
</code></pre>
<p>then i am plotting firs element of this array like this:</p>
<pre><code class="lang-auto">c2_imshow(X[0].numpy().transpose(1, 2, 0))
</code></pre>
<p>but instead of dog picture it outputs something else  like this:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/c/6c8de8659abc4c29bde5073ee60424f80dba25f8.png" alt="tr" data-base62-sha1="fujBnqszpdEMrGSU3yD0wWHdZB6" width="50" height="50"><br>
how can i output real picture(relevant)</p>

username: JuanFMontesinos
cooked: <blockquote>
<p>i am converting numpy matrix to tensors and plotting them to google colab.as a numpy array it works fine but as a tensor it does not:</p>
</blockquote>
<p>So:<br>
numpy-&gt;tensor-&gt;numpy works but<br>
tensor-&gt;numpy doesn’t?<br>
Are you sure you are applying a proper reshaping?<br>
c2_imshow is cv2? are you aware it’s bgr?</p>

username: dato_nefaridze
cooked: <p>no numpy works, numpy-&gt;tensor doesn’t</p>
Error installing Torch

username: Bennett_Zug
cooked: <p>I’ve tried to install pytorch like 10 times. I am running windows 10 and python 3.7. I have been able to install other programs, but pytorch refuses to work no matter what I try. I’ve tried pip install pytorch, which returns a huge error, the most important part seeming to be “no module named tools.nnwrap” (I installed the module tools after seeing this error, didnt fix it) I then looked up the error and found the start locally page on the PyTorch website, and that also didn’t work. that returned the error “ERROR: Could not find a version that satisfies the requirement torch==1.2.0+cu92 (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2)<br>
ERROR: No matching distribution found for torch==1.2.0+cu92” or some variation of that depending on the specific settings i chose. I then scrolled down and selected what I thought was right manually, and that threw back “ERROR: torch-1.0.1-cp37-cp37m-win_amd64.whl is not a supported wheel on this platform.” To my knowledge, I’ve tried every solution on the internet, and none of them work. Does anyone with more experience know how to help me?</p>

username: peterjc123
cooked: <p>Either you have 32-bit Python or you have multiple Python distributions and mixed them together.<br>
For the first case, you could type in the following command to verify.</p>
<pre><code class="lang-auto">python -c "import struct;print( 8 * struct.calcsize('P'))"
# The output should be 64.
</code></pre>
<p>For the second case, please type in the following commands.</p>
<pre><code class="lang-auto">where python
where pip
# A valid output should be like this.
# C:\python37\python.exe
# C:\python37\scripts\pip.exe
# If they don't match, then you could run the command with a specified pip.
# Suppose the output is
# C:\python37\python.exe
# D:\python37\scripts\pip.exe
# Then you can do
# "C:\python37\scripts\pip.exe" install -f https://download.pytorch.org/whl/torch_stable.html torch==1.2.0+cu92
</code></pre>

username: rusty
cooked: <p>Indeed I had the same problem. I was configuring my new machine and forgot to check if I had a 32-bit or a 64-bit python. Removing the 32-bit and installing the 64-bit version of Python 3.7 solved the problem.</p>

username: Sai_Keerthi_K
cooked: <aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://pytorch.org/favicon.ico?" class="site-icon" width="16" height="16">
      <a href="https://pytorch.org/get-started/locally/" target="_blank" rel="nofollow noopener">pytorch.org</a>
  </header>
  <article class="onebox-body">
    <img src="https://pytorch.org/assets/images/pytorch-logo.png" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://pytorch.org/get-started/locally/" target="_blank" rel="nofollow noopener">PyTorch</a></h3>

<p>An open source deep learning platform that provides a seamless path from research prototyping to production deployment.</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>Install via above link. It should work.</p>
Distributed loss function

username: norm_inf
cooked: <p>Is there any example of how to calculate the loss within multiple GPU and merge all of them later after the calculation?</p>
<p>Currently, we could calculate the output from a network by using DistributedDataParalel. However, the result from DistributedDataParallel is collected in device 0. Therefore, the calculation was done in 1 GPU only instead of multi-GPU.</p>

username: pietern
cooked: <p>What do you mean exactly here? Are you looking to compute only a single loss value for a model that gets executed on multiple processes? Or just on multiple GPUs from a single process?</p>
<p>Even though the result is collected in GPU 0, the gradients will propagate back through the activations on all GPUs that contributed in computing the final loss value. The gradients that are computed for every replica are averaged automatically by <code>torch.nn.parallel.DistributedDataParallel</code>, so all replicas contribute to the gradient that is later used by the optimizer to update your model weights.</p>

username: Zhilin_Lu
cooked: <p>I am facing the same problem, have you find any proper solution?</p>
Implementation of Coverage loss, loss is always 1

username: gionanide
cooked: <p>I am trying to implement coverage loss like: <a href="https://arxiv.org/pdf/1704.04368.pdf" rel="nofollow noopener">https://arxiv.org/pdf/1704.04368.pdf</a><br>
activations shape: [batch, timesteps, features]<br>
mask shape: [batch, timesteps] to recognize the padding</p>
<p>For example, If I want to calculate the coverage loss for the 5ith element in the batch. I am taking the element [5, timesteps, features]. I iterate all the timesteps as follows:</p>
<ul>
<li>for the 1st timestep, take the min between [zero_vector, activations[5][1]</li>
<li>for the 10th timestep, take the min between [sum_of_all_previous_activations, activations[5][10]]</li>
</ul>
<p>every time I change sample in the batch I repeat the following procedure.</p>
<ul>
<li>In the end the coverage loss is always 1</li>
</ul>
<pre><code class="lang-auto">import torch
torch.manual_seed(150)
torch.set_default_dtype(torch.float64)

mask = torch.ones(10,10)
activations = torch.rand(10,10,3518)
activations_distribution = torch.nn.functional.softmax(activations,dim=2)
overall_coverage = 0#initialize the overall loss for the batch
or index, activation in enumerate(activations_distribution): #iterate every batch
        
                loss = 0 #initialize the coverage loss for a sentence in the batch
                overall_weights = torch.zeros(activation.shape[1]).to(device)#the first coverage vector equals to zero
                
                for time, timestep in enumerate(activation): #iterate every single timestep           
                        
                        if (mask[index][time]==0):#it means that we foudn the first padding symbol, so there are only padding symbols from this point till the end of sequence, se we stop
                                break
                     
                        loss = torch.sum(torch.min(overall_weights,timestep)) #find the min element of the current dimensions, compare elementwise
                        overall_weights = torch.sum(activation[:time+1,:],dim=0).to(device) #after timestep = 0 || now we are taking the (unormalized)sum of the previous activations
                        
                overall_coverage = overall_coverage + loss # keep track of the general conerage loss of the batch
                
        cov_loss = overall_coverage/activations_distribution.shape[0] #divide by the batch size
print(index,'loss',cov_loss)
</code></pre>

username: gionanide
cooked: <p>The problem was that the classes(in the previous example was 3518) this big number of classes conclude to very small number after the softmax pass. So every time the sum of the previous softmax outputs was all the time bigger than the current output. This led to taking all the time as min the element of the current softmax output concluding in just the sum of all the elements of the current softmax output, which every time is 1.</p>
<p>To change this we have to take the mean of all the previous sums of softmax output.</p>
<pre><code class="lang-auto">overall_weights = ((torch.sum(activation[:time+1,:],dim=0))/(time+1)).to(device)
</code></pre>
How to find closest point in two point sets

username: fan_percy
cooked: <p>Hi,<br>
I have two sets of point in 3D ,as X and Y, of the same shape (Batchsize, Num_Points, 3).</p>
<p>I’m trying to find the index of the closest point in Y for every point in X, and vice versa, the index of the closest point in X for every point in Y.</p>
<p>I’m wondering is there any function in PyTorch I can directly apply?</p>
<p>Thank you in advance for your help!</p>

username: Mendel123
cooked: <p>You can compute a distance matrix first and then find the smallest one.</p>
Speech to Text Training on MFCC feature extraction

username: amittiwari
cooked: <p>I have done preprocessing using MFCC , but model training get stuck in early steps.</p>
<p>python3 train.py -model_type audio -enc_rnn_size 2048 -dec_rnn_size 1024 -audio_enc_pooling 1,1,1,1,2,2,2,2 -dropout 0.1 -enc_layers 8 -dec_layers 6 -rnn_type LSTM -data data/speech/demo -save_model models-global_attention mlp -batch_size 8 -save_checkpoint 10000 -optim adam -max_grad_norm 100 -learning_rate 0.01 -learning_rate_decay 0.5 -decay_method rsqrt -train_steps 150000 -encoder_type brnn -decoder_type rnn -normalization tokens -bridge -window_size 0.025 -image_channel_size 3 -gpu_ranks 0 -world_size 1</p>
<p>Step 30000/150000; acc: 15.65; ppl: 268.82; xent: 5.59; lr: 0.00017;<br>
Step 40000/150000; acc: 17.03; ppl: 253.23; xent: 5.53; lr: 0.00015;<br>
Step 50000/150000; acc: 17.06; ppl: 229.16; xent: 5.43; lr: 0.00013;<br>
Step 60000/150000; acc: 17.52; ppl: 225.95; xent: 5.42; lr: 0.00012</p>
<p>TRAINING accuracy improving very slowly.<br>
What should be done for faster convergence of training?</p>
DataLoader should Cache the same index computation?

username: Rafael_R
cooked: <p>Hi,</p>
<p>I have a dataloader, which for each index, does some computation like mean features etc.</p>
<p>Doing this computation again and again is slowing down training, is there any way the dataloader doesn’t have to compute it?</p>
<p>The obvious solution is to do that computation outside the data loader, however, any other solution (like having a separate function than getitem is welcome.</p>

username: ptrblck
cooked: <p>As you said, you could compute these features offline once and store them as an attribute inside the <code>Dataset</code>.<br>
If you want to compute them in the first epoch using your <code>DataLoader</code>, you might want to use a shared array and store the features there (as shown <a href="https://github.com/ptrblck/pytorch_misc/blob/master/shared_array.py">here</a>), so that they will be reusable in the following epochs from each worker.</p>
Covert list to tensor

username: faizy23395
cooked: <p>I;m working on language modeling and using LSTM. I have txt file in which pair of person name and  and there native language is given e.g (zack,france<br>
ali,india)  and  i have converted it into one list  name of person and in one list i have name of countries .eg.<br>
name=[" Ali",“zack”]<br>
lang=[ “france”,“india”]<br>
now i want to convert these to list into tensors .I have tried like in two ways</p>
<ol>
<li>
<p>name = torch.from_numpy(name)</p>
</li>
<li>
<p>name = [float(name) for x in name]<br>
name = np.asarray(name)<br>
name = torch.from_numpy(name)<br>
but in first case it says list object dont have to obbject  so i tried to convert it into float first so that i can convert them into tensor because in the documentation it says it can convert something like that The only supported types are: double, float, int64, int32, and uint8.<br>
and in second case its giving error like this could not convert string to float: ‘Abreu’ where ‘Abreu’ is person name.<br>
Can anyone help me how to convert this thing into tensor.Thank You</p>
</li>
</ol>

username: ptrblck
cooked: <p>Usually you would use a dictionary to convert words to indices, e.g. as done in <a href="https://pytorch.org/tutorials/intermediate/char_rnn_classification_tutorial.html#turning-names-into-tensors">this tutorial</a>.</p>
Getting the data contents from iterator in torchtext

username: Syed_Ikram
cooked: <p>Here is a snippet of my code:</p>
<p>TEXT = Field(lower=True,sequential=True, use_vocab=True, include_lengths = True)<br>
LABEL = LabelField(sequential=False)</p>
<p>val_fields = [(‘title’,TEXT), (‘label’,LABEL)]</p>
<p>valid_data= TabularDataset.splits(path = ‘./’,<br>
validation = ‘valid.csv’,<br>
format = ‘csv’,<br>
fields = val_fields,<br>
skip_header=True)</p>
<p>valid_iterator = BucketIterator.splits(<br>
(valid_data),<br>
sort_key=lambda x: len(x.title),<br>
sort_within_batch=False,<br>
batch_size=1,<br>
device=device)</p>
<p>for batch in valid_iterator:<br>
print(batch.title[0])<br>
print(batch.label[0])</p>
<p>printing the batch.title[0]  and batch.label[0] gives me a tensor but I want to know what is the text content and label while iterating this loop? I need this cause just wanted to visualize which are the data points which are misclassified during the prediction in eval mode</p>
What does "inputs" means in pytorch LSTM with attention

username: damien
cooked: <p>Detail code is in <a href="https://blog.floydhub.com/attention-mechanism/#bahdanau-att-step2" rel="nofollow noopener">https://blog.floydhub.com/attention-mechanism/#bahdanau-att-step2</a><br>
In function “LuongDecoder-- def forward(self, inputs, hidden, encoder_outputs):”, what does “inputs” means.<br>
And if I want to do text classfication task, how can I produce “inputs”?</p>
How to construct single model multiple output

username: 111137
cooked: <p>Hi,</p>
<p>I try to make multi-classification without one-hot encoding (just an experiment).<br>
I do like as follows at output;</p>
<pre><code class="lang-auto">def BinRepresent(bins):
    b_size, a, b = shape(bins)
    out= np.zeros(b_size)
    for b in range(bsize):
        for i in range(int(np.ceil(np.log(NUM_CLASSES)))):
            out[b] = out[b] + 2**bins[b, 0, I]
    return out
</code></pre>
<p>Input of bins is “4-bit” (NUM_CLASSES==10) through fully-connected later with sigmoid aiming to make zero or one for every bit. The power of two constructs binary code of class number. The out is fed into loss function. I do test training, then its learning is very slow of course depending on setting of hyper parameters.</p>
<p>So, I reconsidered to develop output layer because of this approach affects to all 4-bits by the single loss value, it should be four losses for every bit. Then, I meet issue of how to construct such the multiple losses.<br>
Can anyone advice me about it?</p>
<p>Best,<br>
S.Takano</p>
How to do multi output regression with skorch/pytorch

username: Nidhal_Baccouri
cooked: <p>Hi, I’m trying to figure out how to do multi output regression with skorch or Pytorch. Actually I tried to do this with pytorch and somehow I managed to do this which worked and I got to see my loss decrease over epochs but I’m not sure if it is the best way. here is what I did:</p>
<pre><code class="lang-auto">def train(model, criterion, optimizer, loader, epochs):
    model.train()
    losses = []
    for e in range(epochs):
        epoch_loss = 0
        for x, y in loader:
            optimizer.zero_grad()
            y1, y2 = y[:, 0], y[:, 1]
            y_pred = model(x)
            y_pred1, y_pred2 = y_pred[:, 0], y_pred[:, 1]
            loss1 = criterion(y_pred1, y1)
            loss2 = criterion(y_pred2, y2)
            loss = (loss1 + loss2) * 0.5
            loss.backward()
            optimizer.step()
</code></pre>
<p>you can see that my model return two outputs and I’m calculating every loss separated and then taking the average. I did this because if I called the loss directly on the output and target it will take the average of the whole tensor and that’s bad somehow since I have two different losses for two different values. How I achieve this in pytorch or better in skorch since it is easier to use and to implement cross validation?</p>
<p>moreover can you explain to me how multi regression would work in pytorch? I mean for example if I’m predicting two values that one of them between 100-1000 and the other between 0-1, how would summing and taking the average of the losses work!?</p>
VAE KL divergence is not minimized

username: Rojin
cooked: <p>I  am trying to train a VAE in pytorch. I followed the following tutorial:<br>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/examples/blob/master/vae/main.py" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/examples/blob/master/vae/main.py" target="_blank" rel="nofollow noopener">pytorch/examples/blob/master/vae/main.py</a></h4>
<pre><code class="lang-py">from __future__ import print_function
import argparse
import torch
import torch.utils.data
from torch import nn, optim
from torch.nn import functional as F
from torchvision import datasets, transforms
from torchvision.utils import save_image


parser = argparse.ArgumentParser(description='VAE MNIST Example')
parser.add_argument('--batch-size', type=int, default=128, metavar='N',
                    help='input batch size for training (default: 128)')
parser.add_argument('--epochs', type=int, default=10, metavar='N',
                    help='number of epochs to train (default: 10)')
parser.add_argument('--no-cuda', action='store_true', default=False,
                    help='enables CUDA training')
parser.add_argument('--seed', type=int, default=1, metavar='S',
                    help='random seed (default: 1)')
parser.add_argument('--log-interval', type=int, default=10, metavar='N',
</code></pre>

  This file has been truncated. <a href="https://github.com/pytorch/examples/blob/master/vae/main.py" target="_blank" rel="nofollow noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
<br>
But I noticed that the KL actually increases instead of decreasing.</p>
<p>My impression was that KL needs to go down similar to BCE during the training. Am I missing something here?</p>
Visualizing Tensor

username: Kenzo
cooked: <p>Does anyone know why my code is visualizing feature map in grayscale when I’m using it during forward-pass? I want to visualize it in other colors</p>
<pre><code class="lang-auto">def vis_tensor(data):
    data  =data.data.cpu().numpy()
    data =data[0].transpose((1, 2, 0))
    data = data - data.min()
    data = data / data.max()
    data = data * 255
    mean = np.repeat(data.mean(-1, keepdims=True),3 ,2)
    mean = mean.astype('uint8')
    pylab.imshow(mean)
    pylab.savefig('mean.png')
    max = np.repeat(data.max(-1, keepdims=True), 3, 2)
    max = max.astype('uint8')
    pylab.imshow(max)
    pylab.savefig('max.png')
</code></pre>
<p>Output now is like this:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/d/6/d6aa8ffbb2774af1592d0f8fc64ea2ea9a320d17.png" alt="max" data-base62-sha1="uD1BDkBE28VdDHcCe2VWU0YniAf" width="640" height="480"></p>

username: albanD
cooked: <p>Because you run <code>mean</code> and <code>max</code> on the channel dimension? Since only one channel is left, it considers the image as being grayscale?</p>

username: albanD
cooked: <p>An image with a single channel is a grayscale image. So this is the expected visualization from my point of view.<br>
What were you expecting?</p>

username: ptrblck
cooked: <p>You could chose another colormap in <code>matplotlib</code> to map each grayscale value to a specific color.</p>
What is the mean of (most likely classes )?

username: hunar
cooked: <p>what is the mean of (most likely classes )?</p>
<p>top_p, top_class = ps.topk(1, dim=1)</p>
<h1>Look at the most likely classes for the first 10 examples</h1>
<p>print(top_class[:10,:])</p>

username: ptrblck
cooked: <p>I assume <code>ps</code> gives you the class probabilities (or logits), so <code>topk</code> will give you the highest class probabilities (logits) with the class indices for the current samples.<br>
Since you are using <code>.topk(k=1, dim=1)</code>, you’ll only get the single most likely prediction.</p>
Determinism in pytorch?

username: kirk86
cooked: <p>Do we have anything similar to <a href="https://www.youtube.com/watch?v=TB07_mUMt0U" rel="nofollow noopener">this</a> for determinism in PyTorch. Seems very interesting  and exciting approach to debug and discover sources of non-determinism in our pipelines and models?</p>

username: ptrblck
cooked: <p>I just skimmed through the presentation and it seems a few deterministic recipes are already implemented as described in the <a href="https://pytorch.org/docs/stable/notes/randomness.html">Reproducibility docs</a>, while we still have some undeterministic modules due to atomic ops.<br>
Are you looking for a specific feature presented in the talk or for general determinism?</p>

username: kirk86
cooked: <p>Thanks for the reply.</p>
<aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="61510">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>Are you looking for a specific feature presented in the talk or for general determinism?</p>
</blockquote>
</aside>
<p>Yes, somewhere in the talk he demonstrates how his tool allowed him to identify sources of determinism in his models. This is shown in the part where he compares hashes of different ops where red mean non-determinism introduced in that particular op or step.</p>
<p>Having sth like that would make debugging much better and allow for better understanding of the models.</p>
Are the C++ frontend optimizers supposed to work with script modules?

username: abtion
cooked: <p>I load the torchscript module with torch::jit::load, the module’s parameters doesn’t update while training, how can i train the torchscript module?<br>
here are some codes:</p>
<pre><code class="lang-auto">for(int epoch=0;epoch &lt; 3;++epoch){
        model.train(true);
        optimizer.zero_grad();
        for(int step=0;step&lt;nums;++step){
            std::vector&lt;torch::jit::IValue&gt; train_inputs1;
            train_inputs1.emplace_back(inputs_texts[step]);
            train_inputs1.emplace_back(inputs_pos_tags[step]);
            at::Tensor pos_scores = model.forward(train_inputs1).toTensor();

            std::vector&lt;torch::jit::IValue&gt; train_inputs2;
            train_inputs2.emplace_back(inputs_texts[step]);
            train_inputs2.emplace_back(inputs_neg_tags[step]);
            at::Tensor neg_scores = model.forward(train_inputs2).toTensor();

            at::Tensor loss = loss_func(pos_scores,neg_scores);
            loss.backward({},true,false);
//            loss.backward();
            optimizer.step();
            cout&lt;&lt;loss&lt;&lt;endl;
        }
        cout&lt;&lt;"epoch: "&lt;&lt;epoch&lt;&lt;"/3 completed!\n";
        model.save("./testsave.pt");
    }
</code></pre>
<p>the loss didn’t change for every epoch.</p>

username: albanD
cooked: <p>Hi,</p>
<p>This does not work currently I think. You can check this issue to track the progress on fixing this: <a href="https://github.com/pytorch/pytorch/issues/28478">https://github.com/pytorch/pytorch/issues/28478</a></p>