akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Accessing parameters and weights during training

steveo: <p>Hi All,</p>
<p>In reference to the topic title, is there a way to access the parameters and weights during training?<br>
The proposed solutions I’ve seen so far on the forum are the following:<br>
Using the .parameters() method on the model seems to return the parameters set during the initialization of the model, e.g.,<br>
for p in model.parameters():<br>
print(p.data)<br>
In addition, checking the individual layer of the model using model.Layer.weight.data has the same behavior.<br>
Both the above methods do not change as the models train.<br>
I would like to see the weights changing as a function of epoch/minibatch.</p>
<p>Thanks!!!</p>

harsha_g: <p>Did you take a look at using <code>state_dict</code>? More on that <a href="https://pytorch.org/tutorials/recipes/recipes/what_is_state_dict.html" rel="nofollow noopener">here</a>.</p>

steveo: <p>Yeah that works. It does require me to send the data in the state_dict to the cpu from the gpu if I want to append it to a list and not have it be a duplicate of the initialized parameter though. Thanks!</p>
Transfer Learning is really slow

Pigeon_Jole: <p>Hi guys,<br>
I am working on Traffic sign classification with the German Traffic Sign Dataset (~38k train and ~12k for test).<br>
I implemented a model from scratch and I want to compare it to the state of the art model.<br>
I followed the transfer learning tutorial from Pytorch tutos and tried the pre-trained models.<br>
I am freezing all layers, except the last one.<br>
However, the models (resnet, densenet, inception) are really slow during training (one epoch takes more than 30 mn on GPU).<br>
From my understanding, training these models should be fast (I am working on google colab GPU).<br>
Any help on how to train faster?</p>

JuanFMontesinos: <p>Well the word “slow” is relative. If you have a very big dataset, your epoch will require more iterations and (probably,depending on the variety of the data) less epochs to be trained, meanwhile if your dataset is small, you will have less iterations per epoch.</p>
<p>You can try other optimizers/LR etcetera to converge faster.</p>

Pigeon_Jole: <p>Ok, I see.<br>
Because most layers were frozen and just a couple being trained, I expected the pre-trained model to be faster than my “from scratch model”.<br>
Thanks for your answer. I will try LR scheduler.</p>

JuanFMontesinos: <p>Even if you are freezing layers, you still have to compute the full forward pass to get the signal of the non-frozen layers. You save some time since you don’t compute the full backpropagation but nothing else.</p>
<p>What you can do (having everything frozen), is to pre-compute the output of the last frozen layer (since that will be static epoch per epoch), but it will require lot of memory since there are no compression formats such us jpg developed for that.</p>

Satya_Veera: <p>Have you been able to solve the issue? As we are not computing backward pass when layers are freezed except the last layer shouldn’t it take less time when compared to training all layers?</p>
How to download Tensorboard model graph as .png

Udith_Haputhanthri: <p>Hello,<br>
I am new to PyTorch. I want to know what is the procedure to download a Tensorboard graph which is displayed on the browser interface (without getting screenshot). There is “download png” command on the Tensorboard interface but when I clicked it, it will tries to download a file called “png.html” and failed. There is no any images have been downloaded. Can please someone help?</p>

zcc: <p>I meet a same issue, have you fixed it?</p>

harsha_g: <p>I have  <code>tensorboard 2.2.1</code> and I did the following to download an svg image which I can later convert to any format I want.<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/b/8bc9b89fd934b52829a37816a47007d0f54ad6a8.png" data-download-href="https://discuss.pytorch.org/uploads/default/8bc9b89fd934b52829a37816a47007d0f54ad6a8" title="Screen Shot"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/b/8bc9b89fd934b52829a37816a47007d0f54ad6a8_2_690x456.png" alt="Screen Shot" data-base62-sha1="jWCylKSaqd5Yt9Dm5QbrQFTx3sc" width="690" height="456" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/b/8bc9b89fd934b52829a37816a47007d0f54ad6a8_2_690x456.png, https://discuss.pytorch.org/uploads/default/original/3X/8/b/8bc9b89fd934b52829a37816a47007d0f54ad6a8.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/8/b/8bc9b89fd934b52829a37816a47007d0f54ad6a8.png 2x" data-dominant-color="F3E7E1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot</span><span class="informations">703×465 38.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div>. I</p>
Errors when trying to run jit script

chinmay5: <p>I was trying to run this code section and with my limited understanding of jit, thought I can get a performance improvement on the function.</p>
<pre><code class="lang-auto">@torch.jit.script
def nms_pytorch2(dets, thresh):
    overlap = bbox_overlap(dets, dets)

    treshold_matrix = torch.tril((overlap &gt; thresh), diagonal=-1)

    # Tensor elements indicate whether box should be kept
    is_maximum = treshold_matrix.new_ones(dets.shape[0])

    # loop over all boxes with highest confidence in the scene
    # Apply this vectorized over all boxes in the batch.
    for box in treshold_matrix.unbind(-1):
        # Disable all other boxes in the same scene if the current box is not
        # disabled.
        is_maximum = is_maximum &amp; ~box

        # Also disable the overlaps of boxes which getting disabled right now.
        treshold_matrix &amp;= ~box.unsqueeze(-2)

    return is_maximum
</code></pre>
<p>However, I got the following error and am not sure how to handle it</p>
<pre><code class="lang-auto">torch.jit.frontend.NotSupportedError: unsupported kind of augumented assignment: BitAnd:
         # Also disable the overlaps of boxes which getting disabled right now.
        treshold_matrix &amp;= ~box.unsqueeze(-2)
                        ~~ &lt;--- HERE
    return is_maximum
</code></pre>
<p>Your help would be really appreciated</p>

chinmay5: <p>Just figured the issue out, the operations need to be non in-place. Just putting a temp variable and assigning value of this temp variable in the next line solved the issue for me.</p>
How do I use nn.TransformerEncoder to make a classifier?

jrmiranda: <p>Hey guys, I’m currently studying transformer from Attention is All You Need and I’m trying to implement a simple text classifier by using the <code>nn.TransformerEncoder</code> module and I have a question:</p>
<p>Considering that I have a batch size of <code>batch_size</code>, a sequence length of <code>seq_len</code> and an embedding size of <code>emb_size</code>, the output of the transformer encoder has the shape <code>(batch_size, seq_len, emb_size)</code>. How can I send this output to a simple <code>nn.Linear</code> module? Should I concatenate the values or do some kind of average or just pick the vector corresponding to the last word? Thanks!</p>

andcarnivorous: <p>Have you checked the <a href="https://pytorch.org/tutorials/beginner/transformer_tutorial.html" rel="nofollow noopener">PyTorch</a> tutorial on the Transformer models? You can see the last layer is a linear layer with output size of the number of tokens, because it is a LM model, in your case the output will be the text class of the input. I would just base your work on that guide as it is really nicely written.</p>

jrmiranda: <p>Thanks for your reply! I’m following this tutorial but I always get an output size of <code>[batch_size, seq_len, output_dim]</code> where <code>output_dim</code> is the output of my fully connected layer at the end of the transformer encoder. What is the best approach to make the predictions in the shape <code>[batch_size, output_dim]</code>?</p>

andcarnivorous: <p>I think you could maybe reshape the vector by multiplying the seq_len*output_dim <code>reshape(batch_size, seq_len*output_dim)</code> before you pass it to the linear layer. I do not know if it is the most appropriate way to go for it, also looking at how they deal with this in the huggingface transformer library might help.<br>
If you try this method please let know about the results <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Gradient becomes 0 for 1D convolutions

thecho7: <p>I’m trying to implement the attention module with 1D convolution.<br>
Here’s my custom attention module for that.</p>
<pre><code>class convSelfAttention(nn.Module):
    def __init__(self, in_dim):
        super(convSelfAttention, self).__init__()
        self.channel_in = in_dim
        self.query_conv = nn.Conv1d(in_channels=in_dim, out_channels=in_dim//2, kernel_size=1)
        self.key_conv = nn.Conv1d(in_channels=in_dim, out_channels=in_dim//2, kernel_size=1)
        self.value_conv = nn.Conv1d(in_channels=in_dim, out_channels=in_dim, kernel_size=1)
        self.gamma = nn.Parameter(torch.zeros(1))
        self.activ = nn.Softmax(dim=-1)

    def forward(self, x):
        p_query = self.query_conv(x).permute(0, 2, 1)
        p_key = self.key_conv(x)
        energy = torch.bmm(p_query, p_key)
        attention = self.activ(energy)
        p_value = self.value_conv(x)

        out = torch.bmm(p_value, attention.permute(0, 2, 1))
        out = self.gamma * out + x
        out = out.permute(0, 2, 1)
        return out
</code></pre>
<p>However, the gradients are 0 except ‘self.gamma’.<br>
For your convenience, I plot the gradient flow of my entire architecture.(The last two layers are fc and all layers from ‘~~Encoder’ are convolutions)<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/3/c/3cd5e2da47196df70c169c821aab3ade430f093a.png" alt="Figure_2" data-base62-sha1="8GaTCJYDdrvc2mADhNjuysoEFqa" width="640" height="479"></p>
PyTorch equivalent for tf.sequence_mask

Cognac: <p>I’m dealing with variable-length sequences and I need to apply the mask on a bunch of different tensors. Is there a PyTorch API that provides the same functionality for <a href="https://www.tensorflow.org/api_docs/python/tf/sequence_mask" rel="nofollow noopener">tf.sequence_mask</a>?</p>
<p>Thanks a lot!</p>

DimitrisNightmare: <pre><code class="lang-auto">def sequence_mask(lengths, maxlen, dtype=torch.bool):
    if maxlen is None:
        maxlen = lengths.max()
    mask = ~(torch.ones((len(lengths), maxlen)).cumsum(dim=1).t() &gt; lengths).t()
    mask.type(dtype)
    return mask
</code></pre>
<p>Would this work for you? torch.bool is only available in 1.2 version. If you use something older, try a good old int type.</p>

mp5088643: <p>you can us torch.masked_select</p>

SaviourZ: <p>This implement will have different output compared with <code>tf.sequence_mask</code> when the input is <code>2d-dimension</code>.</p>
<p>e.g When <code>lengths</code> is <code>array([[1],[3], [2],[4]])</code>.</p>
<p>So I improved this implement as following:</p>
<pre><code class="lang-python">    def sequence_mask(self, lengths, maxlen=None, dtype=torch.bool):
        if maxlen is None:
            maxlen = lengths.max()
        row_vector = torch.arange(0, maxlen, 1)
        matrix = torch.unsqueeze(lengths, dim=-1)
        mask = row_vector &lt; matrix

        mask.type(dtype)
        return mask
</code></pre>
<p>Hope this can help those who want to use <code>tf.sequence_mask</code> in PyTorch also.</p>

hccho2: <p>mask = torch.arange(maxlen)[None, :] &lt; lengths[:, None]</p>
Pairwise equality + reduction

Andreas_Georgiou: <p>I am trying to compute pairwise equality of all elements of 2 1D tensors. Then I want to know how many times each element of y occurs in x. Right now I am using the following:</p>
<pre><code class="lang-auto">x = x.view(-1,1)
y = y.view(1,-1)
(x==y).sum(dim=0)
</code></pre>
<p>Now this works fine but the intermediate x==y is quite large and it will not fit into the GPU for large x and y. Is there a way to bypass creating the x==y matrix given that I only care about the final reduced sum?</p>
<p>Any help would be appreciated.</p>

ptrblck: <p>You could trade memory for compute by using loops and perform the comparison without the broadcasting for the sub-elements (including the sum). I don’t know, of there would be another better way “between” these approaches.</p>
List layers/nodes executed in order of forward pass

Stihanssen: <p>I am trying to implement a CNN <a href="https://arxiv.org/pdf/1708.06670.pdf" rel="nofollow noopener">visualization technique</a>. I essentially need a list of operations executed in the order of the forward pass, that the algorithm can work through backwards.</p>
<p>Due to the nature of the algorithm, operations such as pooling and reshaping must also be registered.</p>
<p>I could maybe have the user manually make a list of operations, but it isn’t ideal as I also use forward hooks to grab data between the modules.</p>
<p>I found a <a href="https://discuss.pytorch.org/t/iterate-through-layers-in-the-same-order-as-the-forward-pass/24715">topic</a> that said I could maybe use the jit tracer, but I couldn’t see how that could be done from the documentation.</p>
<p>As I am trying to visualize information about the network, I don’t mind doing a trace for each run through the forward pass, as I would need to run the visualization algorithm each time anyways.</p>
<p>I understand there might not be a pretty way to do it, but I would appreciate any idea that could potentially work!</p>

Stihanssen: <p>Update for those wanting to do something similar to me.</p>
<p>I found kind of a workaround. I registered a forward hook to all “leaf” modules in the model, that is modules that do not have submodules. The hook/function creates a storage object with information (module, in data, out data), this object is put into a global list.</p>
<p>The list if updated in order of the forward pass. Once a forward pass is complete, I have a list of storage objects in order of the forward pass with all the info I need.</p>
<p>There are a few clear weaknesses however, this method can only register hooks on modules that have been initialized. If the model uses i.e. torch.nn.functional.max_pool2d or any other method from torch.nn.functional in the forward method of the model, they won’t show up in model.modules() and won’t get a forward hook.<br>
Secondly, the global list may cause several problems: Unsure how it will work in a parallel setup (multiple GPUs) or even just networks that have branching such as skip connections.</p>

surag: <p>Could you get this to work? As you mention, it won’t register non module calls. The Jit tracer does give code that would still require further parsing to be useful (<a href="https://discuss.pytorch.org/t/how-to-parse-torch-jit-trace-output/57657" class="inline-onebox">How to parse torch jit trace output?</a>).</p>
Passing halve of the weights in model.fc.weight of resnet model to Adam optimizer

rahmanoladi: <p>Hi everyone,<br>
Please I would like to pass halve of the parameters in model.fc.weight of resnet model to Adam optimizer. To be specific, suppose the weights stored in model.fc.weight  is 1000 by 512, then I want to pass halve of the weights, which would be a 500 by 512 matrix, to Adam optimizer. I proceeded as follows:</p>
<p>w_0 = model.fc.weight[0: 501, :]<br>
optimizer = torch.optim.Adam([w_0], lr = 0.001)<br>
optimizer.step()</p>
<p>I get the following error:<br>
ValueError: can’t optimize a non-leaf Tensor</p>
<p>Then I try:</p>
<p>w_0 = model.fc.weight[0: 501, :].retain_grad()<br>
optimizer = torch.optim.Adam([w_0], lr = 0.001)<br>
optimizer.step()</p>
<p>I get the following error:<br>
TypeError: optimizer can only optimize Tensors, but one of the params is NoneType</p>

surya00060: <p>Hi<br>
If I am understanding correctly you want to pass only 50% of weights to the optimizer and not to touch the remaining 50%.</p>
<ol>
<li>
<p>You can’t use <code>requires_grad =True</code> as they work only at the level of tensors.</p>
</li>
<li>
<p>You can achieve this by masking 50% of your weights so that they don’t get updated.</p>
</li>
</ol>
<p>Create a new tensor operation in the computational graph</p>
<pre><code class="lang-auto">mask = torch.zeros(weight.shape, requires_grad = False)
mask[0:501,:] = 1
newWeight = weight * mask
</code></pre>
<p>This small stub shows how to mask neurons and train. I hope you find this useful. I can explain better if more details(ie. Code) is given.</p>
<p>Thanks</p>

ptrblck: <p>Small addition:<br>
I would suggest to mask the gradients instead of the weights directly, so that they won’t be updated.<br>
However, I think weight decay might still update these parameters even if the gradient is zero.</p>

rahmanoladi: <p>Hi Ptrblck,<br>
Thanks for your response. I have realized that it is well worth it to learn Pytorch’s autograd inside out and bottom up. I also realized that you know the autograd engine very well. Please kindly point me to a book or tutorial that explains the autograd engine in great detail in such a way that one understands it and can predict its behavior, and then can avoid several of its pitfalls such as: “RuntimeError: leaf variable has been moved into the graph interior” and a whole host of other errors. Thanks a lot</p>

ptrblck: <p>Hi Abdulrahman,</p>
<p>I’m not a real expert on the autograd engine, but I think the <a href="https://pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html">docs</a> would be a good starter as well as <a class="mention" href="/u/ezyang">@ezyang</a>’s <a href="http://blog.ezyang.com/2019/05/pytorch-internals/">slides</a> on PyTorch internals. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

rahmanoladi: <p>Hi Ptrblck,<br>
Many thanks for the response.</p>
How to parse torch jit trace output?

Arun_Vishwanathan: <p>For my use case, I realize that I need to programmatically derive for each layer, the settings of the layer (like stride, kernel size , etc for a convolution for example) as well as the number of input and output tensors to the layer so as to build up a graph type representation of the network. I tried a simple conv2d model and it outputs the following trace code using</p>
<pre><code class="lang-auto">traced_cell = torch.jit.trace(net, (inputseq,), check_trace=False)
print(traced_cell.code)
</code></pre>
<pre><code class="lang-auto">def forward(self,
    input: Tensor) -&gt; Tensor:
  _0 = getattr(self, "0")
  weight = _0.weight
  _1 = torch._convolution(input, weight, _0.bias, [1, 1], [0, 0], [1, 1], False, [0, 0], 1, False, False, True)
  return _1
</code></pre>
<p>this is a string output so parsing here is too tedious. Is there a way from the <code>traced_cell.graph</code> to derive this information about the layers present as well as the tensors to and from a layer so as to make semantic connections (say I dump information in a JSON file represent graph with layers and tensors) between nodes at my end? I need to only know the number of tensors and the name to a layer so that I can form a connection between layers in the network</p>
<p>In the above example, <code>input</code> is the input to forward as well as is used as the input to the conv function. this kind of information is what I need. Also the output of conv is indeed the output of the model as well so _1 is the model output. As well I need to know that this convolution uses kernel size of 1, etc.</p>
<p>Any suggestions please?</p>

surag: <p>Any progress on this? There should be a way to get a graph in terms of modules and operations.</p>
Implementing layer C3 in LeNet-5

Wesley_Neill: <p>So I just finished a small tutorial on CNN’s, which I feel like really upped my game. I’m excited to try implementing different architectures on the MNIST and CIFAR-10 datasets.</p>
<p>I’m reading <a href="https://engmrk.com/lenet-5-a-classic-cnn-architecture/" rel="noopener nofollow ugc">this article</a> on LeNet-5, and it says</p>
<blockquote>
<p><em>Next, there is a second convolutional layer with 16 feature maps having size 5×5 and a stride of 1. In this layer, only 10 out of 16 feature maps are connected to 6 feature maps of the previous layer as shown below.</em></p>
</blockquote>
<p>With the accompanying image.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/7/178ce988c6447f57ba264bf4c80284971d96e626.png" alt="image" data-base62-sha1="3mkSRAD26MowhbxXWcIbPHIdD5s" width="447" height="232"></p>
<p>I understand the table conceptually, but I’m not sure how to implement it. How does one “tell” a convolutional layer which inputs to accept from the last layer?</p>
<p>Ther is a Keras implementation at the end of the article, but it appears C3 is fully connected to the previous layer:</p>
<pre><code class="lang-auto"># C3 Convolutional Layer
model.add(layers.Conv2D(16, kernel_size=(5, 5), strides=(1, 1), activation=’tanh’, padding=’valid’))
</code></pre>
<p>Did author ignore the table?</p>
<p>How can I implement the table in PyTorch?</p>
<p>Thanks!</p>
Pytorch Tutorial Error. What's the different between the following codes?

wulongjian: <p>Hello, I have learn the tensorboard with <a href="https://pytorch.org/tutorials/intermediate/tensorboard_tutorial.html" rel="nofollow noopener">tensorboard_tutorial</a>,there is a class Net, the official code as following:</p>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(1, 6, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1 = nn.Linear(16 * 4 * 4, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = x.view(-1, 16 * 4 * 4)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x
net = Net()
</code></pre>
<p>well, it’ work well when I run to the code<code>writer.add_graph(net, images)</code>, I recently see another way to implement it, my personal code as following</p>
<pre><code class="lang-auto">    def __init__(self):
        super(Net, self).__init__()
        self.feature1 = nn.Sequential(
            nn.Conv2d(1,6,5),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(2,2),
            nn.Conv2d(6,16,5),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(2,2)
        )
        self.feature2 = nn.Sequential(
            nn.Linear(256, 120),
            nn.ReLU(inplace=True),
            nn.Linear(120,84),
            nn.ReLU(inplace=True),
            nn.Linear(84,10)
        )
    def forward(self, x):
        x  = self.feature1(x)
        x.view(-1, 256)
        x = self.feature2(x)
        return x
</code></pre>
<p>I use the <code>nn.Sequential</code>,when I run to the code <code>writer.add_graph(net, images)</code>, it report Error as follows</p>
<pre><code class="lang-auto">RuntimeError: size mismatch, m1: [256 x 4], m2: [256 x 120] at C:\w\1\s\tmp_conda_3.7_055457\conda\conda-bld\pytorch_1565416617654\work\aten\src\TH/generic/THTensorMath.cpp:752

Process finished with exit code 1
</code></pre>
<p>So what is the different between them?<br>
Thanks for you answer my question~~~</p>

harsha_g: <p>You are missing the assignment <code>x = x.view(-1, 256)</code></p>

wulongjian: <p>thanks,I’m not notice that. <img src="https://discuss.pytorch.org/images/emoji/apple/joy.png?v=9" title=":joy:" class="emoji" alt=":joy:"></p>
How to implement a forward pass of identity matrix?

Huimin_ZENG: <p>Hello everybody,</p>
<p>can someone please tell me if we can compute forward pass of an identity matrix in CNN?</p>
<p>For instance, if we use fully connected layers only, we can definitely feed an identity matrix to the network and get some results.</p>
<p>However, what if we use convolutional layers? We know convolutions are just linear operations, so there must be some equivalent fully connected layers, which can replace the convulutional layers.</p>
<p>Therefore, can we find an input, so that CNN is “equivalently” computing the forward pass of an identity matrix?</p>
<p>thank you!</p>
<p>best</p>
Pytorch tensor dimensions - replace and multiply operations

Aiman_Mutasem-bellh: <p>Dear friends <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>I have a tensor with the dimensions of x  = torch.Size([128, 99, 8, 31]).<br>
I need to multiply the second position ‘channels’ with itself to get x  = torch.Size([128, (99 * 99 ), 8, 31])  =  torch.Size([128, 9801, 8, 31]).<br>
Is this logically accepted, and how to do it?</p>
<p>Also</p>
<p>I have a tensor with the dimnsions of y = torch.Size([128, 1,  99,  99]).<br>
I need to replace the third position dimension (99) with (1), to get y = torch.Size([128, 1,  1,  99]). how I can do that?</p>
<p>I have tried tensor.view and tensor.reshape but it didn’t work with me as I expect.</p>

tom: <p>You likely what to think about what this should do with the values in the tensors.<br>
For the first, there is repeat and tile which might work, the second could be achieved with some reduction (sum or mean with keepdims=True). But you really want to get the values right, too, not just the shape.</p>

Aiman_Mutasem-bellh: <p>Thank you Sir. for your replay.<br>
I’m sorry if my question looked ambiguous. Actually, I’m looking for the shape, not values.</p>

Aiman_Mutasem-bellh: <p>Pytorch experts, any suggestion to work it on <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

ptrblck: <p>As <a class="mention" href="/u/tom">@tom</a> explained, there is not a single correct answer and it depends on what you are trying to achieve.<br>
E.g. the second example would need a reduction (reduce the shape of 99 to 1), which can be accomplished using <code>torch.mean</code>, <code>torch.sum</code>, etc. or simply by indexing.<br>
So the more interesting question is: what kind of operation should be used (or what are you trying yo achieve)?</p>
ImportError: No module named 'torch'

Ajay_Madhavan: <p>I have installed torch in a virtual environment in a cluster server. By still import torch throws “ImportError: No module named ‘torch’”.  How do I fix this issue?</p>

ptrblck: <p>Make sure that you are running the script from the right environment (the conda env name should be shown in the terminal).<br>
You could also check, if <code>torch</code> is found via <code>conda list | grep torch</code>, which should output the install <code>torch</code> version.</p>

Ajay_Madhavan: <p>Hay,</p>
<p>Thank you for the reply. I was able to fix the issue!! I was using different environment, while importing torch.</p>
<p>Thank you for the reply!!</p>
Mixed precision training

ruoshiliu: <p>Hi all,</p>
<p>I’m training a model in double precision. However, I notice a significant slowdown compare to single precision counterpart. Therefore, I would like to train part of the model with single precision and the other part in double precision. In my case, the vision backbone (ResNet-34) could be trained with single precision while the rest of the model need to be trained in double precision.</p>
<p>So I’m wondering if there’s any viable way to implement such functionality? Will such mixed precision training effectively improve training speed?</p>
<p>(I know there’s Apex mixed precision training to train part of model in half-precision. Would that framework also work for double precision?)</p>
Do a train vaidation split at folder level in pytorch

Luca_Pamparana: <p>Currently, my data is organized as follows:</p>
<pre><code class="lang-auto">  - root
       -  class1
            -  img1.png
            -  img2.png
            - imgN.png
       - class2
            - img1.png
            .....
           - imgN.png
      - classN
            .....
</code></pre>
<p>Now I can load the whole dataset as:</p>
<pre><code class="lang-auto">import torchvision.datasets as dset
train_folder_dataset = dset.ImageFolder(root=self.data_path)
</code></pre>
<p>This is great as I can now use this in my dataset and read all image files into my training dataset. Now, I want to split this dataset into training and validatioon but at the folder level (example I want the <code>class1</code> folder to be in the training set and <code>class2</code> in the validation set as an example).</p>
<p>However, I have no idea how to split this from thee <code>ImageFoder</code> class. So, if I do something like:</p>
<pre><code class="lang-auto">all_dirs = dset.ImageFolder(root=self.data_path)
# Here how do I split this all_dirs into training and validation directories. Note, I want to split at the directory level and not at the image level.
#  So hoping for something like:
train_dirs, val_dirs  = spliit_folder(all_dirs)
</code></pre>
<p>Would something like this be possible?</p>

turian: <aside class="quote no-group" data-username="Luca_Pamparana" data-post="1" data-topic="87060">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/luca_pamparana/48/14310_2.png" class="avatar"> Luca_Pamparana:</div>
<blockquote>
<pre><code class="lang-auto">train_folder_dataset = dset.ImageFolder(root=self.data_path)
</code></pre>
</blockquote>
</aside>
<p>I’m doing something similar, but without the dset.ImageFolder class (because I don’t have images).</p>
<p>What I did was preprocess into:</p>
<pre><code class="lang-auto">data/train/
data/val/
data/test/
</code></pre>
<p>at the beginning of my pipeline, so all nets would have the same view on what is train/val/test</p>
Globally set the device

dfalbel: <p>Is there a way to globally set the default device with libtorch?<br>
I know there exists device guard implemented here:</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/blob/8482efb2036475072f26c07e4d7d91e14e5e2c20/c10/core/DeviceGuard.h" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/8482efb2036475072f26c07e4d7d91e14e5e2c20/c10/core/DeviceGuard.h" target="_blank" rel="nofollow noopener">pytorch/pytorch/blob/8482efb2036475072f26c07e4d7d91e14e5e2c20/c10/core/DeviceGuard.h</a></h4>
<pre><code class="lang-h">#pragma once

#include &lt;c10/core/impl/InlineDeviceGuard.h&gt;

namespace c10 {

/// RAII guard that sets a certain default device in its constructor, and
/// changes it back to the device that was originally active upon destruction.
///
/// The device is always reset to the one that was active at the time of
/// construction of the guard. Even if you `set_device` after construction, the
/// destructor will still reset the device to the one that was active at
/// construction time.
///
/// This device guard does NOT have an uninitialized state; it is guaranteed
/// to reset a device on exit.  If you are in a situation where you *might*
/// want to setup a guard (i.e., are looking for the moral equivalent
/// of optional&lt;DeviceGuard&gt;), see OptionalDeviceGuard.
class DeviceGuard {
public:
</code></pre>

  This file has been truncated. <a href="https://github.com/pytorch/pytorch/blob/8482efb2036475072f26c07e4d7d91e14e5e2c20/c10/core/DeviceGuard.h" target="_blank" rel="nofollow noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>But it won’t set it globally and it’s harder to program with when the user can dynamically decide to set it or not.</p>
<p>I wonder if there’s something equivalent to <code> torch::autograd::GradMode::set_enabled(enabled)</code> that we can use to set the grad mode globally.</p>
Tensor to C++ vector

Hiperdyne19012: <p>Is there a way to turn a 2d libtorch torch::Tensor into vector&lt; vector &gt; ?</p>

ericrhenry: <p>This may not be sufficiently general, because it really relies on contiguous data storage: If you know the data type and sizes of the tensor, and the origin (available through data_ptr()), it’s straightforward to step through memory and build up row vectors, and then vectors of these as column vectors. (I might have it backward, but I’ve been able to get such a procedure to work for me.)</p>
Create dataLoader with costume size labels

afshin67: <p>I have a dataset with labels of size <code>64</code>, and the inputs are images of size <code>15*17</code> in <code>3</code> channel. So, the feature (images) and labels are of size <code>n*3*15*17</code> and <code>n*64</code>. I want to create a pytorch dataLoader to get mini-batch of data. I could do this  by expanding the dimension of labels to <code>n*3*15*64</code> and concatenate it with the feature (images) to get batches and then separate before passing them into the model. But, I think it is not a good idea since when <code>n</code> is quite big, this can be quite expensive in terms of memory usage, and also involves some unnecessary computation.</p>
<p>I appreciate any suggestion?</p>
<p>And, here is an example:</p>
<pre><code>import torch import numpy as np 

n = 10 
a = np.ones((n,3,15,17))
b = np.ones((n,64))

data = torch.utils.data.DataLoader(train_data, 5, shuffle = True)
</code></pre>
<p>in which I am not sure how to construct <code>train_data</code> out of <code>a</code> and <code>b</code>.</p>

Antonio_Ossa: <p>Hi <a class="mention" href="/u/afshin67">@afshin67</a>,</p>
<p>If I understand your question corectly, a possible solution is to create a <code>CustomDataset</code> that returns a tuple with label and data or a dictionary containing both, a common approach that is shown in <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html" rel="nofollow noopener">this tutorial</a>. A small example I crafted in case you find it useful:</p>
<pre><code class="lang-auto">class CustomDataset(Dataset):

    def __init__(self, a, b):
    	self.a = a
    	self.b = b

    def __len__(self):
        return len(self.a)

    def __getitem__(self, idx):
    	return self.a[idx], self.b[idx]
        # return {"a": self.a[idx], "b": self.b[idx]}

train_data = CustomDataset(a, b)
data = torch.utils.data.DataLoader(train_data, 5, shuffle=True)
</code></pre>
<p>Hope it helps!</p>

afshin67: <p>That is exactly what I was looking for.<br>
Thanks for the response.</p>
Resume DDP training after out of memory error

Scott_Hoang: <p>I am training a NAS network that would sometimes be out of memory on the GPUs due to different predicted model configurations on each run.</p>
<pre><code class="lang-auto">def check_oom(func):
    def wrapper(*args, **kwargs):
            try:
                  return func(*arg, **kwargs)
            except RuntimeError:
                  torch.cuda.empty_cache()
                  return None
   return wrapper

# model's forward function
@check_oom
def forward(model, input):
       return model(input)

# main loop
def main():
      dataset = Dataloader(...)
      net = Model(...)
      for input in dataset:
             output = forward(net , input)
             (....... training code)
     torch.cuda.synchronize()

</code></pre>
<p>above is the pseudo-code I use for my training. However, in practice, OOM events will hang the entire training with GPUs 's maxed out at 100% utility rate.<br>
What can I do ?</p>

mrshenli: <p>Hey <a class="mention" href="/u/scott_hoang">@Scott_Hoang</a>, yes, this is the expected behavior, as OOM in one of the process will lead to AllReduce communication desync across the entire DDP gang.  <a href="https://pytorch.org/elastic" rel="nofollow noopener">https://pytorch.org/elastic</a> is  built to solve this problem. It will destruct all DDP instances across all processes, reconstruct a new gang, and then recover from the previous checkpoint.</p>
<p>cc <a class="mention" href="/u/kiuk_chung">@Kiuk_Chung</a></p>

Scott_Hoang: <p>This is exactly what I am looking for. Thank!</p>
How to debug custom autograd.Function?

pyscorcher: <p>Apparently the <code>backward()</code> in a custom <code>autograd.Function</code> does not get called by a python method directly (see also this blog entry <a href="https://www.janfreyberg.com/blog/2019-04-01-testing-pytorch-functions/" rel="nofollow noopener">regarding coverage tests</a>), which is why breakpoints set within these <code>backward()</code> methods never get triggered. Is there another way to use a python debugger for these backward passes?</p>

albanD: <p>Hi,</p>
<p>What I usually do when I need this is to trigger it directly from inside the backward function with <code>    import pdb; pdb.set_trace()</code>:</p>
<pre><code class="lang-python">import torch
from torch import autograd

class F(autograd.Function):
  @staticmethod
  def forward(ctx, foo):
    return foo.clone()

  @staticmethod
  def backward(ctx, bar):
    import pdb; pdb.set_trace()
    return bar.clone()

a = torch.tensor([3.2], requires_grad=True)
b = F.apply(a)
g = autograd.grad(b, a)[0]



</code></pre>

pyscorcher: <p>Thank you very much, I forgot about that solution, it is simple and effective!</p>
<p>By the way: In case you <a class="mention" href="/u/alband">@albanD</a> haven’t seen it yet, you can shorten that to <code>breakpoint()</code> which is a built-in since <a href="https://docs.python.org/3.7/library/functions.html#breakpoint" rel="nofollow noopener">python 3.7</a> - saves a little bit of typing:)</p>

albanD: <p>I did not knew that <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:">  Thanks for the tip!</p>
Expected input batch_size (1) to match target batch_size (10)

Rushank_Savant: <p>class Network(nn.Module):<br>
def <strong>init</strong>(self):<br>
super().<strong>init</strong>()<br>
self.conv1 = nn.Conv2d(in_channels=1, out_channels=8, kernel_size=5) # Convolution layer 1<br>
self.conv2 = nn.Conv2d(in_channels=8, out_channels=12, kernel_size=5) # Convolution layer 2</p>
<pre><code>    self.fc1 = nn.Linear(in_features=12 * 100 * 272, out_features=120) # Linear layer 1 [Linear layer also called Fully Connected Layers]
    self.fc2 = nn.Linear(in_features=120, out_features=60) # Linear layer 2
    self.out = nn.Linear(in_features=60, out_features=3) # Linear layer 3 (output layer)
    
def forward(self, t):
    # input layer
    t=t
    # (2) hidden conv layer
    t = self.conv1(t)
    t = F.relu(t)
    t = F.max_pool2d(t, kernel_size=2, stride=2)

    # (3) hidden conv layer
    t = self.conv2(t)
    t = F.relu(t)
    t = F.max_pool2d(t, kernel_size=2, stride=2)
    
    # (4) hidden linear layer
    t = t.reshape(-1, 12 * 100 * 272)
    t = self.fc1(t)
    t = F.relu(t)

    # (5) hidden linear layer
    t = self.fc2(t)
    t = F.relu(t)
    
    # (6) output layer
    t = self.out(t)
    #t = F.softmax(t, dim=1)
    
    return t
</code></pre>
<p>network = Network()</p>
<p>train_loader = torch.utils.data.DataLoader(train, batch_size=10)<br>
optimizer = optim.Adam(network.parameters(), lr=0.01)</p>
<p>for epoch in range(8):</p>
<pre><code>total_loss = 0
total_correct = 0

for batch in train_loader: # Get Batch
    images, labels = batch 

    preds = network(images) # Pass Batch
    loss = F.cross_entropy(preds, labels) # Calculate Loss

    optimizer.zero_grad()
    loss.backward() # Calculate Gradients
    optimizer.step() # Update Weights

    total_loss += loss.item()
    total_correct += get_num_correct(preds, labels)

print("epoch", epoch, "total_correct:", total_correct, "loss:", total_loss)</code></pre>

albanD: <p>Hi,</p>
<p>Do you have a particular question you want to ask?<br>
The error seems quite clear? I guess the batch size of preds and labels don’t match?</p>
Activation function in nn.Sequential

shashanksrikanth: <p>Discrepancy between using <code>nn.ReLU</code> directly in an <code>nn.Sequential</code> block vs defining the activation function in the <code>__init__</code> function and then applying it to the <code>nn.Sequential</code> block.</p>
<p>I am using the <code>torch.nn.Sequential</code> to define my network as follows:</p>
<pre><code class="lang-auto">class ResidualBlock(nn.Module):
    def __init__(self, in_channels, out_channels):
        super(ResidualBlock, self).__init__()
        self.activation = nn.ReLU()
        self.block = nn.Sequential(
            self.get_layer(in_channels, out_channels),
            nn.Conv2d(out_channels, 2 * out_channels, kernel_size=1),
            nn.BatchNorm2d(2 * out_channels)
        )

    def get_layer(self, in_channels, out_channels, kernel_size=1, padding=0):
        return nn.Sequential(
            nn.Conv2d(in_channels, out_channels, kernel_size, padding=padding),
            nn.BatchNorm2d(out_channels),
            self.activation    
        )

    def forward(self, x):
        return self.block(x)
</code></pre>
<p>On using <code>torchsummary</code> <a href="https://github.com/sksq96/pytorch-summary" rel="nofollow noopener">package</a> however, I get multiple ReLUs in my forward pass as shown below:</p>
<pre><code class="lang-auto">----------------------------------------------------------------
        Layer (type)               Output Shape         Param #
================================================================
            Conv2d-1           [-1, 32, 32, 32]           2,080
       BatchNorm2d-2           [-1, 32, 32, 32]              64
              ReLU-3           [-1, 32, 32, 32]               0
              ReLU-4           [-1, 32, 32, 32]               0
            Conv2d-5           [-1, 64, 32, 32]           2,112
       BatchNorm2d-6           [-1, 64, 32, 32]             128
================================================================

</code></pre>
<p>However, this issue does not arise when I replace <code>self.activation</code> in the <code>nn.Sequential</code> with <code>nn.ReLU</code> directly. Is my understanding of <code>nn.Sequential</code> wrong?</p>

kuonlp: <p>What if self.activation = something else? This might be an issue of torchsummary. As it belongs to the same block, I would try to print the content of that block (not with torchsummary).</p>

shashanksrikanth: <p>Seems like an issue with torchsummary. Printing the blocks gives the expected output</p>
Matrix slicing makes GPU training slow?

turian: <p>I have a pytorch lightning model on tabular data, a mix of categorical and continuous.</p>
<p>On Google Colab, switching from CPU to GPU only increases speed by 2x. (batch_size=1024 so it’s not small batches)</p>
<p>I suspect this is because I am doing a lot of matrix slicing, which is reducing locality. Here is a snippet from my forward method:</p>
<pre><code class="lang-auto">    def forward(self, x):
        xcat = []
        for i, (start, end) in enumerate(self.cat_ranges):
            xcat.append(self.embeddings[i](x[:,start:end]))
        xcat = torch.cat(xcat, 1)
        xcat = self.emb_dropout_layer(xcat)

        xcont = x[:,self.cont_range[0]:self.cont_range[1]]
</code></pre>
<p>Is this sort of slicing the reason I can’t speed up a lot on GPU? If so, how do I resolve it? Pre-slice the matrices so they are not views?</p>

turian: <p>I also notice that the datasets are on the CPU. How can I preload them all onto the GPU for faster training? They will fit in GPU memory.</p>
<pre><code class="lang-auto">for i, (x, y, d) in enumerate(model.train_dataloader()):
    print(x.device)
    print(d.device)
    break
</code></pre>
<p>cpu<br>
cpu</p>

ptrblck: <p>If you don’t need to preprocess the data, you could directly push the tensors to the GPU inside your <code>Dataset</code> or you could create the batch directly from the data via indexing.</p>
<p>The loops might slow down your code. Getting rid of it depends of course on your use case and I don’t know, how <code>embeddings</code> is defined and if you could use a single layer for it.</p>

turian: <p>I am pushing all the data onto the GPU.</p>
<p>I am using indexing (views) on the matrix.</p>
<p>I cannot get rid of the for-loops because I have different sized one-hot categoricals. (I explain in another post why I need one-hot and not Longs.) One might have 10 categories, one might have 20, so they each have different embedding matrices.</p>
<p>I am concerned that with indexes so small, that data locality will be lost and that I should pre-splice the matrix by categorical so they each are in their own matrix.</p>
What is the use of map_location in torch.load()?

sahiluppal2k: <p>while loading existing weights, we use torch.load and it have an parameter named map_location. What does this parameter does?<br>
i’m new to pytorch so pardon if this is new newbie question.</p>

harsha_g: <p>Please go through this link <a href="https://pytorch.org/docs/master/generated/torch.load.html" rel="nofollow noopener">https://pytorch.org/docs/master/generated/torch.load.html</a> and see if the examples are helpful.</p>
Visulaizing 3D images

887574002: <p>Hi,</p>
<p>I have a dataset of 3D medical images with .npy extensions. I have already downloaded the dataset using<br>
np.load(filename.py). So, each image is a 4D NumPy array (channel, height, width, depth). I am interested to visualize some images to see how the augmentations would work on them or at least a 2D slice of them that I would be sure images are downloaded properly.  Since the images are grayscales the color can be ignored and the arrays can be seen as 3D. So, how can I visualize the volumes.<br>
Any help is really appreciated.</p>

ptrblck: <p>You could visualize a single slice from the depth dimension e.g. using <code>matplotlib</code> and indexing:</p>
<pre><code class="lang-python">c, h, w, d
x = torch.randn(c, h, w, d)
plt.imshow(x[0, :, :, 0])
</code></pre>
<p>or you could also use <code>torchvision.utils.make_grid</code> to create a single image with all slices:</p>
<pre><code class="lang-python">from torchvision.utils import make_grid

# permute to treat depth as "batch dim"
x_grid = make_grid(x.permute(1, 0, 2, 3)) 
# permute to create channels-last array (for matplotlib)
x_grid = x_grid.permute(1, 2, 0).numpy()
plt.imshow(x_grid)
</code></pre>
<p>Also, have a look at <a class="mention" href="/u/monai">@MONAI</a>’s <a href="https://github.com/Project-MONAI/MONAI/blob/e2ca7e8d1510b4944fd4bd45291f66e2cc1ec546/monai/visualize/img2tensorboard.py#L154"><code>plot_2d_or_3d_image</code></a>, which can apparently plot these 3D images as GIFs into TensorBoard.</p>

887574002: <p>Thank you so much <a class="mention" href="/u/ptrblck">@ptrblck</a>.</p>
nn.Parameter vs. register_parameter()

Carsten_Ditzel: <p>I dont quite get the difference between both mechanisms of adding and registering a possibly differentiable Parameter to a module. The docs dont specify exactly what distinguishes those two methods.</p>
<p>Advise is appreciated</p>

Andreas_Georgiou: <p>No difference. register_parameter() will allow you to register None as a parameter though in case you want to let the model know there will be a parameter with a specific name, but you don’t want to assign the parameter yet</p>