akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How generate batch of mixture distributions of Gaussian

Young_chai_Zheng: <p>I wanna generate a batch of mixture distribution of Gaussian from MLP’s output which represents the log of sigma, means, and logits. Are there any APIs in the distribution module work for this? Or just calculate from the formula?</p>

mrshenli: <p>This looks like a distribution instead of distributed question. cc <a class="mention" href="/u/ptrblck">@ptrblck</a> for help <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

Young_chai_Zheng: <p>I have solved it, thanks for your reply.</p>
Creating New Network After Pruning

Sydney_Faux: <p>Hi all,</p>
<p>So from the other posts that I have read about pruning, I’ve learned that the pytorch pruning module doesn’t actually speed up the model or reduce memory because it just inserts zeros. I read in one post that a potential solution would be to create an entirely new network with the unpruned weights of the original network. Would this be possible to do that with a CNN that has been filter pruned (zeroing out entire kernels)? If so, how would you go about it?</p>
How could I quantize models recursively and load the quantized model as float?

coincheung: <p>Hi,</p>
<p>I want to use dynamic mode to quantize my model. My code is like this:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torchvision

net = torchvision.models.resnet50()

qnet = torch.quantization.quantize_dynamic(net, qconfig_spec={nn.Conv2d, nn.BatchNorm2d, nn.Linear}, dtype=torch.qint8)
</code></pre>
<p>I found that only the last <code>nn.linear</code> module is converted to DynamicQuantizedLinear with other modules not changed. What is my problem and how could I make it work ?</p>
<p>By the way, I would like to convert the quantized model back to a float model(float32 or float16). Somethis like this:</p>
<pre><code class="lang-auto">class Model(nn.Module):
    ....

fmodel = Model()
fmodel.load_quantized_state_dict(torch.load('qstate.pth'))
f.forward(torch.randn(1,3, 32, 32).float())
</code></pre>
<p>How could I do this please ?</p>
Torch tensor initialization consumes a lot of memory

che85: <p>Hi all,</p>
<p>I am trying to figure out, why torch allocates so much memory for a tensor that at least to me doesn’t seem it should allocate this much memory:</p>
<pre><code class="lang-auto">  weights = torch.tensor([0.3242]).cuda()
</code></pre>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/8/487ac3e5b5020c11be8527d897e296028a131749.png" alt="image" data-base62-sha1="alboS1A2Vi8sZlZIAtdyVHiPxyx" width="565" height="65"></p>
<p>This tensor allocates more than 737MB on my GPU and I have absolutely no idea why this would happen.</p>
<p>I am using torch1.1 but also tried with torch 1.3 which results in more than 790MB memory allocation.</p>
<p>Neither</p>
<pre><code class="lang-auto">weights = weights.cpu()
torch.cuda.empty_cache()
</code></pre>
<p>or</p>
<pre><code class="lang-auto">weights = weights.detach().cpu()
torch.cuda.empty_cache()
</code></pre>
<p>NOR</p>
<pre><code class="lang-auto">del weights
torch.cuda.empty_cache()
</code></pre>
<p>have any effect. The memory stays allocated.</p>
<p>Does anyone know what to do in this case?</p>
<p>Thanks a lot.<br>
Christian</p>

ptrblck: <p>The CUDA context will be created on the device before the first tensor is created, which will use memory.</p>

mrartemev: <p>Hi!<br>
Sorry for posting in the old thread, but is there a way to calculate how much this CUDA context will take from GPU memory?</p>

ptrblck: <p>Unfortunately that’s not easily doable, as it depends on the CUDA version, the number of native PyTorch kernels, the number of used compute capabilities, the number of 3rd party libs (such as cudnn, NCCL) etc.<br>
The best way would be to perform a CUDA operation on your system and check the memory usage via <code>nvidia-smi</code>.</p>
How can I pad all 4 dimensions of an NCHW tensor?

ProGamerGov: <p>For example, say I have a tensor of size (3,5,8,8):</p>
<pre><code class="lang-auto">x = torch.ones(3,5,8,8)
</code></pre>
<p>And I want to pad the dimensions by separate padding values: N by 2, C by 1, H by 3, and W by 3.</p>
<p>How can I accomplish this?</p>

ptrblck: <p>You can use <code>F.pad</code>:</p>
<pre><code class="lang-python">x = torch.ones(3,5,8,8)
out = F.pad(x, (1, 2, 1, 2, 0, 1, 1, 1))
print(out.shape)
&gt; torch.Size([5, 6, 11, 11])
</code></pre>
<p>The <a href="https://pytorch.org/docs/stable/nn.functional.html#torch.nn.functional.pad">docs</a> explain the usage of the pad sizes.</p>

ProGamerGov: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> What if I want to use reflection padding? I tried to write a function for it, but there are incorrectly repeated values, and the first dimension is wrong in terms of size:</p>
<pre><code class="lang-auto">def pad_reflective_a4d(x: torch.Tensor, padding: List[int]) -&gt; torch.Tensor:
    """
    Reflective padding for all 4 dimensions of an NCHW tensor
    """

    assert x.dim() == 4
    assert len(padding) == 8

    # Pad width
    if padding[0] != 0:
        x = torch.cat([x, x.flip([3])[..., 0 : padding[0]]], dim=3)
    if padding[1] != 0:
        x = torch.cat([x.flip([3])[..., -padding[1] :], x], dim=3)

    # Pad height
    if padding[2] != 0:
        x = torch.cat([x, x.flip([2])[..., 0 : padding[2], :]], dim=2)
    if padding[3] != 0:
        x = torch.cat([x.flip([2])[..., -padding[3] :, :], x], dim=2)

    # Pad channels
    if padding[4] != 0:
        x = torch.cat([x, x.flip([1])[:, 0 : padding[4]]], dim=1)
    if padding[5] != 0:
        x = torch.cat([x.flip([1])[:, -padding[5] :], x], dim=1)

    # Pad batch
    if padding[6] != 0:
        x = torch.cat([x, x.flip([0])[0 : padding[6]]], dim=0)
    if padding[7] != 0:
        x = torch.cat([x.flip([0])[-padding[7] :], x], dim=0)
    return x

x = torch.arange(1,(3*4*4)+1).view(1, 3, 4, 4).float()
out = pad_reflective_a4d(x, (2,2, 2,2, 2,2, 2,2))
</code></pre>
How does word typos detection network intergrates typos confusion set?

111387: <p>I want to design a word typo detection network based on the attention mechanism. I currently want to integrating typos confusion sets into neural network training. I hope during the network training process, the network will pay attention to the correct words corresponding to the possible wrong words. How to design the network is more reasonable？</p>
<p>The typos confusion set contain many typos confusion pairs:</p>
<p>the typos confusing pairs of typos can represented as:</p>
<p>right_word=wrong_word1|wrong_word2…|wrong_wordn</p>
<p>which indicates that right_word may be wrongly written as wrong_word1 or wrong_word2 or wrong_wordn</p>
RNN On CIFAR10. The size of tensor a (10) must match the size of tensor b (100) at non-singleton dimension 1

111444: <p>Hello. I’m studying neural networks, and the task is to create a NN with this architecture. But the thing is that the error comes in</p>
<blockquote>
<p>ValueError: Expected input batch_size (300) to match target batch_size (100).</p>
</blockquote>
<p>According to assignment, I need to use MSELoss. Tried to fix it by transposing output values, but nothing works. Tried to change function to CrossEntropyLoss, but no luck either. Transposing helped, but the accuracy values did not increase and the error did not drop.<br>
What could be the problem?<br>
How do I get the correct architecture in such a trivial problem without transpositions and get an improvement?<br>
It’s worth noting that the NN works on MNIST without any problems. This only happens on CIFAR10.</p>
<p>Link to Google Colab: <a href="https://colab.research.google.com/drive/1IIVlUoUgcpHYv3psChjLlmqyZu_XI98q?usp=sharing" class="inline-onebox" rel="noopener nofollow ugc">Google Colab</a></p>

ptrblck: <p>You are currently flattening the input tensors such that the channel dimension would move into the batch dimension:</p>
<pre><code class="lang-python">images = images.view(-1, seq_len, input_size)
</code></pre>
<p>and I guess this could yield the shape mismatch for an uneven batch size (the last one in the epoch).<br>
Also, make sure this is indeed the use case you would like to apply, as it’s usually unwanted to move treat the different color channels of an input image as separate samples in the batch.</p>
BILSTM and attention for name entitity recognition

111387: <p>I want to conbine BiLSTM and attention for NER TASK，how to implement?</p>

Abhilash_Srivastava: <p>There are several ways of implementing it. One is described in <a href="https://dl.acm.org/doi/pdf/10.1145/3340997.3341002" rel="noopener nofollow ugc">this paper</a>.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/6/16f22853058120da5cbd6748f2210eb68cf6ff99.png" alt="image" data-base62-sha1="3gZk78o9CJuW3ZhxHQEkRv4GqRb" width="384" height="372"></p>
Best way to do a 2d cross product

alexus37: <p>hi,<br>
I have a tensor with the following shape B x C x 2 x NrVerticies x 2<br>
and now I want to do a cross product for the last dim. I found the function<br>
<a href="https://pytorch.org/docs/stable/generated/torch.cross.html?highlight=cross#torch.cross" rel="noopener nofollow ugc">torch.cross</a><br>
however, this function only supports 3D vectors, so I was wondering what would be the torch way to handle the cross product in 2D.</p>

ptrblck: <p>I think you might misunderstand the docs, as the specified <code>dim</code> should have the size 3 and you can use multiple dimensions as seen here:</p>
<pre><code class="lang-python">a = torch.randn(4, 5, 6, 7, 8, 3)
b = torch.randn(4, 5, 6, 7, 8, 3)
torch.cross(a, b, dim=5)
</code></pre>
<p>For your example it won’t work, since your last dimension has a size of 2.</p>

alexus37: <p>Thanks for the feedback. I understood the docs, but I wanted to use the cross product in 2D.<br>
Finally, I ended up doing this, if someone is having the same issue.</p>
<pre><code class="lang-python"> # pad the last dim to have 3D vector 
s_pad = F.pad(s, (0, 1))
# compute the cross product 
A = torch.cross(s_pad[:, index_list], s_pad[:, index_list_plus], dim=2)
# use the last dim which is the same as if the cross product would be done in 2D
A = A[..., 2]
</code></pre>
Extracting output from an intermediate layer of a pre-trained network

Siladittya_Manna: <p>I want to extract features from the penultimate layer in ResNet18 in PyTorch. I have seen some posts in this discussion forum, suggesting to use hooks to get the output. But I am not sure if I fully understand how it works and will it work if I want to back-propagate gradients.</p>
<p>Is there any alternative away to do that like in Tensorflow “model.get_layer(layer_name).output”?</p>
<p>I tried using the code below to get the output from the penultimate layer</p>
<p><code>rn18 = models.resnet18(pretrained=True)</code></p>
<p>Getting the name of the modules</p>
<pre><code class="lang-auto">cnames = []
for n,c in rn18.named_children():
    cnames += [n]

cnames = cnmaes[:-1]
</code></pre>
<p>Then I tried building a new net using the modules</p>
<p><code>new_net = nn.Sequential(nn.ModuleList([rn18._modules[n] for n in cnames]))</code></p>
<p>But when I tried to get a summary of the model</p>
<p><code>summary(new_net,(3,224,224))</code></p>
<p>I am getting an error</p>
<p><code>TypeError: forward() takes 1 positional argument but 2 were given</code></p>
<p><strong>EDIT</strong>: Problem Solved<br>
<code>rn18._modules</code> is an OrderedDict</p>
<p>So, popping the layers which are not required from the dict (like, <code>rn18._modules.pop('fc')</code>) and then doing<br>
<code>nn.Sequential(rn18._modules)</code> does the job. I get the output from the penultimate layer of the pretrained network.</p>
<p>I think the issue was that I was giving <code>nn.ModuleList</code> as input to the <code>nn.Sequential</code>, whereas giving an OrderedDict as input worked fine.</p>
<p>This also works</p>
<pre><code class="lang-auto">        self.children_list = []
        for n,c in self.pretrained.named_children():
            if n != self.output_layer:
                self.children_list.append(c)
            else:
                self.children_list.append(c)
                break
        self.net = nn.Sequential(*self.children_list)
        self.pretrained = None
</code></pre>
'Example' object has no attribute 'text_x' - Failing during Vocab Creation

amitkayal: <p>Hello All,</p>
<p>I am reading from a CSV file but then during Vocab creation it is failing although I see that column is present into my CSV file. I have further looked into my data after reading from CSV file and i can see the columns properly as shared below.</p>
<p>Can anyone please guide me why this is failing durinh vocab creation by TWEET.build_vocab(train_ds)?</p>
<p><em>{‘text_x’: [’<span class="mention">@tmobilehelp</span>’, ‘I’, ‘think’, ‘there’, “'s”, ‘an’, ‘issue’, ‘in’, ‘my’, ‘area’, ‘accessing’, ‘data’, ‘from’, ‘Apple’, “'s”, ‘Podcast’, ‘both’, ‘streaming’, ‘and’, ‘downloads’, ‘over’, ‘LTE’, ‘.’, ‘Please’, ‘inspect’, ‘.’], ‘text_y’: [’<span class="mention">@351818</span>’, ‘Hey’, ‘Vincent’, ‘!’, ‘Shoot’, ‘me’, ‘a’, ‘DM’, ‘<a href="https://t.co/zMHlDmwgGJ" class="inline-onebox" rel="noopener nofollow ugc">https://t-mo.io/2lJ0Kc1</a>’, ‘so’, ‘I’, ‘can’, ‘take’, ‘a’, ‘closer’, ‘look’, ‘at’, ‘that’, ‘for’, ‘you’, ‘,’, ‘I’, ‘got’, ‘your’, ‘back’, ‘!’, '</em>’, ‘VictorianLang’]}*</p>
<p><strong>CSV File Structure</strong></p>
<div class="md-table">
<table>
<thead>
<tr>
<th>text_x</th>
<th>text_y</th>
</tr>
</thead>
<tbody>
<tr>
<td>Where the hell is nickelodean and MTV <span class="mention">@Ask_Spectrum</span>
</td>
<td>
<span class="mention">@368438</span> Thank you for reaching out. I am sorry your channels are missing. Can you please DM the account number or the pho… <a href="https://t.co/2bqoClob2O" class="inline-onebox" rel="noopener nofollow ugc">https://goo.gl/veJK1t</a>
</td>
</tr>
</tbody>
</table>
</div><p><strong>Code which is reading from CSV file</strong>:<br>
TWEET = data.Field(sequential = True, tokenize = ‘spacy’, batch_first =True, include_lengths=True)</p>
<p>RESPONSE = data.Field(tokenize =‘spacy’,sequential = True, batch_first =True,include_lengths=True)</p>
<p>fields = [(‘text_x’, TWEET), (‘text_y’, RESPONSE)]<br>
train_ds, valid_ds = data.TabularDataset.splits(<br>
path = ‘data’,<br>
train = ‘Tweet_train.csv’,<br>
validation = ‘Tweet_valid.csv’,<br>
format = ‘csv’,<br>
fields = fields,<br>
skip_header = True<br>
)<br>
<strong>Code which is failing</strong><br>
TWEET.build_vocab(train_ds)<br>
RESPONSE.build_vocab(train_ds)</p>
<p><strong>Error</strong><br>
AttributeError                            Traceback (most recent call last)<br>
 in ()<br>
1 # Before creating iterators of the Datasets we need to build the vocabulary for each Field object:<br>
----&gt; 2 TWEET.build_vocab(train_ds)<br>
3 RESPONSE.build_vocab(train_ds)</p>
<p>1 frames<br>
/usr/local/lib/python3.6/dist-packages/torchtext/data/dataset.py in <strong>getattr</strong>(self, attr)<br>
152         if attr in self.fields:<br>
153             for x in self.examples:<br>
–&gt; 154                 yield getattr(x, attr)<br>
155<br>
156     <span class="mention">@classmethod</span></p>
<p>AttributeError: ‘Example’ object has no attribute ‘text_x’</p>
Weight loss/learning rate across sequences in same batch

PLarsen79: <p>Say I want to run a forward pass through the network, using a batch-size of 256, and before backprop I would want to weight the losses across the batch input/target sequences, is that possible and what would be the best practise?</p>
<p>It’s not a solution for me to do preproccessing and collect input/target sequences that will have the same weighted loss/learnrate together and run them in the same batch because what sequence that will be backpropagated with higher learnrate will be determined in comparison with the other sequences in the same batch, after the forward pass.</p>
<p>Is this a catch-22 situation that isn’t possible due to the fact the gradients doesn’t care about the actual different losses inside the same batch? There’s no way to separate different gradients, from different sequences in the same batch, apart from each other, right? Do I have to run the batch once using no_grad and then based on the result filter out/exchange sequences from the batch and then run it again using grad to simulate and approximate this?</p>

ptrblck: <p>You could use an unreduced loss via <code>reduction='none'</code> in the creation of the criterion and multiply the weight matrix with the loss tensor before reducing it and calling <code>backward</code>.</p>

PLarsen79: <p>Thanks, I think I understand the logic but fail to understand what to do with the loss tensor before calling <code>backward</code></p>
<p>In my forward function I have this now:<br>
<code>loss = F.cross_entropy(logits.view(-1, logits.size(-1)), targets.view(-1), reduction='none')</code></p>
<p>…here I perform my matrix mul on the loss tensor …</p>
<p>After that, should I do a simple <code>loss = loss.mean()</code> before calling <code>loss.backward</code>? I don’t understand how that will add a weigthed bias to the gradients? But if it works I will go for it, as simple as it sounds…</p>
<p>Or should I call <code>loss[n].backward()</code> for each batch sequence? Would that make any difference?</p>
<p>EDIT: Oh, nevermind. I figured that was an unneccessary question to ask when the answer was one test and therefore ~2 minutes away <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Question about deepcopy a model

longdct2706: <p>Hi everyone,</p>
<p>I have kind of an unique question about deepcopy a model. Currently, I’m doing a project about Neural Architecture Search to find a new recurrent cell (like LSTM). Suppose I have 3 modules A, B and C, with A and B both use C in their forward. After that, both A and B will be used in a new recurrent cell</p>
<pre><code class="lang-auto">class A(nn.Module):
    def __init__(self, C):
        # other modules
        self.C = C

    def forward(self, x):
        # other modules
        output = self.C(x)

# similar to B

class RecurrentCell(nn.Module):
    def __init__(self, A, B):
        self.A = A
        self.B = B

    def forward(self, x_a, x_b):
        x_a = self.A(x_a)
        x_b = self.B(x_b)
        return x_a, x_b
</code></pre>
<p>Because I want my new recurrent cell to behave like Pytorch RNN/LSTM module, I use deepcopy on A and B to copy the <code>RecurrentCell</code> to use in other layers. My question is that if I use deepcopy like that, will A and B still share the same copy of C (i.e A.C and B.C are the same object)?</p>
<p>Thanks for your help.</p>
How to specify multiple GPU usage

spadel: <p>The documentation for <code>nn.DataParallel</code> is not clear at all to me. How exactly am I supposed to pass input and targets to the GPU, if I specify concrete GPU’s?</p>
<p>My code looks something like this:</p>
<pre><code class="lang-auto">device = torch.device('cuda:' + str(arg.gpu) if torch.cuda.is_available() else 'cpu')
model = Model(arg).to(device)
for epoch in range(epochs+1):
    model.train()
    for step, (original, keypoints) in enumerate(train_loader):
        original, keypoints = original.to(device), keypoints.to(device)
</code></pre>
<p>I want to be able to pass pass GPU’s to the arg_parser through <code>--gpu 5 7</code>, which produces a list <code>[5, 7]</code>. Simply adding the line <code>model = nn.DataParallel(Model(arg), device_ids=[5, 7])</code> is not enough, since I have to specify the device variable. However, <code>device = torch.device('cuda:5,7' if torch.cuda.is_available() else 'cpu')</code> results in an error. What am I supposed to do instead?</p>

Mazhar_Shaikh: <p>Here’s how your code should look:</p>
<pre><code class="lang-auto">model = torch.nn.DataParallel(Model(args), device_ids=arg.gpu)
for epoch in range(epochs+1):
    model.train()
    for step, (original, keypoints) in enumerate(train_loader):
        outputs = model(original, keypoints)
        &lt;do some training&gt;
</code></pre>
<p>Why does this work? For nn.DataParallel, the input tensors can be on any device, even cpu. The module automatically transfers the called arguments to the appropriate cuda device.</p>
<p>What was the issue with your code? torch.device is used to define one and only one device. It can only accept the format ‘cuda:x’/‘cuda’/‘cpu’.</p>

spadel: <p>Thanks for your quick reply <a class="mention" href="/u/mazhar_shaikh">@Mazhar_Shaikh</a>! One question though: During my forward pass I have to create several tensors (for example because I produce a grid for transformations). At the moment, I have to send them to the device as well, so that multiplications and additions with the input tensors work. How will this be effected with your code?</p>

Mazhar_Shaikh: <p>If you are creating tensors inside your forward pass, Please use the below code</p>
<pre><code class="lang-auto">class Model(nn.Module):
    def __init__(self, args):
        &lt;insert initialization code&gt;
    def forward(self, original, keypoints):
        some_tensor = torch.tensor([....], device=original.device)
        &lt;do some thing with some_tensor&gt;
</code></pre>

spadel: <p>Using your code produces the following error for me:</p>
<p><code>RuntimeError: module must have its parameters and buffers on device cuda:5 (device_ids[0]) but found one of them on device: cpu</code></p>
<p>The error occurs in the line <code>outputs = model(original, keypoints)</code>.</p>
How to custom my attention function in torch.nn.MultiheadAttention

jack_ma: <p>Hello! I look up the docs and can not find what attention function used in torch.nn.MultiheadAttention: <a href="https://pytorch.org/docs/stable/generated/torch.nn.MultiheadAttention.html#torch.nn.MultiheadAttention" rel="noopener nofollow ugc">MultiheadAttention — PyTorch 1.7.0 documentation</a><br>
What’s more, I don’t know how to custom my attention function.</p>

matheo_r: <p>Hi jack_ma,</p>
<p>I would advise you to read carefully the paper Attention is all you Need: <a href="https://arxiv.org/abs/1706.03762" rel="noopener nofollow ugc">https://arxiv.org/abs/1706.03762</a> if you want to build your own attention module from scratch.</p>
<p>Basically, the multi head attention takes as inputs queries, keys and values that are mapped from a sequence of embeddings (of words for instance). The keys are conveying information about the nature of the embeddings, the values encode the content itself while the queries indicate the information the embeddings need (<a href="https://arxiv.org/abs/1911.07757" rel="noopener nofollow ugc">https://arxiv.org/abs/1911.07757</a>). I hope this provides some intuition about the triplet query - key - value and on how the attention module can learn which embeddings in the sequence to focus on.</p>

jack_ma: <p>Thank you, it means there is no api provided in <em>torch.nn.MultiheadAttention</em> to do this, right? If I want to use my own attention function I need to implement from scratch without using <em>torch.nn.MultiheadAttention</em> module.</p>

matheo_r: <p>Hi <a class="mention" href="/u/jack_ma">@jack_ma</a></p>
<p>Sorry for the late answer. You could write a custom class using MultiHeadAttention module as its parent:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

class myAttentionModule(nn.MultiheadAttention):
	def __init__(self, embed_dim, num_heads):
		super(myAttentionModule, self).__init__(embed_dim, num_heads)

	def forward(self, query, key, value):
		# your own forward function


query = torch.rand((1,10))
key = torch.rand((1,10))
value = torch.rand((1,10))


my_attention_module = myAttentionModule(embed_dim=10, num_heads=2)
out = my_attention_module(query, key, value)
</code></pre>
Cannot backpropagate using networkx with pytorch

ececinar: <p>Hi,</p>
<p>We are trying to calculate a loss function using networkx functions. Since networkx works with numpy arrays, the computation graph is broken and we cannot backpropagate with the loss. The loss doesn’t have a grad_fn attribute. How can we overcome the issue without having to re-implement the backward function?</p>
<p>Thanks</p>

sinemelifhaseki: <p>I have the exact same issue. I am looking for a solution without trying to implement backward or networkx operations by myself, I am kinda new to pytorch so i would appreciate a lot.</p>

ptrblck: <p>I’m not familiar with the <code>networkx</code> library, but based on the description you would need to implement the backward functions manually, if numpy arrays are used under the hood.<br>
Alternatively, you could also try to use PyTorch tensors instead of numpy arrays (if all operations are supported), but this sounds as if you would have to rewrite the complete library.<br>
Could you explain a bit what <code>networkx</code> is doing and what your exact use case is?</p>

ececinar: <p>We were using networkx to calculate centrality measurements of a given graph. However, we decided to move on with a simpler loss function using different ways of measuring centrality. Thanks for the answer !</p>
Image input range to convolutional network

khws58: <p>When the input of a convolutional neural network is a (grayscale) image, it is common practice to scale the pixel values to the range [0,1], for example by using the <code>torchvision.transforms.ToTensor</code> transform.</p>
<p>Many works then proceed to subtract 0.5 and scale by 2 in order to transform this range to [-1, 1]. This is of course a normalization step. However, both in literature and applications, there are many cases where this last transform is not performed.</p>
<p>What would be the argument for (not) applying this normalization step? Why do some projects incorporate this step in their design and others do not?</p>

ptrblck: <p>Normalizing/whitening/decorrelating the input is beneficial as for very simple examples it can be shown that the loss surface is also decorrelated/round and your model can thus be trained easier.<br>
I would generally recommend to take a look at Bishop’s Pattern Recognition and Machine Learning as well as the <a href="https://www.deeplearningbook.org/">Deep Learning Book</a>.</p>

khws58: <p>What about the specific case for images? Are projects that do not normalize missing out on potential improvement? Is there any reason not to normalize?</p>
Pytorch colormap gather operation

Joris_Mollinga: <p>I am trying to implement the conversion of gray scale image to an RGB image through a colormap myself in PyTorch. I have:</p>
<ul>
<li>A tensor <em>gray_image</em> which is a LongTensor of 512 x 512 with (integer) values between 0 and 255</li>
<li>The <em>color_map</em>, which is a FloatTensor of 256 x 3, where every row represents the RGB value for a specific gray value</li>
<li>An empty output file, which is a FloatTensor of 512 x 512 x 3</li>
</ul>
<p>I fill the output tensor like this:</p>
<pre><code class="lang-auto">output = torch.zeros((512, 512 3) # Initialize empty
gray_image = (gray_image * 255).long() # Tensor values between 0 and 255 and LongTensor

for i in range(gray_image.shape[0]):
    for j in range(gray_image.shape[1]):
        output[i, j, :] = color_map[grads[i, j]]
</code></pre>
<p>This is quite slow and this must be possible in 1 or 2 lines using <code>torch.gather()</code>. I tried to do this myself, but did not succeed and found the documentation of <code>torch.gather()</code> to be quite complex. Can someone help me with this?</p>

Mazhar_Shaikh: <p>You can use this code</p>
<pre><code class="lang-auto">color_map = #Tensor of shape(256,3)
gray_image = (gray_image * 255).long() # Tensor values between 0 and 255 and LongTensor and shape of (512,512)
output = color_map[gray_image] #Tensor of shape (512,512,3)
</code></pre>
<p>Why does this work? The gray_image tensor is used as an index in the 0th dimension of color_map. The color_map tensor is accessed with this index and the new tensor created has the shape of gray_image inserted in the 0th dimension of output.</p>
Different Transformations for Different Classes

Wilan: <p>I would like to perform different transformations on  5 different classes in a multi-class classification problem. What’s the cleanest way to do this using <code>ImageFolder</code> and <code>DataLoader</code>?</p>
<p>Note: The classes also have quite different sizes, ranging from 50-3,000 images<br>
How can I set this up to not over fit on the classes with more data?</p>
<p>My data structure looks like this:</p>
<pre><code class="lang-auto">/data
    /train
        /class1
        /class2
        /class3
        /class4
        /class5
    /test
        /class1
        /class2
        /class3
        /class4
        /class5
</code></pre>

Shisho_Sama: <p>The easiest way that comes to my mind at the moment is to create a new Dataset instance and explicitly use different transformations on different classes.<br>
For the unbalanced dataset, you may want to use WeightedRandomSampler</p>

Wilan: <p>Currently my setup is something like this:</p>
<pre><code class="lang-auto">class ConcatDataset(Dataset):
    '''Concatenates Datasets'''
    
    def __init__(self, *datasets):
        self.datasets = datasets

    def __getitem__(self, i):
        return tuple(d[i] for d in self.datasets)

    def __len__(self):
        return min(len(d) for d in self.datasets)

train_1_transforms = ...
train_2_transforms = ...
train_3_transforms = ...
train_4_transforms = ...
train_5_transforms = ...

train_1_data = ImageFolder(train_1_dir, transform=train_1_transforms)
train_2_data = ImageFolder(train_2_dir, transform=train_2_transforms)
train_3_data = ImageFolder(train_3_dir, transform=train_3_transforms)
train_4_data = ImageFolder(train_4_dir, transform=train_4_transforms)
train_5_data = ImageFolder(train_5_dir, transform=train_5_transforms)

train_concat_data = ConcatDataset(train_1_data, train_2_data, train_3_data, train_4_data, train_5_data)

train_loader = DataLoader(train_concat_data, batch_size=64, shuffle=True, pin_memory=True, num_workers=4)
</code></pre>
<p>I’m getting error:</p>
<pre><code class="lang-auto">RuntimeError: Found 0 files in subfolders of: ./data/train/class_1
</code></pre>
<p>I don’t have any idea how else to apply separate transformations to each class.</p>

Shisho_Sama: <p>Dont use ImageFolder, just read the images using <code>os.walk</code> for example and store their paths and labels (based on which class they reside in you can give them a number) and then in <code>__getitem__</code>, after you have read the image, based on the label that you also have, you can apply the needed transformation  and then return the pair.</p>
Difference between PatchGAN and Fully Convolutional Layer

Eden_Park: <p>Hi.</p>
<p>I’m reading <a href="https://medium.com/@sahiltinky94/understanding-patchgan-9f3c8380c207" rel="noopener nofollow ugc">this post</a> to get to know PatchGAN more.<br>
And I saw this pic.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/7/07293bb6478ec9e49cd38944c5a3f69c9cb3994b.png" data-download-href="https://discuss.pytorch.org/uploads/default/07293bb6478ec9e49cd38944c5a3f69c9cb3994b" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/7/07293bb6478ec9e49cd38944c5a3f69c9cb3994b_2_478x500.png" alt="image" data-base62-sha1="11lGpLVmtGS6CuJYMVfT0LFliHh" width="478" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/7/07293bb6478ec9e49cd38944c5a3f69c9cb3994b_2_478x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/0/7/07293bb6478ec9e49cd38944c5a3f69c9cb3994b_2_717x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/0/7/07293bb6478ec9e49cd38944c5a3f69c9cb3994b.png 2x" data-dominant-color="E9EEED"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">777×812 164 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>How different of between PatchGAN and Fully Convolutional Layer?<br>
Those are looks so same.</p>

Sahil: <p>Hi! I am the author of this blog. I can answer this question.<br>
See, whatever you have heard all about ConvNet like ResNet, U-Net, etc are like usual. Whereas PatchGAN is special case for ConvNet especially Discriminator in GAN theory. In PatchGAN, the output of the architecture only infer you whether it is fake or real. That’s IT!!</p>
<p>Now see the image below and let say,<br>
if each pixel close to ‘0’ means fake<br>
if each pixel close to ‘1’ means real.<br>
(As I already explained in the blog that: each pixel from output comes from 70x70 image patch from Input image)<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/c/5/c57805bc920392c220618f993e1645e8bc0acb4e.png" alt="Capture" data-base62-sha1="saTcVu5G3hgbgE9NwfeRsmo67lc" width="382" height="324"></p>
<p>Now what does its infering you? Its telling to the Discriminator that: there are some image patches from an input image which contain fake and other image patches which contain real.</p>
<p>Now, If you are familiar in GAN theory, you should know they were trying to achieve their own the objective of generator and discriminator model. So, PatchGAN comes under Discriminator.</p>
<p>Hope you got it.</p>
<p>Cheer:D</p>

Eden_Park: <p>Oh, I like your post.<br>
Thanks!</p>
HELP - mixed precision problem

Thiru: <p>I am new to pytorch. I was using tensorflow and keras before but when I came to know about pytorch’s stability and native AMP support. I switched to pytorch.</p>
<p>GPU: RTX 2070 super (vram - 8GB)</p>
<p>I used mixed precision with grad accum of 2. Image resolution is 416 x 416. But I’m getting CUDA out of memory error. I even switched to kaggle kernel, The same error pops up there too.</p>
<p>kaggle GPU: NVIDIA K80(vram-16GB)</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/8/4881a287f1c92121dfd499a67334ebe0af0f39b8.png" data-download-href="https://discuss.pytorch.org/uploads/default/4881a287f1c92121dfd499a67334ebe0af0f39b8" title="Screenshot from 2020-12-20 10-15-10"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/8/4881a287f1c92121dfd499a67334ebe0af0f39b8_2_500x500.png" alt="Screenshot from 2020-12-20 10-15-10" data-base62-sha1="alq7oJdPO53DZESdUSRsrSC3TDy" width="500" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/8/4881a287f1c92121dfd499a67334ebe0af0f39b8_2_500x500.png, https://discuss.pytorch.org/uploads/default/original/3X/4/8/4881a287f1c92121dfd499a67334ebe0af0f39b8.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/4/8/4881a287f1c92121dfd499a67334ebe0af0f39b8.png 2x" data-dominant-color="EEEEEE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2020-12-20 10-15-10</span><span class="informations">708×708 63.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Is my logic correct?</p>
<p>Thanks in advance</p>

ptrblck: <p>The code logic is correct as seen in the <a href="https://pytorch.org/docs/stable/notes/amp_examples.html#gradient-accumulation">amp example</a>, so you might need to reduce the batch size further.<br>
Is the training running fine without gradient accumulation and if so how large is the memory usage?</p>

Thiru: <p>my batch size is 32 for validation and 16 for training with gradient accumulation of 2.<br>
even wihtout grad accum, it’s not working properly.<br>
I even tried with reduced img resolution and batch size 16 for both.</p>
Mat1 dim 1 must match mat2 dim 0

swarup: <p>def forward(self, x):</p>
<pre><code>    """Perform forward."""

    

    # conv layers

    x = self.conv_layer(x)

    

    # flatten

    #x = x.view(x.size(0), -1)

    x = x.view(x.shape, -1)

    # fc layer

    x = self.fc_layer(x)

    return x
</code></pre>
<p>Hi while running this sell  iam getting an error like this :</p>
<p>mat1 dim 1 must match mat2 dim 0</p>
<p>The the length of our train data is 31690 and test data is 2050<br>
can you suggest me a way to solve this error</p>

ptrblck: <p>Print the shape of <code>x</code> before passing it to the linear layer and make sure the number of features in the activation match the <code>in_features</code> from the linear layer.</p>
<p>Also, <code>x = x.view(x.shape, -1)</code> won’t change the shape of the activation, so you might want to use the commented approach.</p>
Torchvision transforms nomalize value selection

111429: <p>Hi,<br>
This is the value for normalizing the kinetics dataset for video classification in the official script.<br>
</p><aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/vision/blob/master/references/video_classification/train.py#L115" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/vision/blob/master/references/video_classification/train.py#L115" target="_blank" rel="noopener nofollow ugc">pytorch/vision/blob/master/references/video_classification/train.py#L115</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="105" style="counter-reset: li-counter 104 ;">
<li>print("torchvision version: ", torchvision.__version__)</li><li></li><li>device = torch.device(args.device)</li><li></li><li>torch.backends.cudnn.benchmark = True</li><li></li><li># Data loading code</li><li>print("Loading data")</li><li>traindir = os.path.join(args.data_path, args.train_dir)</li><li>valdir = os.path.join(args.data_path, args.val_dir)</li><li class="selected">normalize = T.Normalize(mean=[0.43216, 0.394666, 0.37645],</li><li>                        std=[0.22803, 0.22145, 0.216989])</li><li></li><li>print("Loading training data")</li><li>st = time.time()</li><li>cache_path = _get_cache_path(traindir)</li><li>transform_train = torchvision.transforms.Compose([</li><li>    ConvertBHWCtoBCHW(),</li><li>    T.ConvertImageDtype(torch.float32),</li><li>    T.Resize((128, 171)),</li><li>    T.RandomHorizontalFlip(),</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
<br>
i.e.,<br>
normalize = T.Normalize(mean=[0.43216, 0.394666, 0.37645],<br>
std=[0.22803, 0.22145, 0.216989])
<p>And in <a href="https://pytorch.org/docs/stable/torchvision/models.html" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/torchvision/models.html</a><br>
it says that “All pre-trained models expect input images normalized in the same way”<br>
with this norm transform:<br>
normalize = transforms.Normalize(mean=[0.485, 0.456, 0.406],<br>
std=[0.229, 0.224, 0.225])</p>
<p>My case is to use pretrained resnet in torchvision to extract feature on kinetics video frames.<br>
So which norm should I choose?<br>
And how these values come from?<br>
Thanks.</p>

ptrblck: <p>I guess the stats used in the linked script would have been calculated from the Kinetics dataset, while the pretrained (classification) models were trained on ImageNet using the mentioned stats in the docs.</p>
Default value cudnn.enabled

maikefer: <p>Since it is misisng in the docu (<code>https://pytorch.org/docs/stable/backends.html</code>) I’m wondering what is the default value of torch.backends.cudnn.enabled or does it depend on special things like PyTorch version, hardware etc?</p>

ptrblck: <p>If cudnn is available (which is the case for the binaries shipping with CUDA+cudnn), it’s enabled by default and you can check it via <code>print(torch.backends.cudnn.enabled)</code>.</p>
When should we set torch.backends.cudnn.enabled to false, especially for LSTM?

UCDuan: <p>Hi all,</p>
<p>I’m using Captum to do the integrated gradient for an LSTM model. If I just load the model and the weights, it shows <code>cudnn RNN backward can only be called in training mode</code>. I tried to remove model.eval() but it doesn’t give the correct result.</p>
<p>I also tried to put all the data and model to CPU and it works fine, but slowly. I’m not familiar with how PyTorch calculate the weights for LSTM but from the following thread, <a href="https://github.com/pytorch/captum/issues/564#issuecomment-748274352" rel="noopener nofollow ugc">https://github.com/pytorch/captum/issues/564#issuecomment-748274352</a> , adding <code> torch.backends.cudnn.enabled=False</code> fixed the problem.<br>
I noticed the GPU is still in work with <code>cudnn.enabled=False</code>.</p>
<p>My question is what does <code>backends.cudnn.enabled</code> mean here and why LSTM requires it. When should we concern or use it? Thanks!</p>

ptrblck: <p><code>backends.cudnn.enabled</code> enables cudnn for some operations such as conv layers and RNNs, which can yield a significant speedup.<br>
The cudnn RNN implementation doesn’t support the backward operation during <code>eval()</code> and thus raises the error. You could disable cudnn for your workload (as already done) or try to call <code>.train()</code> on the RNN module separately after using <code>model.eval()</code>.</p>

UCDuan: <p>Hi thanks for your reply.</p>
<p>I’m using Captum for the integrated gradient. I have tried either <code>backends.cudnn.enabled=False</code> or avoid <code>model.eval()</code>. But these two methods give different results. Will it affect the gradient for LSTM if we remove <code>model.eval()</code>?</p>
<p>Thanks.</p>

ptrblck: <p>I think it would affect the gradients (assuming you could disable cudnn for the RNN), since e.g. dropout layers won’t be used and the running stats in batchnorm layers are applied to normalize the activation input instead of the batch statistics.</p>

UCDuan: <p>Ah, that makes sense. So we should disable cudnn when calculating the gradient for RNN. Thanks!</p>
<p>Still, could you please give more reasons why cudnn RNN does not support the backward operations?</p>

ptrblck: <p>It’s a limitation from the cudnn implementation.<br>
From the <a href="https://docs.nvidia.com/deeplearning/cudnn/api/index.html#cudnnRNNForward">cudnn docs</a>:</p>
<blockquote>
<p>fwdMode <em>Input</em>. Specifies inference or training mode (CUDNN_FWD_MODE_INFERENCE and CUDNN_FWD_MODE_TRAINING). In the training mode, additional data is stored in the reserve space buffer. This information is used in the backward pass to compute derivatives.</p>
</blockquote>
<p>Thus if you call <code>model.eval()</code>, you won’t be able to calculate the gradients using the cudnn RNN implementation. As described before, calling:</p>
<pre data-code-wrap="python"><code class="lang-python">model.eval()
model.rnn_layer.train()
</code></pre>
<p>could solve this issue, as it would keep the RNN in training mode.</p>
How to stack/concatenate features set of different sizes?

Stanley_C: <p>I currently have multiple feature sets in the format (n,b), with n being the number of “elements” and b being the number of features. For example, I have (1024,128), (512, 256), and (256,512). How do I stack/concatenate my features with padding, so that get (x,y) where x is the largest and y is the largest of features, with the rest being “padded” of the empty features? For example, how would I get (1024, 128+256+512), from my example?</p>
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor []], which is output 0 of SelectBackward, is at version 499; expected version 498 instead

Myth: <p>Hi everyone,</p>
<p>I am getting this error but I do not see why.  I wanted to add a  custom parameter to my RNN. The code is as follows :</p>
<p>class RNN_pre_d(nn.Module):<br>
def <strong>init</strong>(self, hidden_dim=64, num_layers=1):<br>
super(RNN_pre_d, self).<strong>init</strong>()<br>
self.num_layers = num_layers<br>
self.hidden_dim = hidden_dim<br>
self.lstm = nn.LSTM(1, self.hidden_dim, self.num_layers)<br>
self.fcF_c = nn.Linear(self.hidden_dim, 1)<br>
w_final = torch.randn(1)<br>
self.w_final = nn.Parameter(w_final)</p>
<pre><code>def forward(self, x):
    x1, (h_1, c_1) = self.lstm(x)    ## x.shape = [sequence_len, batch_size, hidden_dim]
    
    c_train = torch.log(1+torch.exp(self.fcF_c(x1.squeeze(1))))
    
    c_train = c_train.squeeze(-1)
    
    for i in range(1,len(c_train)):
        c_train[i] = c_train[i] + ( 1/(1+torch.exp(-self.w_final)) ) * c_train[i-1] 

    return c_train, h_1, c_1  
</code></pre>
<p>the problem only appears when trying to backprop</p>
<p>NumbEpochs      = 100</p>
<p>learning_rate   = 10e-04<br>
NumbHiddLayer   = 1<br>
sizeHidd        = 128</p>
<p>nn_pre_d = RNN_pre_d(hidden_dim = sizeHidd,<br>
num_layers = NumbHiddLayer)</p>
<p>optimizer = optim.Adam(nn_pre_d.parameters(),<br>
lr=learning_rate)</p>
<h2>MSE loss function</h2>
<p>loss = nn.MSELoss() ; listLossTrain = []</p>
<h2>Training data standardized</h2>
<p>y = np.random.normal(0, 0.1, 500)<br>
mean_train = np.mean(y) ; std_train = np.std(y) ; Z_stand_train = (y-mean_train) / std_train<br>
data_nn_stand_train = torch.FloatTensor([Z_stand_train])<br>
data_nn_stand_train = data_nn_stand_train.permute(1,0).unsqueeze(-1)<br>
target_train = torch.tensor(y)</p>
<h2>Training</h2>
<p>for epoch in range(NumbEpochs):<br>
optimizer.zero_grad()</p>
<pre><code># Make the prediction in training set.
c_train, h_1, c_1  = nn_pre_d.forward(data_nn_stand_train)
            
# Compute a scalar loss in training set.
loss_train = loss(target_train, c_train)


# Compute the gradient for the model parameters and update model parameters.
loss_train.backward()
optimizer.step()

listLossTrain.append(loss_train.detach().numpy())
</code></pre>
<p>Thank you!</p>

ptrblck: <p>I guess the assignment to <code>c_train[i]</code> might raise this error. Could you append the results to a <code>list</code> and use <code>torch.stack</code> on it instead?</p>
<p>PS: you can post code snippets by wrapping your code into three backticks ```, which would make debugging easier. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

Myth: <p>Thank you very much! It’s working!</p>
<p>I will take notice for next time!</p>
Converting np array to list and back to array causing issues

Dawny33: <p>I am having issues after converting an image array to a list, and converting back to an array</p>
<pre><code class="lang-auto">img_list = img.tolist()
img_arr = np.array(img_list)
</code></pre>
<p>However, directly consuming the image array works fine.</p>
<p>Why is the conversion causing an issue?</p>
<aside class="onebox stackexchange">
  <header class="source">
      <a href="https://stackoverflow.com/questions/65376876/runtimeerror-upsample-bilinear2d-channels-last-not-implemented-for-long-aft" target="_blank" rel="noopener nofollow ugc">stackoverflow.com</a>
  </header>
  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/4993513/dawny33" target="_blank" rel="noopener nofollow ugc">
    <img alt="Dawny33" src="https://i.stack.imgur.com/z0jss.png?s=128&amp;g=1" class="thumbnail onebox-avatar" width="60" height="60">
  </a>
<h4>
  <a href="https://stackoverflow.com/questions/65376876/runtimeerror-upsample-bilinear2d-channels-last-not-implemented-for-long-aft" target="_blank" rel="noopener nofollow ugc">RuntimeError: "upsample_bilinear2d_channels_last" not implemented for 'Long' after converting np array to list</a>
</h4>

<div class="tags">
  <strong>python, numpy, pytorch, detectron</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stackoverflow.com/users/4993513/dawny33" target="_blank" rel="noopener nofollow ugc">
    Dawny33
  </a>
  on <a href="https://stackoverflow.com/questions/65376876/runtimeerror-upsample-bilinear2d-channels-last-not-implemented-for-long-aft" target="_blank" rel="noopener nofollow ugc">04:16AM - 20 Dec 20 UTC</a>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>