akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Is there any better way to do this kind of indexing?

entslscheia: <p>For example, I have a tensor <code>a</code> of shape <code>[k, m, n]</code> and tensor <code>b</code> fo shape <code>[k, ]</code>. What I want to do is to get <code>a[torch.arange(k), b]</code>. Is there any api provided by PyTorch can directly deal with it? Creating a new tensor using <code>torch.arange(k)</code> doesn’t look that elegant.</p>

albanD: <p>Hi,</p>
<p>I’m confused what you expect to get here. what does b contain?<br>
Can you share a small code sample that shows what you expect as output?</p>

entslscheia: <p>Sorry, I should have included more information here. Every element in <code>b</code> is in <code>[0, m)</code>. You can imagine that we have a sequence ranking task. k is the batch size, and for each batch instance, we have m candidate sequences of length n. So now <code>b</code> is the index of candidates, and there are k of them, because we want to choose one candidate for every batch instance. I think there should be some more straightforward way to handle this right? Since it kinda seems like a common need.</p>

albanD: <p>Yes, you can do: <code>a.gather(1, b.view(k, 1, 1).expand(k, 1, n))</code> will give you what you want <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
Basically, use the gather function, and since you want all the elements over the last dimension of size n, you just expand the Tensor of indices there.</p>
loss.backward() keeps running for hours

Hao_Xu: <p>I am using pytorch to train some x-ray images but I ran into the following issue:</p>
<p>in the line :  <code>loss.backward()</code> , the program just keeps running and never end, and there is no error or warning.</p>
<pre><code class="lang-auto">            loss, outputs = self.forward(images, targets)  
            loss = loss / self.accumulation_steps
            print("loss calculated: " + str(loss))

            if phase == "train":
                print("running loss backwarding!") 
                loss.backward()
                print("loss is backwarded!")
                if (itr + 1 ) % self.accumulation_steps == 0:
                    self.optimizer.step()
                    self.optimizer.zero_grad()
</code></pre>
<p>The loss calculated before this is something like  <code>tensor(0.8598, grad_fn=&lt;DivBackward0&gt;)</code> .</p>
<p>Could anyone help me with why this keeps running or any good ways to debug the backward() function?</p>
<p>I am using torch 1.2.0+cu92 with the compatible cuda 10.0.</p>
<p>Thank you so much!!</p>

albanD: <p>Hi,</p>
<p>This is quite unexpected.<br>
We would need more information though to reproduce this.<br>
Could you share a small code repro that runs on colab: <a href="https://colab.research.google.com/notebook#create=true">https://colab.research.google.com/notebook#create=true</a> ?</p>

Hao_Xu: <p>I upgraded my pytorch to the latest version which is now:  <strong>torch 1.4.0 torchvision 0.5.0 with cuda 10.1.</strong> . And now it runs pretty smooth.</p>
<p>thank you !</p>
Video card parameter that leads to faster network learning

Chame_call: <p>I’d like to know the following:<br>
If we have two different video cards(cuda compatible) how to find out which of them will lead quicker network training.</p>

tom: <p>Given that they both are likely to compute the same results, mostly by timing, using %timeit in Jupyter or jost the Python <code>time.time()</code> before and after. The one thing to get right is to do <code>torch.cuda.synchronize()</code> before you start timing and the calculation and after one batch (or whatever) is done.</p>
<p>Now, when you have different amounts of memory and that affects your batch size or one card is fp16-capable and not (e.g. GTX cards are not), you probably want to inlcude that in your estimate.</p>
<p>Best regards</p>
<p>Thomas</p>

Chame_call: <p>Is amount of FLOPS not playing any role in this?<br>
or cuda cores?</p>

tom: <p>Cuda cores, memory bandwidth, software support, task characteristics. But really it’s complicated enough that people usually do empirical measurements. If you want simple and both cards are still on the market, you could go by $$$.</p>
Imagenet Train/Val Split

crnyu: <p>Can anyone share just the imagenet train/val split? I have the dataset, but cannot get the split because I am having trouble getting them to approve my access to the download.</p>
<p>Thanks!</p>

crnyu: <p>This appears to be the validation split: <a href="https://raw.githubusercontent.com/soeaver/caffe-model/master/cls/ILSVRC2012_val.txt" rel="nofollow noopener">https://raw.githubusercontent.com/soeaver/caffe-model/master/cls/ILSVRC2012_val.txt</a>.</p>
The gradients of the first few conv layers are None in a custom model

astrogrl: <p>I have defined a custom layer which takes in features from the CONV layers and will output a loss and an output using the CONV inputs. The output from this layer then goes into the FC layers for classification. The forward method is described as follows:</p>
<pre><code class="lang-auto">def forward(self, x):
        x = F.relu(F.max_pool2d(self.conv1(x), 2))
        x = F.relu(F.max_pool2d(self.conv2_bn(self.conv2(x)), 2))
        x = x.view(-1, 500)
        loss, x = self.custom_layer(x)
        x = F.relu(self.fc1(x))
        return loss, F.log_softmax(x, dim = 1)
</code></pre>
<p>Two different optimizers (both SGD) are used, one of them is for the parameters of the custom layer while the other is for the CONV and FC layers. While training however, the losses from the custom layer and the cross-entropy loss from the model is backward propagated.</p>
<pre><code class="lang-auto">optimizer1.zero_grad()
optimizer2.zero_grad()
loss1, outputs = model(inputs)
loss2 = criterion(outputs, labels)
total_loss = loss1 + loss2
total_loss.backward()
optimizer1.step()
optimizer2.step()
</code></pre>
<p>When I try to visualize the gradients, the gradients of the CONV layers are None while the gradient is being updated for the FC and custom layer, even though the requires_grad flag is true for all the layers. Is there anything else I need to define for the gradient to flow through the entire network?</p>

albanD: <p>You need to make sure that your custom layer does not break the flow of the gradients for its inputs.<br>
In particular, you should not <code>.detach()</code> or unwrap/rewrap the tensor that is given as input (or use <code>.data</code> but that is a general rule everywhere).</p>
<p>Can you share the forward code for your custom_layer?</p>

astrogrl: <p>Yes, you are right, I was detaching a variable at some point in the forward code of the custom_layer. Thank you!</p>
Matrix multiplication source code?

bdusell: <p>I’m interested in finding out some specific implementation details of matrix multiplication in PyTorch. Where would one find the source code (CPU implementation and CUDA kernel) for PyTorch’s implementation of matrix multiplication? Specifically, where would one find the code implementing <a href="https://pytorch.org/docs/stable/torch.html#torch.bmm" rel="nofollow noopener">torch.bmm()</a>?</p>
<p>I see that this question was already asked <a href="https://discuss.pytorch.org/t/matrix-matrix-multiply-source-code/41634">here</a>, but not answered.</p>
<p>Thanks!</p>

tom: <p>Here is a <a href="https://lernapparat.de/selective-excursion-into-pytorch-internals/" rel="nofollow noopener">blog post</a> how to get from Python PyTorch function to ATen.<br>
For CPU, this will get you to <a href="https://github.com/pytorch/pytorch/blob/758ad516f32708de243f194144ee0f7b9e0f5117/aten/src/ATen/native/LinearAlgebra.cpp#L296" rel="nofollow noopener"><code>bmm_cpu</code></a> in LinearAlgebra.cpp. This in turn will call <a href="https://github.com/pytorch/pytorch/blob/758ad516f32708de243f194144ee0f7b9e0f5117/aten/src/ATen/native/LinearAlgebra.cpp#L201-L210" rel="nofollow noopener"><code>bmm_out_or_baddbmm_</code></a> in the same file. For very small operands it has a (somewhat lame) kernel it calls, for larger (floating point) operands it dispatches to MKL if available or it will use several <code>mv</code>.<br>
For cuda, you end in <a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/cuda/LinearAlgebra.cu" rel="nofollow noopener">LinearAlgebra.cu</a> which (at the time of writing this) forwards to the THC legacy implementation <a href="https://github.com/pytorch/pytorch/blob/758ad516f32708de243f194144ee0f7b9e0f5117/aten/src/THC/generic/THCTensorMathBlas.cu#L525" rel="nofollow noopener"><code>baddbmm</code></a>. For technical reasons (the old Torch7 “generic” macro based tensor dtype support) that in turn uses functions from <a href="https://github.com/pytorch/pytorch/blob/master/aten/src/THC/THCBlas.cu" rel="nofollow noopener">THCBlas.cu</a> to finally dispatch to cuBlas or rocBlas.</p>
<p>Phew. Now I know why no one answered in the question you linked.</p>
<p>Best regards</p>
<p>Thomas</p>

bdusell: <p>Wow, very thorough <a class="mention" href="/u/tom">@tom</a> ! I really appreciate your taking the time. It’s quite the labyrinth, isn’t it.</p>
Mistake in MSELoss documentation

simaiden: <p>MSELoss documentation  (<a href="https://pytorch.org/docs/stable/nn.html#mseloss" rel="nofollow noopener">link</a>) seems to have a mistake at:</p>
<pre><code class="lang-auto">The **sum** operation still operates over all the elements, and divides by n .

The division by n can be avoided if one sets `reduction = 'sum'` .
</code></pre>
<p>I guess instead of “sum operation” it would be “mean operation”.</p>
Inspecting a tensor in Visual Studio

G_B: <p>Hi,</p>
<p>I found very hard to inspect a tensor (sizes, dim and data) through the watcher of the Visual Studio IDE. I can read the data with the following line (the tensor variable is <code>myTensor</code> with numel() == 200):</p>
<p><code>((float *)myTensor.impl_.target_-&gt;storage_.storage_impl_.target_-&gt;data_ptr_.ptr_.data_),200</code></p>
<p>but I found no way to watch the sizes, for example. If I try to print the sizes through the immediate window, I obtain the message (although I’m in debug configuration):</p>
<p><code>Function at::Tensor::sizes has no address, possibly due to compiler optimizations.</code></p>
<p>This is really too frustrating. Any solution to this?</p>
<p>Thanks in advance.</p>

G_B: <p>Ok, I can inspect sizes with (if dim() == 2):</p>
<p><code>(int64_t *)(*((c10::AlignedCharArray&lt;8,8&gt;*)&amp;(*((c10::SmallVectorTemplateCommon&lt;__int64,void&gt;*)&amp;(*((c10::SmallVectorTemplateBase&lt;__int64,1&gt;*)&amp;(*((c10::SmallVectorImpl&lt;__int64&gt;*)&amp;myTensor.impl_.target_-&gt;sizes_)))))).FirstEl)).buffer,2</code></p>
<p>Phew, what an ugly expression! IMHO this should be improved absolutely (at least in debug configuration).</p>
What is self.state used for in optimizers?

pinocchio: <p>I saw that field but when I try to print things from it in my code it looks empty…</p>
<pre><code class="lang-auto">meta_opt.state
defaultdict(&lt;class 'dict'&gt;, {})
</code></pre>
<p>what is its purpose?</p>
<hr>
<p>related:</p>
<ul>
<li>what is <code>self.state</code>: <a href="https://discuss.pytorch.org/t/what-is-self-state-used-for-in-optimizers/71182" class="inline-onebox">What is self.state used for in optimizers?</a>
</li>
<li>is the update func I wrote ok: <a href="https://discuss.pytorch.org/t/how-do-i-load-a-new-copy-of-my-parameters-to-be-trained-in-the-optimizer-without-breaking-it-in-a-subtle-way/71184" class="inline-onebox">How do I load a new copy of my parameters to be trained in the optimizer without breaking it in a subtle way?</a>
</li>
</ul>
How do I load a new copy of my parameters to be trained in the optimizer without breaking it in a subtle way?

pinocchio: <p>I am using higher and there is an unusual bug where my optimizer stops having a pointer to the real parameters I am trying to update (details here: <a href="https://github.com/facebookresearch/higher/issues/32" rel="nofollow noopener">https://github.com/facebookresearch/higher/issues/32</a>). I am trying to overcome it by giving a pointer to the new (unfortuantely deep copied) versions of my parameters so that the optimizer can optimize (in place) the right thing.</p>
<p>So I made this function:</p>
<pre><code class="lang-auto">    def load_new_params(self, params):
        self.param_groups = []

        param_groups = list(params)
        if len(param_groups) == 0:
            raise ValueError("optimizer got an empty parameter list")
        if not isinstance(param_groups[0], dict):
            param_groups = [{'params': param_groups}]

        for param_group in param_groups:
            self.add_param_group(param_group)
</code></pre>
<p>which I believe should work but I noticed there was a weird <code>self.state</code> field in optimizers which worries me that is not being tracked correctly (or something else that is subtle).</p>
<p>Is doing:</p>
<pre><code class="lang-auto">meta_opt.load_new_params(new_params)
meta_opt.step()
</code></pre>
<p>enough to guarantee that the right version of the parameters are updated correctly and no other unexpected bug is happening due to <code>self.state</code> or otherwise?</p>
<hr>
<p>related:</p>
<ul>
<li>what is <code>self.state</code>: <a href="https://discuss.pytorch.org/t/what-is-self-state-used-for-in-optimizers/71182" class="inline-onebox">What is self.state used for in optimizers?</a>
</li>
<li><a href="https://github.com/facebookresearch/higher/issues/32" rel="nofollow noopener">https://github.com/facebookresearch/higher/issues/32</a></li>
</ul>
Image segmentation for small matrix (32x32)

Robin_Lobel: <p>I’m new to machine learning, and I’d like to do image segmentation on small matrix (1x32x32 patterns). Unfortunately it seems that most networks (Unet, ResNet, etc) are designed to handle large pictures (3x256x256 and more) and I have no idea how to scale them down to my size and if that’s even a good idea.</p>
<p>What would be the simplest network to start with for such small patterns ? Basically I’m trying to discriminate noise from feature in those patterns, say which are noise pixels and which are feature pixels (so 1x32x32 as input, 1x32x32 as output). I can randomly generate as many noise+feature patterns and masks as needed for the training. I’m using the C++ version of PyTorch, libtorch.</p>
<p>thanks !</p>

tom: <p>You could try a UNet-like architecture without the downscaling, but with a similar arrangement of skip connections (i.e. have one from the first to the last, one from second to one-but last). You probably don’t need very many layers…</p>
<p>Best regards</p>
<p>Thomas</p>

Robin_Lobel: <p>Thanks, so basically, this architecture could work ? -&gt; <a href="https://i.imgur.com/mzcJTP4.png" rel="nofollow noopener">https://i.imgur.com/mzcJTP4.png</a><br>
I’ve made this mockup by tweaking the original UNet pdf figure to fit my case. I wasn’t sure about the number of channels to start with (right after the very first layer), the original paper was using 64, I set it arbitrarily to 32. Any comment on that ?</p>
<p>As for the resolution, my original 32x32 resolution is not critical, it just carries some extra subtilities. In my case 16x16 is the lowest resolution that carries the information (the ability to discriminate between noise and signal). However I can still benefits from the fine details of the 32x32, so I hope my proposed UNet can still benefit from it, even if one level of downsampling occurs ?</p>
Reading equations in documentation

Jeremy_Rutman: <p>I’d like to be able to see the formatted equations in the documentation e.g.  <a href="https://pytorch.org/docs/stable/_modules/torch/nn/modules/loss.html" rel="nofollow noopener">here</a> , is there some simple way to do so?</p>

mmisiur: <p>You linked the source code for losses, but in most cases there is great general documentation of PyTorch modules and functions, which is formatted with user-friendly style - for the losses (so the code that you’ve linked to) it would be <a href="https://pytorch.org/docs/stable/nn.html#loss-functions" rel="nofollow noopener">here</a>.</p>
What is an idiomatic way to access the encoder layer in an autoencoder?

zeets: <p>Lets say I have an autoencoder class</p>
<pre><code class="lang-auto">class Autoencoder(nn.Module):
  def __init__(self):
    self.encoder = nn.Sequential(...)
    self.decoder = nn.Sequential(...)

  def forward(self, x):
    x = self.encoder(x)
    x = self.decoder(x)
    return x
</code></pre>
<p>I have trained this model to my satisfaction. I now have some inputs I want to just encode. I have thought of two solutions:</p>
<ul>
<li>adding a method that return self.encoder(x), or</li>
<li>separating this class into two classes</li>
</ul>
<p>Are those approaches idiomatic?</p>

tom: <p>If you have <code>ae = Autoencoder()</code> just using <code>ae.encoder(x)</code> is fine. Also <code>enc = ae.encoder</code> will give you the encoder module and it’ll work on it’s own, too (you can even save and load etc.).</p>
<p>Best regards</p>
<p>Thomas</p>
CNN training on multiple images

R_Tux: <p>Hi, I am quite new to PyTorch.<br>
I am trying to set up a simple Convolutional Neural Network (CNN) but am having trouble training on multiple images. The images are ones i create myself have have size = (50,50), I can create as many as i want.</p>
<p>Currently I have been training on a single image by transforming to</p>
<pre><code class="lang-python">X = torch.tensor(image, dtype=torch.float)
X = X.view(1,1,image.shape[0],image.shape[1])
</code></pre>
<p>before training. So the input is of size <code>(1,1,50,50)</code>.<br>
My question is how to I expand my code to train on different images?</p>
<p>If possible could you give a general example of what I need to do? I have searched the forums but to no avail.</p>
<p>cheers</p>

G.M: <p>What do u mean by training on “different images”? If u mean more images in a single batch, u can concatenate/stack multiple  images together. Pytorch 2d conv expects an input of (B, C, H, W) where B is the batch size. If each images are of shape (1, 1, H, W), u can do <code>tc.cat(IMAGES)</code>; if it’s of shape (H, W), u can do <code>tc.stack(IMAGES).unsqueeze_(1)</code>.</p>
How to average weights from different saved epochs?

Nils: <p>I’m training a model and want to load the last three saved checkpoints and do an average/mean of the weights and save it into one new average model/weight, all are from the same architecture and trained data, idea?</p>

Nils: <p>I can easily do it in Keras/TF, more experience in that area but need someone here who is more experienced in Pytorch.</p>

sdv4: <p>A simple way to go about this would be to load each checkpoint in succession, add the parameter values of each into appropriately sized tensors, and then divide by 3 to get the mean.</p>
<p>A simple 1 layer example would be:</p>
<pre><code class="lang-auto">layer_1 = 0
for param in model.named_parameters():
    if param[0] == 'fc.weight':
        layer_1 = torch.zeros_like(param[1].data)
</code></pre>
<p>Then for each checkpoint do:</p>
<pre><code class="lang-auto">for param in model.named_parameters():
    if param[0] == 'fc.weight':
        layer_1 += param[1].data
</code></pre>
<p>Now divide layer_1 by three, create a new model instance, and run:</p>
<pre><code class="lang-auto">for param in model.named_parameters():
    if param[0] == 'fc.weight':
        param[1].data = layer_1
</code></pre>
<p>There is probably a much more elegant way to do this, but this is what comes to mind for me.</p>

Nils: <p>I found an already implemented technique for it – ‘Stochastic Weight Averaging’ and it works!</p>
Nums of AnchorGenerator output anchors

Deltap: <p>I need to know what feature location every anchor box is corresponding to, so I modify the <code>torchvision.models.detection.rpn.AnchorGenerator</code> in this way:</p>
<pre><code class="lang-auto">class AnchorGeneratorMP(AnchorGenerator):
    def __init__(
            self,
            sizes=(128, 256, 512),
            aspect_ratios=(0.5, 1.0, 2.0),
    ):
        super(AnchorGeneratorMP, self).__init__(sizes, aspect_ratios)

    def forward(self, image_list, feature_maps):
        grid_sizes = tuple([feature_map.shape[-2:] for feature_map in feature_maps])
        image_size = image_list.tensors.shape[-2:]
        strides = tuple((image_size[0] / g[0], image_size[1] / g[1]) for g in grid_sizes)
        self.set_cell_anchors(feature_maps[0].device)
        anchors_over_all_feature_maps = self.cached_grid_anchors(grid_sizes, strides)
        anchors = []
        for i, (image_height, image_width) in enumerate(image_list.image_sizes):
            anchors_in_image = []
            for anchors_per_feature_map in anchors_over_all_feature_maps:
                anchors_in_image.append(anchors_per_feature_map)
            anchors.append(anchors_in_image)
        # here is the only place where I modified, except the returns
        anchor_len = [len(anchor) for anchor in anchors_over_all_feature_maps]
        anchors = [t.cat(anchors_per_image) for anchors_per_image in anchors]
        return anchors, anchor_len
</code></pre>
<p>Conforming to the <code>FasterRCNN</code> implemented by pytorch, the <code>sizes</code> and <code>aspect_ratios</code> are</p>
<pre><code class="lang-auto">anchor_sizes = ((32,), (64,), (128,), (256,), (512,))
aspect_ratios = ((0.5, 1.0, 2.0),) * len(anchor_sizes)
</code></pre>
<p>So it is supposed to output 15 anchors per feature location. What makes me confused is that for every <code>anchor_len</code>, it is only 3 times the total locations of the corresponding feature. Where am I missing?</p>

Deltap: <p>And also, is it correct to get the corresponding feature location of every anchor in this way? It is in a ResNet50+FPN backbone Faster RCNN</p>
<pre><code class="lang-auto">anchor_sizes = ((32,), (64,), (128,), (256,), (512,)), 
aspect_ratios = ((0.5, 1.0, 2.0),) * len(anchor_sizes)
ag = AnchorGeneratorMP( anchor_sizes, aspect_ratios)
anchors, anchor_len = ag(image_list, features)
features = list(features.values())
anchor = anchors[:anchor_len[0]]
anchor = anchor.reshape(features[0].size()[-2:])
# is anchor[i][j] corresponding to features[0][:][:][i][j]?
</code></pre>
Which is better 3D Conv or CNN+LSTM

Surekha_Gaikwad: <p>I am working on project where I am trying to process video data but I would like to know which is better to use either 3D Conv or CNN+LSTM?</p>

G.M: <p>If u have enough amount of data and a good gpu, I would suggest using 3d conv+ LSTM.</p>

Surekha_Gaikwad: <p>Do you mean by 2d conv + LSTM? Please correct me if I am wrong<br>
Can I not directly use only 3D conv to extract features from videos and use it for further classification?<br>
I would like to know apart from mentioned reasons at what level does two of them differ? (like better feature extraction at both spatial and temporal level or anything else)</p>

G.M: <p>No, I mean 3d conv with lstm.<br>
Yes, u can use 3d conv to extract features directly. Conv is able to recognize spatial patterns; for example conv for pictures can recognize edges and curves and etc. LSTM is good at recognizing temporal information. A LSTM can remember information before the current time steps; for example, LSTM is used in machine translation because we can’t words later in the sentence can have a relation with words in the beginning of the sentence.</p>

Surekha_Gaikwad: <p>okay thanks got it <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Setting multiple kernels for multiple size of input CNN

jerry0098: <p>Hi, I have a CNN code like this:</p>
<pre><code>	CNN_modules = []
	CNN_modules.append(nn.Conv2d(1, 256, (x, 33)))
	CNN_modules.append(nn.ReLU())
	
	CNN_modules.append(nn.Conv2d(256, 256, (17, 17)))
	CNN_modules.append(nn.ReLU())

	CNN_modules.append(nn.Conv2d(256, 256, (9, 9)))
	CNN_modules.append(nn.ReLU())

	CNN_modules.append(nn.Conv2d(256, 256, (5, 5)))
	CNN_modules.append(nn.ReLU())

	CNN_modules.append(nn.Conv2d(256, 256, (3, 3)))
	CNN_modules.append(nn.ReLU())

	CNN_modules.append(nn.Conv2d(256, 256, (2, 2)))
	CNN_modules.append(nn.ReLU())
</code></pre>
<p>The problem is that the input data of that CNN has different heights, for example, they could be [256, 64], [93, 64], … so the question is how can I set “x” in the first layer, which makes all of the input be [32,32] in the second layer?</p>
<p>Thank you so much</p>

tom: <p>The usual way is to do rescaling of the inputs or <a href="https://pytorch.org/docs/stable/nn.html#adaptiveavgpool2d" rel="nofollow noopener">AdaptiveAvgPool2d</a> (or the MaxPool equivalent) somewhere later.</p>
<p>Best regards</p>
<p>Thomas</p>
Matrix-Matrix multiply source code

symashayak: <p>Hi,</p>
<p>Where can I find the source code of matrix-matrix multiply kernel that PyTorch uses?</p>
<p>Thank you!</p>

tom: <p>In the new thread I’m chasing things through the PyTorch code for bmm:</p>
<aside class="quote" data-post="1" data-topic="71071">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/bdusell/48/17300_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/matrix-multiplication-source-code/71071">Matrix multiplication source code?</a> 
  </div>
  <blockquote>
    I’m interested in finding out some specific implementation details of matrix multiplication in PyTorch. Where would one find the source code (CPU implementation and CUDA kernel) for PyTorch’s implementation of matrix multiplication? Specifically, where would one find the code implementing <a href="https://pytorch.org/docs/stable/torch.html#torch.bmm" rel="noopener nofollow ugc">torch.bmm()</a>? 
I see that this question was already asked <a href="https://discuss.pytorch.org/t/matrix-matrix-multiply-source-code/41634">here</a>, but not answered. 
Thanks!
  </blockquote>
</aside>
None type for gradient?

pytorchnewbie: <p>Hi! I’m having trouble understanding why I’m getting nonetype for my “resampled” function right after I call loss.backward().</p>
<p>Below is the code:</p>
<pre><code class="lang-auto">def problem_l2loss(data,target):

  not_done = True
  learning_rate=0.07
  count = 1
  cd = torch.tensor([0,0]).type(torch.FloatTensor)
  cd = torch.unsqueeze(cd,dim=1).cuda()
  scale_a = torch.unsqueeze(torch.tensor([.5]).type(torch.FloatTensor),dim=1).cuda()
  cd.requires_grad_()
  scale_a.requires_grad_()
  
  while(not_done):

    one_zero = torch.tensor([1,0]).cuda()
    zero_one = torch.tensor([0,1]).cuda()
    temp_a = one_zero*scale_a
    temp_b = zero_one*scale_a
    two_by_two = torch.cat((temp_a,temp_b),dim=0)   

    M= torch.cat((torch.tensor(two_by_two),cd),dim=1)[None,:,:]
    #The None,:,: is for the batch size
    pdb.set_trace()
    
    grid = F.affine_grid(M,data.shape)
    grid.requires_grad_()
    resampled = F.grid_sample(data.cuda(), grid, mode='bilinear')[0]

    criterion = torch.nn.MSELoss()

    loss = -1*criterion(resampled, batch_cat_alone.cuda())
    loss.backward()
    #why is the gradient of resampled None here?

    scale_a_grad = scale_a.grad.data
    cd_grad = cd.grad.data

    scale_a.data = scale_a.data + learning_rate*scale_a_grad
    cd_grad.data = cd_grad.data + learning_rate*cd_grad_grad

    count = count + 1
  
    if count == 100:
      print("here")
      not_done = False
    
  return 
</code></pre>
<p>Further comments: I made sure to send everything to cuda before saying that I wanted gradient_true() on all my variables. I also checked, and if I multiply two variables where one has gradient_true and the other I didn’t explicitly set, the product has gradient_true. Also, “resampled” has resampled.requires_grad = True, yet it has none type for resampled.grad … Obviously scale_a and cd also have None type, despite both also having requires_grad = True when I check them after loss.backward() with a pdb trace… Thanks!</p>

tom: <p>By default, only leaves, but not intermediate results get gradients. You can do <code>resampled.retain_grad()</code> to get intermediate gradients.</p>
<p>Best regards</p>
<p>Thomas</p>
Tensorboard for pytorch 0.4.1

Vishnu_Gopireddy: <p>Is there any way to use tensorboard for pytorch 0.4? I came to know that tensorboard comes with newer versions of pytorch.</p>

jmandivarapu1: <p>I know there is package called tensorboardX. (<a href="https://github.com/lanpa/tensorboardX" rel="nofollow noopener">https://github.com/lanpa/tensorboardX</a>).<br>
I used to use this before the official version came and I assume it should work for all pytorch versions.</p>
How to serialize torch::jit::IValue or torch::Tensor to protobuffer?

sigmoid: <p>I want to develop a torch-serving backend with libtorch.<br>
as the libtorch c++ frontend example shows:</p>
<pre><code class="lang-auto">torch::jit::script::Module module;
  try {
    // Deserialize the ScriptModule from a file using torch::jit::load().
    module = torch::jit::load(argv[1]);
  }
  catch (const c10::Error&amp; e) {
    std::cerr &lt;&lt; "error loading the model\n";
    return -1;
  }
  std::cout &lt;&lt; "ok\n";

  std::vector&lt;torch::jit::IValue&gt; inputs;
  inputs.push_back(torch::ones({1, 3, 224, 224}));
  // Execute the model and turn its output into a tensor.
  at::Tensor output = module.forward(inputs).toTensor();
  std::cout &lt;&lt; output.slice(/*dim=*/1, /*start=*/0, /*end=*/5) &lt;&lt; '\n';
</code></pre>
<p>now I can load traced model with libtorch , and call module.forward to do predict.<br>
however the predict requests with inputs are coming from network by another client like python/java/c++ &amp;.<br>
so I need serialize the request from the client, and deserialize the request at the server backend, now I choose protobuffer to do the thing.<br>
but I’m confused about which protodef file I can use to define the   torch::jit::IValue or torch::Tensor ??</p>
<p>Is there anyone can help me ???</p>
Why can't I successfully open an exe(a littel game) by os.system

GROX: <p>here is the code<br>
import os<br>
os.system(‘D:\游戏ai实验\elonaplus_free.exe’)<br>
and the result:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/d/7/d742003975a1e63ed572007b8a25ed6dd40184ea.png" alt="9P_8HDMF`(~XTQ_H(~J%HM" data-base62-sha1="uIg3RLccVa0c21ePBYpnGk7eclA" width="296" height="72"></p>
<p>which means the “Initializing   elona.dll not exist”<br>
but i double click the exe, it work successfully.<br>
that’s why and what’s the right way for an NN to get the information from an exe such as a computer game?</p>

ptrblck: <p>What kind of information from the executable are you looking for?</p>

GROX: <p>For example , I want  NN get the Game‘s picture,frame by frame, like our human watch the computer screen when we are playing game.(Sorry, I really don’t know what the technical term for this kind of information）</p>

ptrblck: <p>Injecting into a running program to grab the frames doesn’t sound like a trivial task and it won’t work by simply open an executable.<br>
I would recommend to create the frames by a capturing software e.g. fraps while your game is running.</p>

GROX: <p>THANKS,<br>
you are my hero !</p>
How to tradeoff mean square loss with binary cross entropy loss

Praveen_raja_sekar: <p>Hello,<br>
I am working on domain adaptation for regression. So my predictive loss function is mean square loss and domain loss is binary cross-entropy loss. When I am training the network by jointly optimizing both the loss functions with the same optimizer and learning rate, my mean square loss converges soon and domain loss gets saturated in a few epochs and does not get converged latter. Therefore, I am not getting an improvement in performance using domain adaptation. COuld you please help me on how to fix this problem. Should I use different learning rates for the loss functions? If so how should I choose the learning rate decays of bothe loss functions and how can I use different learning rates. Should I initialize different optimizers? You help will be greatly appreciated.</p>
Affine transformation question

pytorchnewbie: <p>Hi! I’m trying to learn the correct parameters for an image’s scale and crop. If I want to learn the correct parameters for a crop, that means scale + translation. So would my parametrization be:</p>
<p>[a 0 c<br>
0 e f]</p>
<p>or</p>
<p>[a 0 c<br>
0 a f]</p>
<p>or</p>
<p>[1 0 c<br>
0 1 f]    all multiplied by a scalar lambda, where lambda is a parameter?</p>
<p>Thanks!</p>
How to train pre-trained model on multi-class classification?

Amr_Hesham: <p>I am new at using the torch library to make baseline models. so I tried to train a pre-trained model on multi-class classification with a custom loss function called focal loss. so my question is there anything wrong with my code? because when I try to train the model the training loss, test loss, and test accuracy not changing,  so did I calculate training loss, test loss, and test accuracy wrong? or did I use focal loss wrong or maybe both and this is my [code]. (<a href="https://colab.research.google.com/drive/1f3CY8xEWZ3i61QCwUMVnPm2JXWvlH2HA" rel="nofollow noopener">https://colab.research.google.com/drive/1f3CY8xEWZ3i61QCwUMVnPm2JXWvlH2HA</a>)</p>
Tracing down to C++ code about implementation of GPUs and CPU communication /synchronization on DataParallel module

zzsunshine: <p>Hi all,</p>
<p>I am doing the multi-gpus test (eg. 4 gpus) on pytorch with DataParallel module.<br>
For the “gather” function in DataParallel module, it needs to collect the data from other GPUs. I want to know after this function, it may have communication/synchronization among GPU and CPUs.<br>
But i got stuck on tracing down to C++ code with the implementation of GPUs and CPU communication /synchronization.  I cannot find the correct place.  I am already tracing down to " csrc/cuda/comm.cpp".</p>
<p>Thanks for any help.</p>
Deep NN Regression

asepehri93: <p>I am trying make my NN regression model overfit my data.<br>
I used the following settings:</p>
<p>x is of shape:  (12, 132399)<br>
y is of shape:  (1, 132399)<br>
The number of training examples m:   132399<br>
The number of features per examples:   12<br>
Relu activation<br>
He initialization<br>
Adam optimization</p>
<p>I have tried different learning rates, number of layers, nodes, and epochs, still overfitting is not happening. The best prediction I could get for my regression was R^2=0.6.</p>
<p>I appreciate if you let me know of the error in my code:</p>
<pre><code class="lang-auto">class NN(nn.Module):
    
    #Constructor
    def __init__(self,layers):
        super(NN,self).__init__()
        self.hidden=nn.ModuleList()
        
        for D_in, D_out in zip(layers,layers[1:]):
            linear_transform=nn.Linear(D_in,D_out)
            torch.nn.init.kaiming_uniform_(linear_transform.weight, nonlinearity='relu')
            self.hidden.append(linear_transform)
            
    #Prediction
    def forward(self,x):
        L=len(self.hidden)
        for l,transform in zip(range(L),self.hidden):
            if l&lt;L:
                x=relu(transform(x))
            else:
                x=transform(x)
        return x
</code></pre>
<pre><code class="lang-auto">def train(model, criterion, trainloader, optimizer,scheduler, epochs = 100):
    cost=[]
    total=0
    i=0
    for epoch in range(epochs):
        total=0
        
        for x,y in trainloader:
            optimizer.zero_grad()
            yhat=model(x)
            loss=criterion(yhat,y)
            loss.backward()
            optimizer.step()
            total+=loss.item()
            
        scheduler.step()
        i+=1    
        cost.append(total)
        print(str(i)+':   '+str(total))
    return cost   
</code></pre>
<pre><code class="lang-auto">layers=[12,200,200,200,1]
model=NN(layers)
criterion=nn.MSELoss()
lr=0.00003
optimizer=optim.Adam(model.parameters(),lr=lr)
milestones=[500,1000]
scheduler=optim.lr_scheduler.MultiStepLR(optimizer, milestones, gamma=0.6, last_epoch=-1)
cost=train(model,criterion,trainloader,optimizer,scheduler,epochs=1000)
</code></pre>

ptrblck: <p>What range do your features have?<br>
Have you tried normalizing them?<br>
Also, I assume you are passing the inputs as <code>[batch_size, nb_features]</code> to the model.</p>

asepehri93: <p>Thank you for your reply.<br>
Yes I normalized all my features having zero mean and 1 stdv.<br>
Also I create my train loader as below:</p>
<pre><code class="lang-auto">
trainloader = torch.utils.data.DataLoader(train_data, batch_size=64)

</code></pre>
<p>The over fitting happens very slowly regardless of the network size…</p>
Any build-in function to avoid apply an expensive softmax on padding tokens in a sequence

Zheng_Chen: <blockquote>
<p>This is not about how to get distribution (like attention weights) over a sequence excluding the paddings so that the normalization is only done over non-padding part of the sequence.</p>
</blockquote>
<p><strong>My problem is this</strong>, support you have a vector (embedding) sequence<br>
$x_1$, $x_2$, $x_3$, …, $x_n$, $p_1$, $p_2$, …, $p_m$</p>
<p>and I need to generate a probability distribution by softmax for each of the non-padding ones x1, x2, x3, …, xn. I don’t need distribution over the paddings p1, …, pm (usually they are all zero vectors)</p>
<p>Of course, I can generate the distributions for all then discard the last m distributions.<br>
But if the distribution is large, like a vocabulary distribution, 30000+, then softmax is very slow.<br>
and the sequence lengths vary a lot and so there are a lot of paddings in a batch</p>
<p><strong>Is there a build-in way to avoid computing softmax on the useless paddings</strong> ?</p>
PyTorch Cuda vs C++ Cuda

Kenzo: <p>Hi all,</p>
<p>I have a question about speed and performance in C++ and will appreciate that if anyone can answer my question! I’m doing instance segmentation with PyTorch + Cuda and the maximum Fps (frame per second) I can achieve is 12 Fps on full HD videos. I wonder if I convert my code to C++ with Cuda, What Fps should I expect? How many percentages will improve my Fps in C++ with Cuda?</p>

tom: <p>It depends a lot on what you are doing, but if it’s CNN/Unet-Style (so no for loops over individual pixels), my guess would be a rather very small speedup (&lt;10%, possibly much slower) from moving from Python to C++.</p>
<p>Best regards</p>
<p>Thomas</p>

Kenzo: <p>I’m doing instance segmentation with Yolact++ (<a href="https://github.com/dbolya/yolact" rel="nofollow noopener">https://github.com/dbolya/yolact</a>) to get the mask of objects in real-time. I wanted to know if I convert the code too C++ with Cuda, what will be my Fps approximately? because right now I can only achieve to 12-14 Fps on 1080p video which I want to reach to 25Fps. Thanks!</p>

tom: <p>So &lt;10% speedup over 12-14fps would be 12-16fps.<br>
Moving to C++ is not the efficiency gain you’re looking for.</p>
<p>Best regards</p>
<p>Thomas</p>

anilg: <p>You can import your trained model to C++ for the difference on Fps. I guess that it won’t increase as you expected. However, you can try ONNX inference tool for increasing the fps. Please let me know if it resolves your issue</p>
Choice of loss function

farhanrahman: <p>Hello there, I am training a network (ConvNet) which tries to regress values sized of 500x500x13 from whereas it has been trained on 3 channel RGB images (500x500x3). I have started using the vanilla unet and made a few small changes in it. I have a dataset of around 9000 images. By the way the values in those 12 channels are independent of each other. All those value range between 0-1. Each channel represent a certain property. (Like red channel shows red values throughout the channel, similarly each of those 12 channels on the output carry significance on there own).</p>
<p>However after starting the training, straight from epoch 1 the loss seems to go down. However the validation loss seems to go up for first 3/4 epochs then go down for 5/6 epochs more (go down slightly), then it flattens, maybe goes up or stays flat.</p>
<p>During this I was using sum of squared errors which would give a loss of lets say 100,000 or around that at the beginning of training since it is essentially summing up all the errors per pixel on each channel. Should I instead use regular MSE? The reason I did not do it is I get very small loss values from start if I do the mean (values like 0.08, 0.5 or as such).</p>
<p>Is there anything that I am doing very very wrong? Am I missing something very stupid here? Is my choice of loss very wrong? And why would I get a upward rising loss from epoch 1 to around epoch 4?</p>
<p>Your suggestions would be greatly appreciate and will make sure that I am at least on the right ballpark. Thanks!</p>

sdv4: <p>How are you splitting your data into training and validation sets? My hunch is that your data isn’t shuffled before splitting, which can result in your training set not being representative of the dataset as a whole.</p>

farhanrahman: <p>I am using random_split to get the train and validation set. Its a 90/10 split.</p>

sdv4: <p>I would try the following two things as initial trouble shooting:</p>
<ul>
<li>Can you try initializing with a different random seed? This will rule out that you did not just get an unlucky split of your data.</li>
<li>Can you try using a larger validation set? I would try a 70/30 split, just to ensure that the distribution of the val set is closer to that of the training set.</li>
</ul>
<p>Also, what optimizer are you using? There is nothing obviously wrong with your choice of loss, but I find that using MSE with sum or average reduction depends on if I am using SGD or something like Adam.</p>
Handle an imbalanced dataset

xinqi: <p>I have an imbalanced dataset for a classification task. There are 7 classes for output.</p>
<p>Each proportion is around 10%, 3%, 6%, 39%, 16%, 6%, 20%.</p>
<p>I use cross-entropy loss and I have tired weighted loss as [2, 8.5, 3.5, 4, 1.5, 4, 8], but it does not help.</p>
<p>The loss curve is clearly overfitting. I could achieve around 99% accuracy for training but, get an unchanged 70% for validation.</p>
<p>What else really helpful could I do?</p>

sai_m: <p>Hi, are you using accuracy as your metric? if so, please use f1-score as your metric and check train and valid set scores. As accuracy is not the right one to use for imbalanced data sets. Hope this helps</p>

anilg: <p>You can use python <strong>imgaug</strong> tool on your dataset and easily augment the dataset as you want. Also, you can use the weight parameter on your optimizer and that will affect your loss curve. Please let me know if it resolves your problem.</p>

xinqi: <p>Hi. I understand that F1-score would be able to make the metric more meaningful. However, would it be helpful for my neural network’s training?</p>

anilg: <p>Hi<br>
Yes. It will. For imgaug, it depends on your augmentation on the image. However, it may not work well on your dataset. For weights, If you don’t pass the weights the loss value, your loss value doesn’t really catch the real loss and won’t learn anything. Have you tried?</p>
Given an array of co-variance matrix; Convert each of them to correlation matrix in an efficient way without remove them from computation graph?

ElleryL: <p>Consider I have an array of matrix <code>M.shape = (N,D,D)</code> where <code>D</code> is the dimension of co-variance matrix and <code>N</code> is the number of co-variance matrices.</p>
<p>I have a method that convert a single co-variance matrix to a correlation matrix defined as</p>
<pre><code>def convert_to_correlation(m):
    inv = torch.inverse(torch.diag(torch.diag(m)).sqrt())
    r = inv@m@inv
    return r
</code></pre>
<p>When I have a list of such co-variance matrix I work as</p>
<pre><code>def convert_all_to_correlation(M):
    res = []
    for i in range(len(M)):
        m = convert_to_correlation(M[i])
        res.append(m)
    res = torch.stack(res)
    return res
</code></pre>
<p>There are two problems in above approach;</p>
<p>(1) The array could have large first dimension; hence very slow in iterative loop<br>
(2) My <code>M</code> is an output of neural network (hence part of computation graph); I don’t want my approach in <code>convert_to_all_correlation</code> remove them from graph.</p>
<p>Any help is appreciated! Thanks</p>

albanD: <p>Hi,</p>
<p>I’m a bit confused by your <code>convert_to_correlation</code>. It does not use <code>m</code>.</p>
<p>For 1: all the operations your use support batching (.diagonal, .inverse and matmul), so you can write a convert_to_correlation that works on the whole batch at once.<br>
For 2: we have an implementation of derivatives for inversion and all other functions you use. So the result will be differentiable <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> (You can check that res.requires_grad = True).</p>

ElleryL: <p>Sorry; I got typo there; I’ve fixed the typo.</p>
<p>I’ve also checked the <code>torch.diag()</code> method; it seems only work on 2D tensor ?</p>

albanD: <p>Yes, <code>.diag()</code> is an old api that only supports 2D Tensor. That is why I mentionned <code>.diagonal()</code> above that is the new api that supports batching (and diag of arbitrary dimensions which is super cool).</p>

ElleryL: <p>Thanks for the advice. However, I’m confused with <code>.diagonal()</code> usage here</p>
<p>for example:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; x
 tensor([[[1,  2],
          [ 3, 4]],

          [[5, 6],
          [7,  8]]])

  &gt;&gt;&gt; torch.diagona(x)

tensor([[1., 7.],
        [2., 8.]])

</code></pre>
<p>while I’m expecting with  <code>[[1,4],[5,8]]</code>. Am I missing anything?</p>

albanD: <p>You can check the <a href="https://pytorch.org/docs/stable/torch.html?highlight=diagonal#torch.diagonal">doc</a> for the function.<br>
If you want 1 4, 5 8, that means that you want the diagonal for the dimensions 1, 2 instead of the default 0, 1.</p>

ElleryL: <p>perfect ! Thanks for the help!</p>
Lmdb or H5py for data loading in python

yptheangel: <p>What is the current best practice for loading large image dataset (500GB) into pytorch?<br>
I have tried a lmdb way by using this <a href="https://github.com/Lyken17/Efficient-PyTorch" rel="nofollow noopener">repo</a> and the loading time improved as compared to the ImageFolder+DataLoader pair.</p>
<p>Given that hard disc space and multiprocessing are factors in consideration.</p>
<p>Thanks.</p>
Adding value to loss variable

yumu: <p>hello im currently working on a spiking neural network and trying to implement exponential regularization</p>
<p>the cost function on exponential regularization give a loss value per neuron that i then sum before adding it to the loss variable by doing loss = F.nll_loss(output, target) + lossum<br>
however doing this seems to do nothing even when i replace lossum by something like 200000 (wich should make the network behave completely diffently than normal) so i guess this isn’t the right way to do it</p>
<p>how can i add a value to my loss so that it actually is taken into account when backpropagating ?</p>

albanD: <p>Hi,</p>
<p>This value is taken into account.<br>
But if it is constant,<code> grad(loss + C) = grad(loss) + grad(C ) = grad(loss)</code>. So adding a constant to your loss won’t change the gradients you get.</p>

yumu: <p>thank you i guess that makes sense my regularization attempt had a nearly constant value to so i guess thats why it didn’t do anything either</p>
Amp delay unscale with apex ddp

krishansubudhi: <p>I am applying amp delay unscale to accumulate gradient. I am also using apex DDP for doing allreduce across processes. I am disabling apexDDP during gradient accumulation using disable_allreduce(self) function.</p>
<p>Just before forward pass of the iteration where I want to reduce my gradient, I call enable_allreduce(self) on apex DDP, set delay_unscale to false, call backward() on scaled loss, clip the gradients and  then step the optimizer.</p>
<p>Is this a correct approach?</p>