akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
DataLoader for strings in multi-gpu training

username: Han_Brian_Lee
cooked: <p>Hi,</p>
<p>I have data that consists of numbers as strings, i.e.:<br>
[image c x w x h tensor, image c x w x h tensor, groundtruth number c x w x h tensor, strings],</p>
<p>where tuple of strings is not just “cat”, “dog” etc that can be encoded to 0,1,…, but just some information about where the tensor data came from (i.e. “/data/asdf/qwer.png”).</p>
<p>I’ve been successfully using Pytorch’s Dataset an DataLoader to load the data (excluding strings part) onto 4 GPUs I have. (i.e. with batch size of 32, each gpu gets the 2 image tensors and groundtruth number tensor in size of 8).</p>
<p>Then I wanted to also feed in the strings data as well because I needed to do some processing based on that in forward method of my network, and realized strings cannot be turned into tensors. <strong>The DataLoader is able to group randomly sampled 32 of the strings into a batch successfully, but each of the 4 GPUs are getting the exact same 32 strings, instead of 8 strings into each GPU.</strong></p>
<p>Is there any way this can be done?</p>
<p>Thanks,</p>
Torch.no_grad works for operations only

username: gsasikiran
cooked: <p>In the following code, I see, <code>requires_grade</code> is  False for <code>a**2</code> but True for <code>a</code> even after using <code>torch.no_grad()</code>. Why is <code>a.requires_grad</code> is still True?</p>
<pre><code class="lang-auto">print(a.requires_grad)
print((a**2).requires_grad)
with torch.no_grad():
  print(a.requires_grad)
  print((a**2).requires_grad)
</code></pre>
<p>Output:</p>
<pre><code class="lang-auto">True
True
True
False
</code></pre>

username: albanD
cooked: <p>Hi,</p>
<p><code>torch.no_grad()</code> only changes the behavior of pytorch ops by forcing them not to track gradients.<br>
So if you do no op, it has no effect.<br>
It is unclear if we want to change that in the future (to make a.requires_grad print False here), but this is the current behavior.</p>

username: gsasikiran
cooked: <p>Is there any reason to keep it the same way?</p>

username: albanD
cooked: <p>The main one is that there is no consensus that this is the right way to do it.<br>
Having <code>t.requires_grad</code> change based on a global flag that could be controlled from other code could be quite surprising to the user as well.<br>
Also many people are now used to the current behavior so we want to have a solid argument if we change it.</p>
How to write a custom cpu kernel

username: nikhilmishra000
cooked: <p>I have a custom op for which I implemented a cuda kernel (following the instructions in <a href="https://pytorch.org/tutorials/advanced/cpp_extension.html" rel="nofollow noopener">https://pytorch.org/tutorials/advanced/cpp_extension.html</a>).</p>
<p>Now I want to have a CPU implementation of the same op:</p>
<ul>
<li>The tutorial shows how to call methods of Tensor from c++, but my op cannot be decomposed into built-in functions (need the equivalent of a cuda kernel on cpu).</li>
<li>Can I still use <code>tensor.packed_accessor&lt;...&gt;()</code> and <code>AT_DISPATCH_FLOATING_TYPES</code>, and then just call my own c++ function instead of invoking a cuda kernel?</li>
<li>Are there any macros/helper functions for vectorization / multi-threading? or should i just use OpenMP or similar myself?</li>
<li>Any other docs, resources, or examples in the codebase that would be helpful?</li>
</ul>

username: Kaii
cooked: <ul>
<li>For CPU tensor, I think the equivalent would be Tensor.accessor&lt;…&gt;<br>
or you can use a vanilla Tensor.data_ptr&lt;…&gt;.</li>
<li>You can use <code>at::parallel_for</code> to enable OpenMP or other multi-threading libraries. <a href="https://discuss.pytorch.org/t/using-at-parallel-for-in-a-custom-operator/82747" class="inline-onebox">Using at::parallel_for in a custom operator</a>
</li>
</ul>

username: nikhilmishra000
cooked: <p>Thanks, this was really helpful! Got it working.</p>

username: tom
cooked: <p>Note that for largeish operations, using accessors is really inefficient - you use element by element arithmetic. You may see much better results by using vectorized (AVX, NEON, …) intrinsics even if that means that you need to take care of contiguous tensors yourself.</p>
<p>Best regards</p>
<p>Thomas</p>

username: nikhilmishra000
cooked: <p>So just to clarify, the inefficiency is because accessors preclude the use of vectorized intrinsics?</p>
<ul>
<li>Suppose that I made the tensors contiguous &amp; operated on the pointer directly, but did not use intrinsics</li>
<li>In this case, I would have to do the index arithmetic myself</li>
<li>Using accessors should be at least as good as doing this?</li>
</ul>

username: tom
cooked: <p>So the tensor accessors are a really more or less equivalent to doing pointers and pointer arithmetic (this is why the dimension is part of the template, really).<br>
If you have a really good compiler, it might do the vectorization, but last I tried, writing intrinsics got things to run faster.</p>
<p>Best regards</p>
<p>Thomas</p>
Error when transforming an image during prediction

username: Frank_Ofoedu
cooked: <p>Hello. I deployed a trained model via flask on heroku. I noticed my predict method keeps failing at the point of applying a transform to an input image.</p>
<p>This is the error message</p>
<p><strong>RuntimeError: The size of tensor a (4) must match the size of tensor b (3) at non-singleton dimension 0</strong></p>
<p>This is the predict method</p>
<pre><code class="lang-auto">def transform_image(infile):
    my_transforms = transforms.Compose([transforms.Resize(256),
                                        transforms.CenterCrop(224),
                                        transforms.ToTensor(),
                                        transforms.Normalize(
                                            [0.485, 0.456, 0.406],
                                            [0.229, 0.224, 0.225])])
    image = Image.open(infile)
    timg = my_transforms(image)
    timg.unsqueeze_(0)
    return timg
</code></pre>
<p>It fails at</p>
<pre><code class="lang-auto">timg = my_transforms(image)
</code></pre>
<p>The stack trace</p>
<p><strong>File “/app/application.py”, line 30, in transform_image<br>
2020-06-14T10:18:45.253206+00:00 app[web.1]:     timg = my_transforms(image)<br>
2020-06-14T10:18:45.253206+00:00 app[web.1]:   File “/app/.heroku/python/lib/python3.7/site-packages/torchvision/transforms/transforms.py”, line 61, in <strong>call</strong><br>
2020-06-14T10:18:45.253206+00:00 app[web.1]:     img = t(img)<br>
2020-06-14T10:18:45.253207+00:00 app[web.1]:   File “/app/.heroku/python/lib/python3.7/site-packages/torchvision/transforms/transforms.py”, line 166, in <strong>call</strong><br>
2020-06-14T10:18:45.253207+00:00 app[web.1]:     return F.normalize(tensor, self.mean, self.std, self.inplace)<br>
2020-06-14T10:18:45.253208+00:00 app[web.1]:   File “/app/.heroku/python/lib/python3.7/site-packages/torchvision/transforms/functional.py”, line 208, in normalize<br>
2020-06-14T10:18:45.253208+00:00 app[web.1]:     tensor.sub_(mean).div_(std)</strong></p>

username: ptrblck
cooked: <p>This error might be raised, if your input tensor has 4 channels, while the <code>Normalize</code> transformation expects a 3 channel image. This might happen, if some of your input channels contain an alpha channel.<br>
After loading the <code>PIL.Image</code>, you could convert it to RGB via <code>image = image.convert('RGB')</code>.</p>

username: Frank_Ofoedu
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thank you. That solved it. I am a newbie to pytorch. Why should that happen? I used a similar image to the image I trained the model with.</p>
<p>I also used the same transformer when loading my dataset during training</p>

username: ptrblck
cooked: <p>This shouldn’t happen, if you are using the same images, which were working before.<br>
However, if you switched e.g. from the training to the validation dataset (or any other dataset), some images might have been stored with an alpha channel, which is usually not needed.</p>
How to time the running time of each layer of a model?

username: marsggbo
cooked: <p>Now I have to time the running time of each layer of a model, including forward, update and backward. Does pytorch provide such tools? Thanks!</p>

username: Kushaj
cooked: <p>Maybe <code>torch.autograd.profiler</code> can help, but I haven’t used <code>torch.autograd</code>. But you can do it yourself with the <code>time</code> library.<br>
For the forward pass, you can manually place <code>time.start()</code> and <code>time.end()</code>. And for the backward pass you can use hooks. Refer to this post for reference <a href="https://discuss.pytorch.org/t/how-to-split-backward-process-wrt-each-layer-of-neural-network/7190">link</a></p>

username: marsggbo
cooked: <p>Thanks for your reply. But I want to measure the running time of each layer,  and I don’t understand how to use <code>time</code> library to do that, could you explain more?</p>

username: Kushaj
cooked: <p>I don’t understand. Why you don’t want to ise time lib? It can do what you want. But if you still use hooks maybe.</p>

username: torch_fresh
cooked: <p>I have the same issue for profiling the runtime of each layer of a model, including forward, backward and update during training. Do you find some pytorch tools? Or give me some detailed implement hints. Thank you.</p>
About error "more than one element of the written-to tensor refers to a single memory location."

username: honglh
cooked: <p>I seems to run into this error very now and then, most times unexpectedly.  I tried to explain or find explanation it using tensor storage models but still do not have a reasonable explantion.</p>
<pre><code class="lang-auto">RuntimeError: unsupported operation: more than one element of the written-to tensor refers to a single memory location. Please clone() the tensor before performing the operation.
</code></pre>
<p>For simplicity, I use these two similar statements in python (C++ has same result).  One is successful, one has the <code>RuntimeError</code></p>
<p><strong>Success example</strong></p>
<pre><code class="lang-auto">a=torch.ones(3,3)
(a.mean(1, True) + a).expand_as(a).mul_(a)
</code></pre>
<p><strong>Runtime Error</strong></p>
<pre><code class="lang-auto">a=torch.ones(3,3)
(a.mean(1, True)).expand_as(a).mul_(a)

RuntimeError: unsupported operation: ... ...
</code></pre>

username: albanD
cooked: <p>Hi,</p>
<p>The error happens because you’re trying to do an inplace operation on a Tensor that has overlapping memory (created by the expand).</p>
<p>In the first case, this does not happen because the <code>+ a</code> actually creates a new Tensor of the size of <code>a</code> and so the <code>expand_as()</code> does nothing in this case.</p>

username: honglh
cooked: <p>Got it…so the <code>+a</code> created a tensor with a shape same as the target (3x3), while the other one relies on the <code>expand_as().clone()</code> to create a tensor of same shape. Thanks for the prompt help</p>

username: albanD
cooked: <p>Yes.<br>
<code>expand()</code> is very efficient because instead of allocating new memory, it only plays with the stride of the Tensor to make it the right size. But as you saw here, it has limitations.</p>
Computing the 2nd derivative of parameters

username: FrozenNep
cooked: <p>I’m designing an Optimizer that needs to compute the 2nd derivative of parameters. but when I print the p.grad.grad it I got <code>None</code>,</p>
<pre><code class="lang-auto">for group in self.param_groups:
    for p in group['params']: 
        grads2 = p.grad.grad
 
</code></pre>
<p>Has there any method to solve this?<br>
Thank you.</p>

username: albanD
cooked: <p>Hi,</p>
<p>You can set the <code>create_graph=True</code> flag while running the backward pass to be able to backward through the backward pass.<br>
But you won’t get a <code>.grad.grad</code>.<br>
In general, when doing higher order derivatives, I would suggest that you use <code>autograd.grad()</code> that returns the gradients directly. That will avoid any confusion about gradient accumulation coming from different backward passes.</p>
Why do we need to specify the input of a linear layer after flatten?

username: Valerio_Biscione
cooked: <p>Hello, I don’t mean to be polemic, I am just curious. I was wondering why, in PyTorch, we need to specify the input size of a linear layer. In keras, after I flatten a layer, I can feed this to a linear layer without having to specify the input size (I assume this can be computed by flatten()). This is very convenient when we want to feed an image of different size to a VGG network and we don’t want to do any model surgery.<br>
I know that in PyTorch we have the AdaptiveGlobalAveragePooling exactly for this reason. However, this means that feeding a big image in a VGG in Keras or Pytorch would actually use a different model. In keras, one without AdaptiveGAP, in PyTorch with it. If we wanna mimic the keras model, we need to change the VGG layer in PyTorch. If the linear layer could automatically compute the input, we wouldn’t need that.</p>

username: albanD
cooked: <p>Hi,</p>
<p>The main reason is that pytorch needs to create the weights at the initialization of the layer (and thus know the input size).<br>
This is to make sure that calls like <code>model.cuda()</code> or <code>your_opt = optim.SGD(model.parameters(), ...)</code> works as expected even before your model has seen a single sample.</p>
<p>Note that changing this is under discussion (but tricky to make sure it works fine with the other constructs in pytorch) and you can find the details in this issue: <a href="https://github.com/pytorch/pytorch/issues/23352">https://github.com/pytorch/pytorch/issues/23352</a></p>

username: Valerio_Biscione
cooked: <p>thanks, exactly the explanation I was looking for</p>
Mini-batch gradient and Loss.backward()

username: cindybrain
cooked: <p>I am confused that Loss.backward() calculate the gradients many times over the samples in the mini-batch or just calculate it one time?<br>
and if I only want to update the weights according to the gradients of the subset samples in the batch, what can I do? thanks</p>

username: albanD
cooked: <p>Duplicate question. Answered there: <a href="https://discuss.pytorch.org/t/is-loss-backward-function-calculate-gradients-over-mini-batch/66873/5" class="inline-onebox">Is Loss.backward() function calculate gradients over mini-batch</a></p>
C++ inferenec on detection model

username: jinfagang
cooked: <p>Hello, I have trained a model with pytorch, but how to inference using c++.</p>
<p>It’s a detection model similar as RFBNet and SSD, so is there anyone try to load model in C++?</p>

username: tom
cooked: <p><a href="https://github.com/facebookresearch/maskrcnn-benchmark/pull/138" rel="nofollow noopener">MaskRCNN-benchmark has a PR for tracing the model.</a></p>
<p>Best regards</p>
<p>Thomas</p>

username: yf225
cooked: <p>To use a PyTorch model for inference in C++, we need to convert the model to TorchScript first. This is a good tutorial: <a href="https://pytorch.org/tutorials/advanced/cpp_export.html" rel="nofollow noopener">https://pytorch.org/tutorials/advanced/cpp_export.html</a>.</p>

username: jinfagang
cooked: <p>I know, I have successfully trace a classification model to C++, not sure about the detection models.</p>
<p>BTW, does there any comparation of C++ inference speed with libtorch and Caffe or Tensorflow? Who is faster in terms of same models architecture?</p>

username: AlexandrGraschenkov
cooked: <p><a class="mention" href="/u/jinfagang">@jinfagang</a> do you successfully run model inference from c++?</p>

username: jinfagang
cooked: <p>Yes, but it doesn’t get any speed up (just a littel).</p>
<p>I am going using onnxruntime as inference engine instead. it can using intel cpu mkl as acceleration or using cuda tensorrt acceleration.</p>

username: Shisho_Sama
cooked: <p>Did you face any difficulties in exporting the detection model into C++?<br>
did everything run smoothly for you?</p>
Specialized scatter operations for batched data

username: Stannislav
cooked: <p>Hi,</p>
<p>PyTorch provides the <code>scatter_add</code> operation, which can be very useful, but unfortunately is not deterministic. A deterministic mode for this operation is AFAIK not yet implemented (see <a href="https://github.com/pytorch/pytorch/issues/23151" rel="nofollow noopener">this issue</a>).</p>
<p>While the built-in <code>scatter_add</code> is very general, I’m wondering if there are solutions for the special case  where the index tensor is sorted. This could be a very relevant usecase where the index is just a segmentation/mask for variable-sized batches that were stacked together.</p>
<p>I’m thinking of something like this:</p>
<pre><code class="lang-python">&gt;&gt;&gt; batch_tensor = torch.tensor([1, 2, 3, 4, 5])
&gt;&gt;&gt; index_tensor = torch.tensor([0, 0, 0, 1, 1])

# either
&gt;&gt;&gt; scatter_add(batch_tensor, index_tensor)
tensor([6, 9])

# or
&gt;&gt;&gt; torch.zeros(2).scatter_add(batch_tensor, index_tensor)
tensor([6, 9])
</code></pre>
<p>we have a batch with 2 elements, <code>[1, 2, 3]</code> and <code>[4, 5]</code>, and we’re doing a sum global pool.</p>
<p>It seems there already are thoughts about sorted indices out there (e.g. point 3 <a href="https://github.com/pytorch/pytorch/issues/24621#issuecomment-592921295" rel="nofollow noopener">here</a>), but I don’t have enough insight to assess the current status. I’m wondering:</p>
<ul>
<li>Is there already something like this in torch?</li>
<li>Is there already something like this in cuda?</li>
<li>If not, how straightforward is it to implement this?</li>
</ul>
<p>Sure one can implement this directly in python by iterating over the index array, but I guess this would be painfully slow and there must be a better way that leverages the GPU.</p>
<p>I think a solution to this would have an immediate benefit for graph neural networks, but potentially also for other areas like CNNs with variable-sized images.</p>
<p>Stan.</p>
My model only predicts values in a very narrow range

username: Lanorius94
cooked: <p>This is my model and its variables:</p>
<p>input_size_prot = 1024<br>
input_size_comp = 196<br>
hidden_size_prot = 32<br>
hidden_size_all = 100<br>
output_size = 1</p>
<p>batch_size = 40</p>
<p>class pcNet(nn.Module):</p>
<pre><code>def __init__(self, input_size_prot, input_size_comp, hidden_size_prot, hidden_size_all, output_size):
    super(pcNet, self).__init__()
    self.fc_prot = nn.Linear(input_size_prot, hidden_size_prot)
    self.fc_all = nn.Linear(hidden_size_prot+input_size_comp, hidden_size_all)
    self.fc2 = nn.Linear(hidden_size_all, output_size)

def forward(self, x):
    out = F.leaky_relu(self.fc_prot(x[0]))
    out = torch.cat((out, x[1]), dim=1)
    out = F.leaky_relu(self.fc_all(out))
    out = F.relu(self.fc2(out))
    return out
</code></pre>
<p>The labels I am trying to predict are all values between 5 and 11, however my model only predicts values between 5 and 7.5<br>
The loss converges around 20. It is possible that my data (two sets of tensors) does not allow for better results, but perhaps you have ideas for my model.</p>

username: thomasjo
cooked: <p>Based on what you have shared, I need some more details to attempt to help you figure things out;</p>
<ul>
<li>What type of loss function are you using?</li>
<li>Which optimization method are you using, and what are its hyperparameters?</li>
<li>How big is your training dataset?</li>
<li>How long are you training for, i.e. how many epochs?</li>
<li>Are you using any type of learning rate scheduling, early stopping, or similar?</li>
</ul>

username: Lanorius94
cooked: <p>loss and optimization:<br>
criterion = torch.nn.MSELoss(reduction=‘sum’)<br>
optimizer = optim.Adam(model.parameters(), lr=0.0001)</p>
<p>As you can see there are two inputs for data, one set has 375 elements the other 63, and I am checking all possible interactions so 23625 in total.<br>
I am using a 80/20 split, so 18900 interactions for training and 4725 for testing.<br>
This also means that all off the original 375 and 63 elements appear in both training and testing (it’s supposed to be like that).</p>
<p>Currently I am training for 5 epochs, but the model converges after two.</p>
<p>I am not using any type of learning rate scheduling</p>
Unable to reproduce results using PyTorch in Colab

username: Kalyan_Katikapalli
cooked: <p>I’m running a model  written in PyTorch  in Colab on P100 GPU and I have set all the seeds using the following code</p>
<pre><code class="lang-auto">torch.manual_seed(seed)
torch.cuda.manual_seed_all(seed)
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False
np.random.seed(seed)
random.seed(seed)
os.environ['PYTHONHASHSEED'] = str(seed)
</code></pre>
<p>Apart from the above, I have set seed value wherever it is required in the code.</p>
<p>I’m able to get reproducible results only when I run the code in the same session. When I close the session and then rerun the code in a new session, results are varying.<br>
Is it not possible to get reproducible results using PyTorch in Colab? <a class="mention" href="/u/smth">@smth</a> <a class="mention" href="/u/ptrblck">@ptrblck</a></p>

username: Bjorn_Lindqvist
cooked: <p>One problem may be that when you start a new Collab session you might not get the same GPU. Sometimes you get a fast GPU and sometimes you get a slow GPU which you have to share with other Collab users. I don’t know if that makes any difference but it very well could.</p>

username: Kalyan_Katikapalli
cooked: <p>Thanks for the information.</p>
<p>Every time I will  run the code on P100 GPU only with the same RAM.  To get reproducible results,  isn’t enough to run the code on same type of GPU?</p>

username: Bjorn_Lindqvist
cooked: <p>I don’t know. But as a first step ensure that you get identical results when you run the code on the cpu of your own computer.</p>

username: Kalyan_Katikapalli
cooked: <p>What are you thoughts on this? <a class="mention" href="/u/rasbt">@rasbt</a></p>
How to rotate a 3d image stack

username: watersun06
cooked: <p>Hi there,<br>
I have an image stack in tiff format，in shape of (depth* width<em>height). I want to rotate it around axis 0 . I tried ndimage.rotate, but it runs too slow as my image has a large size, 600</em>1200*1200 typically. So I turn to Pytorch as it has cuda backend. How can I do my rotation with pytorch? I really need help. Thanks!</p>
Variational RNN

username: sait
cooked: <p>Original LSTM cell uses dropout that uses different mask at every time step which is ad-Hoc and it leads to unstable results.According this paper we should use same dropout masks at every time step. <a href="https://arxiv.org/pdf/1512.05287.pdf" rel="nofollow noopener">Variational RNN</a></p>
<p>Here is the screenshot what should ideally happen</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://user-images.githubusercontent.com/50786143/80912419-6c31d680-8d5a-11ea-8d57-33b4d5c61838.png" title="Screenshot_2020-05-03 1512 05287 pdf" rel="nofollow noopener"><img src="https://user-images.githubusercontent.com/50786143/80912419-6c31d680-8d5a-11ea-8d57-33b4d5c61838.png" alt="Screenshot_2020-05-03 1512 05287 pdf" width="690" height="264"><div class="meta">
<svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use xlink:href="#far-image"></use></svg><span class="filename">Screenshot_2020-05-03 1512 05287 pdf</span><span class="informations">704×270</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use xlink:href="#discourse-expand"></use></svg>
</div></a></div></p>
<p>Keras supports this with (dropout and recurrent dropout)<br>
Is there any neat implementation for this pytorch? Thanks for Helping</p>

username: Bjorn_Lindqvist
cooked: <p>I think the answer is no. I have also looked for this feature but not found anything. In my Keras model it improves accuracy by about 5% so it’s the main reason I haven’t been able to port my model to PyTorch yet.</p>

username: Bjorn_Lindqvist
cooked: <p>I spoke to soon. Seems like this LSTM supports variational dropout: <a href="https://github.com/keitakurita/Better_LSTM_PyTorch" rel="nofollow noopener">https://github.com/keitakurita/Better_LSTM_PyTorch</a></p>
Keras LSTM output shape vs Pytorch

username: garthtrickett
cooked: <p>In keras I have an input of (None, 100, 192)<br>
which Is put through layers.LSTM(64)<br>
which outputs a shape of (None, 64)</p>
<p>In pytorch I have an input of  [64, 192, 100]) and then put it through nn.LSTM(100, 64) to get torch.Size([64, 192, 64])</p>
<p>What do I need to do instead with the LSTM layer in pytorch so that it outputs<br>
(batch_size, 64) which is equivalent to the (None, 64) in keras rather than  [64, 192, 64]</p>

username: ptrblck
cooked: <p>I guess Keras’ LSTM layer might return the last temporal step by default, so you could index the output from the PyTorch LSTM layer via <code>output[:, -1]</code> assuming that your current batch size is 64 and you’ve used <code>batch_first=True</code>.</p>
Policy gradient training loss goes to zero

username: sepehr78
cooked: <p>Hi all,</p>
<p>I am training my network with the policy gradient approach. My rewards are either -1 or 1, and I calculate the loss and update the parameters just like in <a href="https://github.com/pytorch/examples/blob/master/reinforcement_learning/reinforce.py" rel="noopener nofollow ugc">here</a>. However, the loss increases and goes to zero and the average reward in each epoch quickly converges to -1. I attached the loss and reward plots. My optimizer is SGD, and I tried ADAM and also played around with the learning rate, but no success. Does anyone have any ideas?</p>
<p>Thanks.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/d/7d2dca24839cc0c40a4aa7b6027041bdc456b52f.png" data-download-href="https://discuss.pytorch.org/uploads/default/7d2dca24839cc0c40a4aa7b6027041bdc456b52f" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/d/7d2dca24839cc0c40a4aa7b6027041bdc456b52f_2_690x316.png" alt="image" data-base62-sha1="hRnMPAqehcRaGrwdRZkpy5FuhRl" width="690" height="316" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/d/7d2dca24839cc0c40a4aa7b6027041bdc456b52f_2_690x316.png, https://discuss.pytorch.org/uploads/default/original/3X/7/d/7d2dca24839cc0c40a4aa7b6027041bdc456b52f.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/7/d/7d2dca24839cc0c40a4aa7b6027041bdc456b52f.png 2x" data-dominant-color="F9F8F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">829×380 24.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Edit: at the output I have a softmax layer with 11 outputs (actions) and it seems that after training, the model “learns” to output 0 for all actions but one. Then, the log probability of that action will be 0 and that is why the loss is zero. This is very bizarre to me because I expect the loss to decrease and not increase.</p>

username: iffiX
cooked: <p>Could you please share your implementation?</p>
Traverse torch::Tensor by batch

username: copythatpasta
cooked: <p>Hello, I would like to know how I can traverse a tensor with dims{5,3,36,36} by batch. I would like to traverse each batch and return it into another tensor to do an operation using the c++ torch front end. I am using v1.3.0 so cannot use Slice in the new v1.5.0 libs. How can I do this ?</p>

username: albanD
cooked: <p>Hi,</p>
<p>You can use <code>.select(dim, idx)</code>. Where you want to use dim=0 and idx being the batch index.</p>
ResNet Input Dimensions

username: seedship
cooked: <p>Does it make a difference if I input images to ResNet as (CxHxW) or (HxWxC)? In order to use Matplotlib imshow, it needs to be in HxWxC, so that is the format I am currently using. However, I introduced transforms to perform data augmentation, and that is returning imsages in CxHxW.</p>
<p>When using pretrained pytorch models, which layout is the best? Thanks!</p>

username: Nikronic
cooked: <p>Hi,</p>
<p>PyTorch works only on CHW format although there are some experimental features for HWC. For matplotibl imshow, you can just convert tensor to numpy using <code>tensor.numpy</code> and then permute its channels.</p>
<p>For instance, if you are using PIL:</p>
<pre><code class="lang-auto">x = Image.open('cat.jpg')
transform = transforms.Compose([transforms.Resize([256, 256]),
									transforms.RandomCrop(224),
									transforms.ToTensor()])
new_x = transform(x)
plt.imshow(new_x.permute((1, 2, 0)))
</code></pre>
<p>and if <code>new_x</code> is a tensor, just add <code>new_x.numpy()</code>.</p>
<p>Bests</p>
TorchScript model with C++ is much slower than pytorch and ONNX

username: knitvoger
cooked: <p>I tried 3 ways to run a torch.nn.GRU model on a cpu. The model is like<br>
model = nn.GRU(512, 256, batch_first=True, bidirectional=True)</p>
<ol>
<li>run with pytorch;   2. convert to TorchScript and run with C++;   3 convert to ONNX and run with python</li>
</ol>
<p>Each test was run 100 times to get an average number. The result is TorchScript with C++ is much slower than the others. Pytorch and ONNX only take about 40ms to run, but c++ takes about 120ms!</p>
<p>All these 3 ways use same mkldnn backend, such big performance seems not expected. Does any one know why and how to improve the performance in C++?</p>
<p>Thanks very mych!</p>

username: copythatpasta
cooked: <p>Did you build pytorch with mkl ? I would suggest building with the MKL includes and sos you can get in a conda environment. MKL is much faster than eigen</p>
Save the torch dataset Datase

username: Gur_Amrit
cooked: <p>I am trying to train GPT2 on a custom dataset, the training text file is over 100GB, when creating the Dataset for training, it is taking too long and I was hoping if there is any way to save the dataset for later use.</p>
<p>The dataset is torch dataset.</p>
<p>I am using hugging face blog post on how to train transformers from scratch as a guide (<a href="https://huggingface.co/blog/how-to-train" rel="nofollow noopener">https://huggingface.co/blog/how-to-train</a>).</p>
<p>Thank You</p>
K fold cross validation for CNN

username: Muhammad_Izaz
cooked: <p>can i perform kfold cross validation for training my CNN model</p>

username: sepehr78
cooked: <p>I think this <a href="https://discuss.pytorch.org/t/what-is-the-best-way-to-apply-k-fold-cross-validation-in-cnn/15035">answer</a> might help.</p>
Sparse tensor multiply a diagonal tensor

username: Chen-Cai-OSU
cooked: <p>Hello,</p>
<p>I would like to multiply a (symmetric) sparse tensor with a diagonal tensor. Since both tensors are sparse and the output is also sparse, I expect that this can be done very efficiently. However, my current solution use torch.sparse.mm and I have to convert one of tensor to be dense, which is not very memory efficient.</p>
<p>Do you have some good suggestions? Thank you!</p>

username: ptrblck
cooked: <p>Sparse matrix multiplication doesn’t seem to be implemented yet as seen in <a href="https://github.com/pytorch/pytorch/issues/5262">this feature request</a>, so I assume you would have to use a dense matrix for one tensor.</p>

username: Chen-Cai-OSU
cooked: <p>Thank you. I end up transferring the tensor to cpu and use scipy.sparse to do the job. Hope there will be more features to support common sparse tensor computation.</p>
Very Slow training on RTX 2070 Max Q

username: vishal-purohit
cooked: <p>Hey, I am trying to train CNN from scratch with 4 convolution layers. I am using dataset from kaggle and code is hosted on git hub : <a href="https://github.com/vishal-purohit/Histopathological-Image-Classification" rel="nofollow noopener">https://github.com/vishal-purohit/Histopathological-Image-Classification</a></p>
<p>Following are the specification of my laptop:<br>
-&gt; RAM : 16 GB<br>
-&gt; Intel i7 - 9th gen (4.0GHz)<br>
-&gt; Nvidia RTX 2070 Max Q (8GB)<br>
-&gt; Windows 10<br>
-&gt; pytorch version 1.5</p>
<p>RAM usage : 11.7 GB<br>
CPU Usage : 14% - 25 %<br>
GPU Usage : 0 - 6 %<br>
GPU clock speed : 885 MHz<br>
Batch_Size : 2048<br>
GPU_Memory used : 6.7GB / 8GB</p>
<p>The GPU utilization is very low mostly it is 0% and fluctuates between 6%. What might be the bottle neck in this case?</p>
<p>one epoch takes approximately 22 minutes</p>

username: thomasjo
cooked: <p>I have not looked extremely closely at your code, but two things I noticed that are missing, which might have some impact on performance, is to optimize for better CPU and GPU operation interleaving by using asynchronous data transfers. This might not have a huge impact in your case, but since it is quick and easy to implement, I recommend giving it a shot.</p>
<p>First, on your data loaders, add the keyword argument <code>pin_memory=True</code>. Second, in your train/eval loop where you copy data from CPU memory to GPU memory using <code>.to(...)</code>, append the keyword argument <code>non_blocking=True</code>.</p>
<p>Specific examples of these changes to your notebook:</p>
<pre><code class="lang-python"># DataLoader with pin_memory=True
train_data_loader = DataLoader(train_dataset, batch_size=2048, shuffle=True, num_workers=0, pin_memory=True)

# Tensor.to(...) with non_blocking=True
xb = xb.to(device, non_blocking=True)
</code></pre>
<p>How much impact this will have in your specific case is really hard to determine, since I cannot see how much time the GPU spends waiting on data transfer, etc. But again, making these changes is quick, so I recommend testing to see if it speeds things up. If the impact is minimal, we will need to see more details related to GPU utilization; e.g. output from <em>nvprof</em> or similar.</p>

username: vishal-purohit
cooked: <p>No change in the training speed. Still GPU utilization is 0% -2% and sometime 6%</p>
Comparsion through torch.eq

username: userdyk
cooked: <p>Hi, all.</p>
<p>While checking the values of torch tensor, I have found a weird boolean on torch.eq. Distinctly, tensor values between a, b on the captured photo are all the same, but, the booleans are not all the same. How come these the result was coming?</p>
<blockquote>
<p>import torch<br>
import torch.nn as nn</p>
<p>h_src = torch.Tensor(10,3,5).uniform_(0,1)<br>
h_t_tgt = torch.Tensor(10,1,5).uniform_(0,1)</p>
<p>model = nn.Linear(5,5, bias=None)<br>
a = model(h_t_tgt)<br>
b = torch.einsum(‘lk,ijk-&gt;ijl’, [model.weight, h_t_tgt])</p>
<p>print(torch.eq(a.data, b.data))<br>
print(a)<br>
print(b)</p>
</blockquote>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/a/0a6a0d23cb1bee39cb24488392c9df0e0f7f4958.png" data-download-href="https://discuss.pytorch.org/uploads/default/0a6a0d23cb1bee39cb24488392c9df0e0f7f4958" title="제목 없음"><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/a/0a6a0d23cb1bee39cb24488392c9df0e0f7f4958.png" alt="제목 없음" data-base62-sha1="1u7Zih9G9vAyR0Ygle28REgoHSo" width="470" height="500" data-dominant-color="878787"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">제목 없음</span><span class="informations">1340×1423 11.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I understood this problem is related with a floating-point calculation. Then, is it hard for torch.eq to be precisely used?</p>

username: ptrblck
cooked: <p>To compare floating point numbers, which might have small errors due to the limited floating point precision, you should use <code>torch.isclose</code> or <code>torch.allclose</code> instead of a direct comparison.</p>
Pytorch+ Tensorboard

username: Prakhar_Pradhan
cooked: <p>Hi everyone.<br>
I am working with pytorch, and trying to implement a semantic segmentation method.<br>
I want to visualize a feature map whose size is [10,192,128,128], 10 is the batch size, 192 is the number of channels and (128,128) being spatial dimension.<br>
Is there anyway to visualise this feature map using tensorboard in pytorch.??<br>
I tried using add_image(), but I think it works only with 3 channels.</p>

username: ptrblck
cooked: <p>You could visualize each channel for all samples as a grayscale image (which would result in 1920 images). <code>torchvision.utils.make_grid</code> might be useful to create a smaller grid of these images.</p>
Retain_graph = True out of memory issue

username: maroo-sky
cooked: <p>I’ve tried to train intermediate layers independently but when I run my code I’ve got out of memory issue.</p>
<p>I would like to know <strong>retain_graph = True</strong> is the problem? or not and what I have to do to solve this problem.</p>
<p>I am using hugging face BERT model and this is my code</p>
<p>Thank you.</p>
<pre><code>
t_outputs = t_model(**intputs)
s_outputs = s_model(**inputs)
encoder_layers = args.encoder_layers

loss = torch.nn.KLDivLoss(reduction='batchmean')
batch, row, col = s_outputs[3][1].size()

for i, k in enumerate(encoder_layers):
    output = loss(F.log_softmax(s_outputs[3][i+1], dim=2).view(batch*row, 1, col), F.softmax(t_result[3][k+1],dim=2).view(batch*row, 1, col))
   # to freeze layers
    for name, p in s_model.named_parameters():
        if "layer." + str(i-1) in name:
        p.requires_grad = False
        output.backward(retain_graph=True)
</code>
</pre>

username: ptrblck
cooked: <p><code>retain_graph=True</code> might be the cause of the OOM issue, as it will force Autograd to keep the computation graph alive (and will thus use more memory than in the default use case).</p>
<p>To clear the intermediate tensors, the last <code>backward</code> call should use <code>retain_graph=False</code>.</p>
<p>Note that you are currently accumulating the gradients, as you are calling <code>output.backward()</code> for each parameter, while setting the <code>requires_grad</code> attribute of a single parameter to <code>False</code>.</p>
RuntimeError: CUDA out of memory during training

username: bolt25
cooked: <p>My model has 195465 trainable parameters and when I start my training loop with batch_size = 1 the loop works. But when I try to increase the batch_size to even 2 then the cuda goes out of memory.</p>
<p>I tried to check status of my gpu using this block of code</p>
<p>device = torch.device(‘cuda’ if torch.cuda.is_available() else ‘cpu’)<br>
print(‘Using device:’, device)<br>
print()</p>
<pre><code class="lang-auto">#Additional Info when using cuda
if device.type == 'cuda':
    print(torch.cuda.get_device_name(0))
    print('Memory Usage:')
    print('Allocated:', round(torch.cuda.memory_allocated(0)/1024**3,1), 'GB')
    print('Cached:   ', round(torch.cuda.memory_cached(0)/1024**3,1), 'GB')
</code></pre>
<p>which gave the following output:-</p>
<pre><code class="lang-auto">Using device: cuda

GeForce GTX 1050
Memory Usage:
Allocated: 2.9 GB
Cached:    2.9 GB
</code></pre>
<p>Can anyone help me understand the above output and why is this happening?</p>

username: SiQube
cooked: <p>Hey bolt25,</p>
<p>did you try torch.cuda.empty_cache()?</p>

username: bolt25
cooked: <p>Yes I did, but no use</p>

username: ptrblck
cooked: <p>During the forward pass intermediate tensors will be stored (output activations), which are needed for the gradient calculation during the backward pass.<br>
Depending on the model architecture these activations might use more memory than the model parameters.<br>
You could check the peak memory via <code>torch.cuda.max_memory_allocated()</code> using a batch size of 1.</p>
<p>Calling <code>torch.cuda.empty_cache()</code> will <strong>not</strong> avoid the OOM memory issue, but will instead only slow down your code.</p>
Backpropagation skips layers

username: NIkolaStaykov
cooked: <p>I am trying to use a temporal conv net with different kernel sizes simultaneously for regression. Net.modules() does not include the 1d convolutions, but rather only the fc layers at the end.</p>
<pre><code class="lang-auto">class TemporalConvNet(nn.Module):
    def __init__(self, in_channels, out_channels, kernel_sizes=[2,3], dilation_size=1):
        super(TemporalConvNet, self).__init__()

        self.group_norm1 = nn.GroupNorm(in_channels, in_channels)
        self.g_norms_out = []
        for i in range(len(kernel_sizes)):
            self.g_norms_out.append(nn.GroupNorm(out_channels, out_channels).double().cuda())
        self.mish = Mish()
        
        self.temporal_blocks = []
        for kernel_size in kernel_sizes:
            self.temporal_blocks.append(TemporalBlock(in_channels, out_channels, kernel_size, stride=1, dilation=dilation_size,
                                    padding=kernel_size-1))
        
        self.linear_part = Linear_out(len(kernel_sizes)*out_channels*100, 1)

    def forward(self, x):
        x = self.group_norm1(x)
        
        conv_outputs = []
        for ind in range(len(self.temporal_blocks)):
            temp_block = self.temporal_blocks[ind]
            group_norm = self.g_norms_out[ind]
            temp_block.cuda().double()
            conv_out = temp_block(x)
            conv_out = group_norm(conv_out)
            conv_out = self.mish(conv_out)
            conv_outputs.append(conv_out)
            print(conv_out[0][0])
        conv_outputs = torch.cat(conv_outputs, dim=1)
        x = torch.flatten(conv_outputs, start_dim=1)
        print("After flatten: ", x[:2])
        torch.Variable(x) = self.linear_part(x)
        return x.view(-1)
</code></pre>
<pre><code class="lang-auto">
TemporalConvNet(
  (group_norm1): GroupNorm(259, 259, eps=1e-05, affine=True)
  (mish): Mish()
  (linear_part): Linear_out(
    (dense_1): Sequential(
      (0): Linear(in_features=8000, out_features=64, bias=True)
      (1): Mish()
      (2): Dropout(p=0.2, inplace=False)
    )
    (dense_2): Sequential(
      (0): Linear(in_features=64, out_features=10, bias=True)
      (1): Mish()
      (2): Dropout(p=0.2, inplace=False)
    )
    (dense_3): Linear(in_features=10, out_features=1, bias=True)
  )
)
</code></pre>

username: ptrblck
cooked: <p>Plain Python <code>lists</code> won’t register <code>nn.Modules</code> properly inside the parent model and you should use <code>nn.ModuleList</code> instead.</p>
Joint Model Training

username: Ahmed_Abdelaziz
cooked: <p>I have two torch models that are trained separately. My goal is to jointly train them instead of fully train the first model, save the output and then load it into the second model.</p>
<p>Every model has its own loss, the first model loss is kinda complicated because it’s an average of three different losses - Three joint tasks - (You may choose to ignore this detail). So I am not sure how should I approach this, is it better to drop the first model’s losses and only backpropagate the seconds? If not, how can one initialize backpropagation from Two losses in one model?</p>
<p>I am also aware that I can use something like learning rate control to speed up the first model learning while slowing down the second’s in the beginning and switch as well as forcing ground-truth in the early epochs. Appreciate your help!</p>
Tensorboard API for PyTorch C++ frontend

username: theskushwaha
cooked: <p>I am using C++ frontend to train my networks. But it seems there is no way to log data for viewing in Tensorboard. Please let me know if there is such a thing, or link to some alternatives that I can directly use from c++.</p>

username: Kaii
cooked: <p>You can use pybind 11 to call python libraries from C++ side.<br>
<a href="https://pybind11.readthedocs.io/en/stable/advanced/pycpp/object.html#accessing-python-libraries-from-c" rel="nofollow noopener">https://pybind11.readthedocs.io/en/stable/advanced/pycpp/object.html#accessing-python-libraries-from-c</a></p>

username: theskushwaha
cooked: <p>I’m using C++ frontend because I did not want to use python in my project. I think pybind is not way to go. Come on pytorch, I know, you can do better.</p>

username: thomasjo
cooked: <p>I have not tried this personally, but from browsing through the code (specifically the tests), it looks like this library should solve your issue: <a href="https://github.com/RustingSword/tensorboard_logger" rel="nofollow noopener">https://github.com/RustingSword/tensorboard_logger</a></p>
<p>Of course, this is not official PyTorch C++ frontend support for TensorBoard, but I am assuming that is not a hard requirement (in the short term).</p>
Sneeze and cough detection with pytorch

username: Sundar_Krishna
cooked: <p>Hi guys I am a newbie to pytorch so please bear with me if my question is wrong!<br>
I googled lot about sneeze and cough detection but didnt get any useful resources. My main objective is to detect a person who sneezes or coughs with a live running camera with pytorch models along with torchvision or opencv.So can anyone please provide me with some examples , resources , source codes or links which I can refer to? Examples with Raspberry pi are most welcome as Iam planning to implement with it.</p>

username: Angry_potato
cooked: <p>I hear that it is impossible to sneeze with eyes open. U can train a model to identify if a person has closed eyes or open eye. if his eyes are closed then u can look for further details to classify it as sneeze or something else</p>

username: Sundar_Krishna
cooked: <p><a class="mention" href="/u/angry_potato">@Angry_potato</a> is it possible in pytorch to train with audio files? Like classifying with the sound of sneezing or coughing?</p>

username: boto
cooked: <p>
Come on guys, do some basic research <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=9" title=":smile:" class="emoji" alt=":smile:"><br>
Here are two papers using neural networks for cough detection: <a href="https://ieeexplore.ieee.org/abstract/document/7488043" rel="nofollow noopener">Cough detection using speech analysis</a> and <a href="https://ieeexplore.ieee.org/abstract/document/7570164" rel="nofollow noopener">Deep Neural Networks for Identifying Cough Sounds</a>. As I’m still a student, I’m able to access them through the VPN of my uni. I’m not sure if you can download the papers.
<br>
I misread the question, see bellow.</p>

username: Sundar_Krishna
cooked: <p><a class="mention" href="/u/boto">@boto</a> there are lot of papers and theoretical approach based on this which I went through.  I posted this question in the forum to get some help in the coding and implementation or hoping to get resources of datasets to train my model. So no one posts a question without good amount of research.</p>

username: boto
cooked: <p>Oh damn, I misread your question. Sorry about that!<br>
Unfortunately, I can’t help you with the implementation.</p>

username: Sourav_Raihan
cooked: <p>Hi! I saw your question on opencv forum as well. Have you had any progress so far? I was wondering if it could be implemented with thermal images taken from an IR camera.</p>

username: Sundar_Krishna
cooked: <p>Hi ,<br>
Yes i have accomplished it using mel-spectograms. To differentiate different sounds</p>
How should I deploy my Neural Style Transfer Model on the web?

username: sharmadhavs
cooked: <p><strong>Hello</strong></p>
<p>I just implemented <a href="https://arxiv.org/abs/1508.06576" rel="nofollow noopener"> A Neural Algorithm of Artistic Style</a> and now I wish to deploy it to the web in the form of a web application in which users can upload their input image and choose among some presets of style images. After pre-processing, the model should output the generated image.</p>
<p>I have experience with web development and deep learning individually but not together. What tech stack should I choose to accomplish this and also can someone please tell me the <em>flow</em> I’ll have to follow?</p>
<p>How do I deploy my model on the web and allow user interactions? Any help would be really appreciated.</p>