akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Fastest way of loading very small datasets

username: iman.mirzadeh
cooked: <p>Hi,</p>
<p>I was wondering what would be the fastest way to work with very <strong>small</strong> datasets like MNIST or CIFAR10 with PyTorch.<br>
Is it loading the dataset into the RAM? If so, how can I do it?</p>

username: JuanFMontesinos
cooked: <p>At the time of defining the dataset class you can read data there  so that it gets loaded in RAM (instead of reading  from hard disk)<br>
You can also directly load them into the GPU (using the main thread) and preprocessing directly in the gpu.<br>
The drawback is image preprocessing is PIL based, so you would have to reimplement them as pytorch functions.</p>

username: iman.mirzadeh
cooked: <p>Is there a forum link/documentation on how to do that?</p>

username: charan_Vjy
cooked: <p><a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/data_loading_tutorial.html</a></p>
<p>Pytorch’s tutorial is extensive enough. You can push in all of the heavy lifting in the Dataset’s getitem attribute including loading using PIL and conversion to torch.</p>
Problems with Cross Entropy Loss

username: smani
cooked: <p>I am getting two different loss values in the following situations, although I expect them to be same.</p>
<ol>
<li>
<p>ce = torch.nn.CrossEntropyLoss(classWeights, reduction=‘none’)<br>
loss = ce(input, targets).mean()</p>
</li>
<li>
<p>ce = torch.nn.CrossEntropyLoss(classWeights, reduction=‘mean’)<br>
loss = ce(input, targets)</p>
</li>
</ol>
<p>What could be the reason for that?</p>

username: ptrblck
cooked: <p>You would have to normalize the unreduced and weighted loss with the <code>classWeights</code> as shown <a href="https://discuss.pytorch.org/t/does-weightedrandomsampler-increase-interations/51514/16">here</a>.</p>
Normalize data from scratch

username: snip3r77
cooked: <p>Objective = Normalize the data: subtract the mean RGB (zero mean)</p>
<p>X_train.size()</p>
<blockquote>
<p>torch.Size([50000, 3, 32, 32])</p>
</blockquote>
<p>X_train.mean(dim=0, keepdim=True).mean(dim=2, keepdim=True).mean(dim=3, keepdim=True).size()</p>
<blockquote>
<p>torch.Size([1, 3, 1, 1])</p>
</blockquote>
<p>just to confirm, we are flattening X_train over dim 0,2,3.<br>
the reason we skipped dim=1 because we want to preserve the mean of each of the RGB channels?</p>
<p>Thanks</p>

username: ptrblck
cooked: <p>Yes, you are skipping dim1, as you don’t want to calculate the mean in this dimension.<br>
The code can also be written as <code>X_train.mean([0, 2, 3], keepdim=True)</code>.</p>
How to generate random tensor, excluding some elements?

username: shaoteng
cooked: <p>Hi, I want to know how to generate random tensor, excluding some elements.<br>
For example, I am used to using this code to generate random tensor:</p>
<pre><code class="lang-auto">rand_tensor = torch.zeros(N).random_(0, K)
</code></pre>
<p>where (0,K) is the range<br>
However, sometimes I want to exclude some probability. For example, I want to generate number in (0,K), but not K/2 and K/3.<br>
What is the most efficient way to do in this situation?</p>

username: ptrblck
cooked: <p>You could probably use <a href="https://pytorch.org/docs/stable/torch.html#torch.multinomial"><code>torch.multinomial</code></a> and specify the probability distribution manually.</p>
How to extract foreground region feature vectors

username: JasonChenhx
cooked: <p><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/b/ab9fbc1a7f524b1a670974928dbe2547c0846c0b.jpeg" alt="2007_000129" data-base62-sha1="oufSpCxY7z1G4Z3IjjBCwbcsUjh" width="334" height="500"> <img src="https://discuss.pytorch.org/uploads/default/original/3X/2/b/2b4415c3437a718575cbb75577b62cf284bccf2d.png" alt="2007_000129" data-base62-sha1="6aKoXWNanQJ2r7rQGKVfbdTeJfT" width="334" height="500"><br>
I have obtained the feature map of the original through resnet101, and I have converted the segmentation result to an array of 01(The foreground pixel is 1 and the background pixel is 0).<br>
And then I dot the 01 array with the feature map to see if I can get the foreground feature vectors?</p>
Merging networks

username: James_e
cooked: <p>I have a network such as</p>
<pre><code class="lang-auto">class NeuralNet(nn.Module):

    def __init__(self):
        super(NeuralNet, self).__init__()
        self.fc1 = nn.Linear(10, 5)
        self.fc2 = nn.Linear(5, 2)
    def forward(self, x):
        x = nn.functional.relu(self.fc1(x))
        x = nn.functional.relu(self.fc2(x))
       
        return x
</code></pre>
<p>How can I do losses if i want to split them up? e.g</p>
<pre><code class="lang-auto">

class Layer1(nn.Module):

    def __init__(self):
        super(NeuralNet, self).__init__()
        self.fc1 = nn.Linear(10, 5)
    def forward(self, x):
        x = nn.functional.relu(self.fc1(x)
        return x

class Layer2(nn.Module):

    def __init__(self):
        super(NeuralNet, self).__init__()
        self.fc2 = nn.Linear(5, 2)
    def forward(self, x):
        x = nn.functional.relu(self.fc2(x)
        return x
</code></pre>
<p>The only loss is from the final layer where it compares to a label.</p>
<p>How can you call .backward() on this?</p>

username: ptrblck
cooked: <p>You could just call these model in a sequential manner (same as in <code>NeuralNet</code>):</p>
<pre><code class="lang-python">layer1 = Layer1()
layer2 = Layer2()

x = torch.randn(1, 10)
target = torch.randint(0, 2, (1,))

out = layer1(x)
out = layer2(out)

loss = criterion(out, target)
loss.backward()
</code></pre>

username: James_e
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>  Will loss.backward() calculate the gradient for all parameters in layer 1 and layer 2?</p>

username: ptrblck
cooked: <p>Yes, Autograd will track all operations and will make sure to compute the gradients for all used parameters.<br>
You can treat “models” as a standard layer.<br>
In fact, your custom models and PyTorch’s layers both derive from <code>nn.Module</code>, so you are able to chain models as you would do with layers.</p>
RuntimeError: The size of tensor a (320) must match the size of tensor b (10) at non-singleton dimension 2

username: rai24
cooked: <p>correct = 0<br>
total = 0<br>
with torch.no_grad():<br>
for data in testloader:<br>
images, labels = data<br>
outputs = train_model(images)<br>
_, predicted = torch.max(outputs.data, 1)<br>
total += labels.size(0)<br>
correct += (predicted == labels).sum().item()</p>
<p>print(‘Accuracy of the network on the 10000 test images: %d %%’ % (<br>
100 * correct / total))</p>
<p>Error occurred at --------------------------------------------------------------------------<br>
RuntimeError                              Traceback (most recent call last)<br>
 in <br>
7         _, predicted = torch.max(outputs.data, 1)<br>
8         total += labels.size(0)<br>
----&gt; 9         correct += (predicted == labels).sum().item()<br>
10</p>
Mask Select with Variable Lengths

username: ameliatqy
cooked: <p>Hey guys,</p>
<p>I was wondering, is there a way to mask select a Tensor that would end up with inconsistent dimensions? For example: we have tensor [[1,2,], [3,4]] and mask [[1,0],[1,1]]. When we put it through a custom_mask_function:</p>
<p>custom_mask_function([[1,2,], [3,4]], [[1,0],[1,1]]) = [[1], [3,4]]</p>
<p>As you can see, the tensor does not have consistent dimensions as the first array is of length one and the second one of length two. Is creating something like custom_mask_function possible?</p>
<p>Thanks in advance!</p>
Initializing a Tensor

username: Yasesf
cooked: <p>Hey,<br>
I have a 4D Tensor (say A)  and an integer b which will change with respect to each model. and I want to initialize a 5D tensor of size (b,A). for example if A.size() is [32,1,3,3] and b is 5, I want to initialize a tensor of size [5,32,1,3,3]. I do torch.zeros(b, A.size()) but it doesn’t do the job!<br>
I would appreciate if someone helps out.</p>

username: ayalaa2
cooked: <p>You can use asterisks in Python to unpack.</p>
<p><code>torch.zeros((b, *A.size()))</code></p>
<p>That should do the trick^</p>
'Good' Discriminator -> no Learning for Generator

username: sigma_x
cooked: <p>When training GANs, a ‘bad’ Discriminator D is the one that correctly classifies real and fake, generated by Generator G data, yielding small derivatives. Therefore, a D that outputs values ~0.5 reagrdless of the input is supposed to be ‘good’, yielding large derivatives.</p>
<p>In one of the models I’m training an error D(1-G(z)) quickly reduces to ~0.69 (i.e. -log(0.5)), which should be good, because we are fooling D, but G actually doesn’t learn anything from it quickly converging to outputting all 0s (for example).</p>
<ol>
<li>What is the reason for this convergence?</li>
<li>Why doesn’t G learn from the errors?</li>
</ol>

username: ayalaa2
cooked: <blockquote>
<p>Therefore, a D that outputs values ~0.5 regardless of the input is supposed to be ‘good’</p>
</blockquote>
<p>This isn’t necessarily a flag that your generator and discriminators are amazing. If your Discriminator refuses to adjust itself, then your discriminator fell into a local optimum that it’s unable to escape from.<br>
There’s can be many reasons for this to occur, GANs are notoriously tricky to train. I believe this might be affecting both behaviors you’re noticing.</p>
<p>I would suggest exploring the learning rates you have set, different batch sizes, loss functions, and/or optimizers. I would also suggest logging your losses (both gen and discrim) over time to help provide more insight. You shouldn’t see one model getting better than the other, it should be a consistent switch in the “winner” or a steady chase after the winner. If you see one model slowly winning over time, the training is diverging.</p>

username: sigma_x
cooked: <aside class="quote no-group" data-username="ayalaa2" data-post="2" data-topic="74259">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ayalaa2/48/21195_2.png" class="avatar"> ayalaa2:</div>
<blockquote>
<p>should be a consistent switch in the “winner” or a steady chase after the winner</p>
</blockquote>
</aside>
<p>I’m not sure I understand this bit. By ‘winner’ do you mean G or D?</p>

username: ayalaa2
cooked: <p>Either one really.</p>
<p>When I say consistent switch, I mean, the generator wins, then loses some iterations later, wins some iterations later, and so on.</p>
<p>When I say a steady chase, it means either G or D is contentiously winning BUT isn’t leaving the other behind.</p>
<p><em>From experience</em>, I normally see the consistent switch behavior. Depending on your task, it’s hard to determine the performance of your GAN (other than if a generated sample <em>looks</em> realistic)</p>
How to use Ignite with multiple output model / loss?

username: odats
cooked: <p>I have a model with multiple outputs. How to use multiple losses?</p>
<pre><code class="lang-auto">import torch.nn as nn

class NeuralNetwork(nn.Module):
  def __init__(self):
    super(NeuralNetwork, self).__init__()
    self.linear1 = nn.Linear(in_features = 3, out_features = 1)
    self.linear2 = nn.Linear(in_features = 3,out_features = 2)

  def forward(self, x):
    output1 = self.linear1(x)
    output2 = self.linear2(x)
    return output1, output2
</code></pre>
<p>output1 - CrossEntropyLoss, output2 - MSE</p>
<p>For one output it is easy:</p>
<pre><code class="lang-auto">criterion = nn.CrossEntropyLoss()
trainer = create_supervised_trainer(model, optimizer, criterion, device=device)
</code></pre>

username: Muhammad_Izaz
cooked: <p>Try to use this, this will work</p>
<p>def forward(self, x):<br>
out= self.linear1(x)<br>
out = self.linear2(out)<br>
return out</p>

username: vfdev-5
cooked: <p><a class="mention" href="/u/odats">@odats</a> please see these issues labelled as ‘question’:</p>
<ul>
<li><a href="https://github.com/pytorch/ignite/issues/353" rel="nofollow noopener">https://github.com/pytorch/ignite/issues/353</a></li>
<li><a href="https://github.com/pytorch/ignite/issues/819" rel="nofollow noopener">https://github.com/pytorch/ignite/issues/819</a></li>
</ul>
<p>HTH</p>
com.facebook.jni.CppException: version_ <= kMaxSupportedFileFormatVersion

username: hurricane013
cooked: <p>I am trying to integrate keypoint_rcnn on android. But after solving errors i succeeded to generate .pt file using following script with torch 1.5 and torchvision 0.6.0</p>
<pre><code class="lang-auto">import torch
import torchvision

model = torchvision.models.detection.keypointrcnn_resnet50_fpn(pretrained=True)
model.eval()
example = torch.rand(1, 3, 800, 800)
traced_script_module = torch.jit.script(model, example)
traced_script_module.save("/home/bemrr/Desktop/model4.pt")
</code></pre>
<p>But now while installing it on android device, error is like this:</p>
<blockquote>
<p>java.lang.RuntimeException: Unable to start activity ComponentInfo{org.pytorch.helloworld/org.pytorch.helloworld.MainActivity}: com.facebook.jni.CppException: version_ &lt;= kMaxSupportedFileFormatVersion INTERNAL ASSERT FAILED at …/caffe2/serialize/inline_container.cc:132, please report a bug to PyTorch. Attempted to read a PyTorch file with version 3, but the maximum supported version for reading is 2. Your PyTorch installation may be too old. (init at …/caffe2/serialize/inline_container.cc:132)</p>
</blockquote>
<p>Thats why I am assuming that as my gradle dependency is like</p>
<pre><code class="lang-auto">implementation 'org.pytorch:pytorch_android:1.4.0'
implementation 'org.pytorch:pytorch_android_torchvision:1.4.0'
</code></pre>
<p>is incompatible with .pt file which is generated using torch 1.5, so I want to take a pull from latest pytorch repo and then build torch mobile using source. But I can not find any good instructions anywhere. Can someone please help me?</p>

username: David_Reiss
cooked: <aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://pytorch.org/favicon.ico?" class="site-icon" width="32" height="32">
      <a href="https://pytorch.org/mobile/android/#building-pytorch-android-from-source" target="_blank">pytorch.org</a>
  </header>
  <article class="onebox-body">
    <img src="https://pytorch.org/assets/images/pytorch-logo.png" class="thumbnail onebox-avatar" width="500" height="500">

<h3><a href="https://pytorch.org/mobile/android/#building-pytorch-android-from-source" target="_blank">PyTorch</a></h3>

<p>An open source deep learning platform that provides a seamless path from research prototyping to production deployment.</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>(See the section “Building PyTorch Android from Source”)</p>
Is there an elegant way to calculate integral image using PyTorch api？

username: chenchr
cooked: <p>Hello. I want to calculate the integral image, also known as summed area table of a tensor, is there an efficient way to do it using PyTorch api ?<br>
Thanks !</p>

username: albanD
cooked: <p>Hi,</p>
<p>I think you can implement this by doing two <code>torch.cumsum</code>, one in each dimension that correspond to the height and width of your image. No?</p>

username: chenchr
cooked: <p><a class="mention" href="/u/alband">@albanD</a> Thank you. I found that. It did works. However I realize that a integral image is not what I want…<br>
My task is that:<br>
given a edge image, for every pixel, I need to specify whether the neighbor pixel in a rectangular area is on the same side with that central pixel. Initialy I think integral image can do that, however I found the counter example that it fails… Any suggestion ?</p>

username: albanD
cooked: <p>I would say, if you image contains only 0s and 1s where the edges are.<br>
You can put it in a convolution whole kernel size the the rectangular area you want. And all weights are 1 (no bias).<br>
Every point of the output that has a non-zero value has it’s rectangle touching an edge.<br>
Is that good for you?</p>

username: chenchr
cooked: <p><a class="mention" href="/u/alband">@albanD</a>  Maybe I’ve expressed unclearly.  The task is that: in every rectangular center on one pixel, I want to determine those pixel that stand on the same side with the central pixel, and those that don’t stand on the same side with the central pixel. For example, if for one central pixel, there are no edges go across the rectangular corresponding to the pixel, them we can conclude that all the pixel in that rentangular are standing on the same side with the central pixel. if there is a edge goes across the rectangular, them I need to seperate the pixel in the rectangular into two categories. One for those stand on same side with the central pixel, one for the others.</p>

username: albanD
cooked: <p>I’m not sure you can do that without having to loop through most of these rectangles  to generate these two sets.</p>

username: chenchr
cooked: <p><a class="mention" href="/u/alband">@albanD</a>  Doing that by customing cuda layer is possible. However it’s not device friendly…  Therefore I am trying to figure out whether it can be done by Pytorch api. To generate all the rectangular, I can use <code>unfold</code>, but I am not sure whether the task can be accomplished by some calculation, without resorting to cuda layer…</p>

username: marioviti
cooked: <p>Hello,<br>
Just a small intrusion, If you compute the integral image and than use a binary kernel with ones in the edges and all zeros everywhere else to compute the local sum, it should be the same as directly doing your convolution with all ones.<br>
If you use a Gpu I don’t think you get any advantage at doing convolution this way.</p>
Using torch.save and torch.load on very heavy files

username: yumu
cooked: <p>i used torch.save to save two very big tensors one being the training dataset and the other one being the test dataset (separated by torch.utils.data.random_split)</p>
<p>the original tensor was of size (4803, 354, 4000)</p>
<p>when i do that it’s very slow to save (wich is understandable), both files have the same size (wich is weird since the testing set is only 20% of the original while the training set is 80% of the original)<br>
and when i try to load the dataset using torch.load it gives me this error (note : i am on a gpu kernel but i get the same error on a cpu kernel)</p>
<p>RuntimeError                              Traceback (most recent call last)<br>
 in <br>
----&gt; 1 training_set = torch.load(‘trainaction.pt’)<br>
2 testing_set = torch.load(‘testaction.pt’)<br>
3<br>
4 batch_size = 200<br>
5 DATA_PATH = ‘./data’</p>
<p>~.conda\envs\py36\lib\site-packages\torch\serialization.py in load(f, map_location, pickle_module, **pickle_load_args)<br>
527             with _open_zipfile_reader(f) as opened_zipfile:<br>
528                 return _load(opened_zipfile, map_location, pickle_module, **pickle_load_args)<br>
–&gt; 529         return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)<br>
530<br>
531</p>
<p>~.conda\envs\py36\lib\site-packages\torch\serialization.py in _legacy_load(f, map_location, pickle_module, **pickle_load_args)<br>
700     unpickler = pickle_module.Unpickler(f, **pickle_load_args)<br>
701     unpickler.persistent_load = persistent_load<br>
–&gt; 702     result = unpickler.load()<br>
703<br>
704     deserialized_storage_keys = pickle_module.load(f, **pickle_load_args)</p>
<p>~.conda\envs\py36\lib\site-packages\torch\serialization.py in persistent_load(saved_id)<br>
661             location = _maybe_decode_ascii(location)<br>
662             if root_key not in deserialized_objects:<br>
–&gt; 663                 obj = data_type(size)<br>
664                 obj._torch_load_uninitialized = True<br>
665                 deserialized_objects[root_key] = restore_location(obj, location)</p>
<p>RuntimeError: [enforce fail at …\c10\core\CPUAllocator.cpp:47] ((ptrdiff_t)nbytes) &gt;= 0. alloc_cpu() seems to have been called with negative number: 18446744066554005248</p>

username: richard
cooked: <p>This sounds like a bug or some limitation. Either way, it shouldn’t fail with an assertion failure, the framework should be able to provide a nice error message. Could you file an issue at <a href="http://github.com/pytorch/pytorch/">http://github.com/pytorch/pytorch/</a> ?</p>

username: yumu
cooked: <p>i filed the issue</p>
<p>do you have any idea on how to solve the problem ? also do you think it comes from the save or from the load ? both files having the same weight when they shouldn’t seems suspicious but im not sure</p>
Brightness Normalization in Alexnet paper

username: xianqian
cooked: <p>I am trying to implement alexnet. In the paper, they have mentioned using Brightness normalization(3.3). I am not quite sure how to implement this in PyTorch. Is there any documentation I am missing? or do I have to implement this myself?</p>
<p><a href="https://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf" rel="nofollow noopener">Alexnet Paper</a></p>

username: xianqian
cooked: <p>Respected community members,<br>
I have no clue how to move on with this. I am stuck in t<br>
his implementation. Can I atleast get some help regarding how to approach this problem? Or what do you do when you get to a point where you dont understand how to implement the paper anymore?</p>

username: simaiden
cooked: <p>The “brightness normalization” it’s the same as “Local Response Normalization” and the expression is given in the same section. It’s a type of normalization but it seems it’s not using frecuently (<a href="https://stats.stackexchange.com/questions/145768/importance-of-local-response-normalization-in-cnn" rel="nofollow noopener">related</a>). In fact, if you look at pytorch alexnet implementation (<a href="https://github.com/pytorch/vision/blob/master/torchvision/models/alexnet.py" rel="nofollow noopener">link</a>), this not use any kind of normalization .</p>
<p>So my advice is try to not spend so much time in some implementation that will have minor or nothing improvements.</p>

username: xianqian
cooked: <p>Thanks a lot, sir for the reply. I am thinking how did you find this? And also I am just implementing all these to get myself to learn some stuff. I am planning to implement things - if you have any suggestion you can let me know here.<br>
Also what can I do to understand pytorch better?</p>
Each csv is one training example

username: skw
cooked: <p>Hi,<br>
I have <strong>many</strong> csv files that has multiple rows and columns which are mostly floating point numbers (some are categorical but one-hot encoded).<br>
<strong>Each csv fil</strong>e is the representation of <strong>one training example</strong>. It contains dependent and independent variables in the same file.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/6/f611d13dc06cd86c090e75f87b3c700f1358b270.png" data-download-href="https://discuss.pytorch.org/uploads/default/f611d13dc06cd86c090e75f87b3c700f1358b270" title="Capture"><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/6/f611d13dc06cd86c090e75f87b3c700f1358b270.png" alt="Capture" data-base62-sha1="z6PD8qfNorY5cVUK2a9kEYn4Yb6" width="690" height="343" data-dominant-color="E7E8E9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Capture</span><span class="informations">948×472 17.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
** The above image is the representation of one csv</p>
<p>I want to build a simple ann or any other neural net model. I have problem in processing input data. As each csv is one single training example, in which format should i have to store data to pass to a neural net.</p>
<p>Thanks in advance,<br>
skw</p>

username: yash1994
cooked: <p>Hi,</p>
<p>You can build a custom data loader, something like <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html" rel="nofollow noopener">this</a>. Where you initialize your data loaders with all path of all CSV files and in <code>__getitem__</code> method you can load CSV and return input and target. If your data is small enough to easily fit into your memory then you can load all CSV files in data loader initialization itself.</p>

username: skw
cooked: <p>Hi,<br>
Thank you for the reply.<br>
I forgot to mention that, the length/shape of the csv (only rows) may vary</p>
Gradients over Transformation and reshape operations

username: AnupKumarGupta
cooked: <p>I have to apply the following operations on a video (frame-wise)</p>
<ol>
<li>RGB to grayscale</li>
<li>Center Crop</li>
<li>Color Normalize</li>
<li>Give the video (the sequence of the frames) as input to a model.</li>
</ol>
<p>Moreover, I have to compute gradients over these operations later. I asked this <a href="https://discuss.pytorch.org/t/gradients-over-transformation-operations/73934">Question</a> and was suggested to use <a href="https://github.com/kornia/kornia" rel="nofollow noopener">Kornia</a> which supports gradients over these operations, which works well for me.</p>
<p>There is another problem though. The model expects an input of the shape (1, <span class="hashtag">#frames</span>, height, width) whereas the above operations expect the shape (<span class="hashtag">#frames</span>, <span class="hashtag">#channels</span>, height, width). So I need to apply <code>reshape()</code> operation, due to which I am unable to compute the gradients. Kindly provide me with a workaround or point out what am I doing wrong.</p>

username: albanD
cooked: <p>Hi,</p>
<p>The reshape operation is differentiable in pytorch. So it should not prevent you from computing any gradient.<br>
Also I think you want to permute dimensions, not reshape them: <code>input = input.permute(1, 0, 2, 3)</code>.</p>

username: AnupKumarGupta
cooked: <p>Thanks a lot. Is the permute operation differentiable too?</p>

username: albanD
cooked: <p>Yes. In general, all the operations in pytorch that take continuous numbers and return continuous numbers are differentiable.</p>
Regression with nn.Sequential network fail

username: tr3cks
cooked: <p>Hi there I was trying to use nn.Sequential module in a simple regression problem, just a line. The thing is that I print MSELoss every iteration and it is trying to converge at 1, intead of the values I am wondering.</p>
<p>Here is the code:</p>
<pre><code class="lang-auto">NN = nn.Sequential(
    nn.Linear(1,3),
    nn.Sigmoid(),
    nn.Linear(3,4),
    nn.Sigmoid(),
    nn.Linear(4,7),
    nn.Sigmoid(),
    nn.Linear(7,1),
    nn.Sigmoid()
)

criterion = nn.MSELoss()
optimizer = optim.Adam(NN.parameters(), lr=0.01)

X = torch.rand(200,1)
y = 2*X + 3

print(X.shape,y.shape)
for i in range(1000):  
    NN.train()
    optimizer.zero_grad()
    out = NN(X)
    #print(out,y)
    loss = criterion(y,out)
    loss.backward()
    optimizer.step()

    with torch.no_grad():
        print ("#" + str(i) + " Loss: " + str(loss.item()))  
</code></pre>
<p>If I try this to check the results:</p>
<pre><code class="lang-auto">newX = []
newY = []
for i in range(5):
    r = random.uniform(0, 1)
    newX.append(r)
    newY.append(NN(torch.tensor([r])).item())
</code></pre>
<p>I get <code>[0.9996629953384399, 0.9996600151062012, 0.9996635913848877, 0.9996637105941772, 0.9996615648269653]</code>.</p>
<p>I have tryed lot of things changing every on the network but don’t know what is happening.</p>

username: yash1994
cooked: <p>Hi,</p>
<p>There is an issue with your model architecture, your target values are all greater than one (because of 2x+3 operation) and last operation in your network is <code>nn.Sigmoid()</code> which will return values in the range 0 to 1 so your model would never be able to converge.</p>

username: tr3cks
cooked: <p>Oh,I feel so silly its true. And what should I do, change the structure of the network or normalize inputs and outputs values?</p>

username: yash1994
cooked: <p>Just remove last <code>nn.Sigmoid()</code> operation from your model.</p>

username: tr3cks
cooked: <p>That works. Thank you really much!!</p>
Multithreaded DataLoader: correct way to initialize external libraries

username: vadimkantorov
cooked: <p>I’d like to use an external audio i/o library that requires call-once initialization before usage (e.g. torchaudio uses sox that requires <a href="https://pytorch.org/audio/#torchaudio.initialize_sox" rel="nofollow noopener">https://pytorch.org/audio/#torchaudio.initialize_sox</a> ) and call-once resource cleaning after usage is finished.</p>
<p>What is the correct way to use multi-threaded DataLoader wrt shared library initialization and cleanup? Does worker thread creation reload the shared library many times in separate processes? Does <code>start_method</code> affect any of these?</p>
<p>Usecase: I designed my own ffmpeg-based C extension for reading audio: <a href="https://github.com/vadimkantorov/readaudio/blob/master/decode_audio.c#L48" rel="nofollow noopener">https://github.com/vadimkantorov/readaudio/blob/master/decode_audio.c#L48</a>. In most ffmpeg versions (up until 4.0 which is not shipped in stable Ubuntu yet) the user must call onc <code>av_register_all()</code> once before initialization.</p>
Cublas runtime error : an access to GPU memory space failed

username: Sfitsos
cooked: <p>All of a sudden I get the following error when running my pytorch code.</p>
<p>cublas runtime error : an access to GPU memory space failed.</p>
<p>Occasionally I also get:</p>
<p>THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1579022060824/work/aten/src/THC/THCCachingHostAllocator.cpp line=278 error=700 : an illegal memory access was encountered</p>
<p>I have two 2080Ti GPUs on the machine. The error only happens when I run my code on the gpu with id:1 (the GPU with my monitor connected to it as well). Could this be a hardware issue? I have been able to run the code on either GPU in the past.</p>

username: ptrblck
cooked: <p>Could you check the GPU memory on this device with <code>nvidia-smi</code> and make sure you are not running out of memory?<br>
I assume the code works fine on the first device?</p>

username: Sfitsos
cooked: <p>Yes the program takes about 3Gb of memory on the GPU. The graphics just take 119Mb (Unfortunately due to self-isolation and lockdown I can’t know if the display looks OK). If I make the batch size very small (2) then the program runs (taking 1.5 Gb on the memory).</p>

username: Sfitsos
cooked: <p>OK so I started gradually increasing the batch size and now all of a sudden I don’t get the error (??). I even increased it past what I was using originally (it now occupies almost all of 11 Gb) and it works. I wonder if this is an issue caused by overheating? The card is the one on the bottom so it gets  20C higher than the other one.</p>

username: ptrblck
cooked: <p>The error shouldn’t be raised by overheating, but your system / GPU should instead shutdown to prevent damage.</p>
<p>Maybe some zombie process was still using some GPU memory and thus creating an OOM error.</p>

username: Sfitsos
cooked: <p>Yeah although the error persisted even after several reboots. Also it happens still now although more spuriously ¯_(ツ)_/¯.</p>
Geting new value of calulated tensor

username: matej1408
cooked: <p>Suppose that I created some computational graph like this</p>
<pre><code class="lang-auto">a = torch.autograd.Variable(torch.ones((2,2)), requires_grad=True)
b = a*2
</code></pre>
<p>then I update <em>a</em> values like this</p>
<pre><code class="lang-auto">a.data = a.data*2
</code></pre>
<p>My question how can I get new values of <em>b</em> (that should be now [[4,4],[4,4]]) without “creating” new <em>b</em> (e.g. running b = a*2)?</p>

username: ptrblck
cooked: <p><code>Variables</code> are deprecated since PyTorch <code>0.4</code> and the usage of the <code>.data</code> attribute is not recommended, will be removed in the future, and might yield side effects.</p>
<p>That being said, could you explain your use case a bit more and what you are trying to achieve?</p>

username: matej1408
cooked: <p>Lets say that my <code>b</code> is:</p>
<pre><code class="lang-auto">b = 2*a
</code></pre>
<p>but to determine that <code>b = 2*a</code> and not <code>b = 3*a</code> or some other coefficient I had to look up giant table. After that I define some loss function and optimize <code>a</code>  with Adam. After <code>a</code> is updated I want to find new value of b without doing look up again.<br>
You can argue that I just need to save my  coefficient in some variable and use that, but my calculation is really complex and I would spend a lot of time to figure out how to rewrite everything in “tensor” form.</p>
<p>Edit: I am hoping that <code>b = 2*a</code> is already saved somewhere in the graph so I can just “evaluate” it again.</p>
PyTorch not working in jupyter?

username: useronhere
cooked: <p>I installed pytorch using conoda. However when I try to import torch in Jupyter I get: <code>ModuleNotFoundError: No module named 'torch'</code> Also when I run <code>where conda.exe</code> <code>conda.exe not found</code></p>
<p>Does anyone know how to fix my problem?</p>

username: Tony-Y
cooked: <aside class="quote" data-post="1" data-topic="31860">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/c89c15/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/torch-is-not-found-in-jupyter-notebook/31860">Torch is not found in jupyter notebook</a> 
  </div>
  <blockquote>
    I am using Ubuntu 18.04 and I have installed the anaconda and installed the pytorch using conda install pytorch-cpu torchvision-cpu -c pytorch. The import of pytorch works well in terminal but whenever I try to import it in jupyter notebook, it fails. 
I have installed the whole anaconda two times but it is not working. I have tried with different environment but it is not working. I have also tried to update anaconda but it already up to date and also have tried to downgrad mkl to 2018
  </blockquote>
</aside>
How to stop training?

username: krishnavishalv
cooked: <p>I’m trying to implement Early Stopping in pytorch. I want to know how to stop training when the early stopping criterion is met ? Is there any function that achieves this ?</p>

username: ptrblck
cooked: <p><a href="https://github.com/ncullen93/torchsample/blob/master/README.md" rel="nofollow noopener">Here</a> you can find an implementation of early stopping. Maybe it can be useful for your use case.</p>

username: vpn
cooked: <p><a href="https://gist.github.com/stefanonardo/693d96ceb2f531fa05db530f3e21517d" rel="nofollow noopener">Here</a> is a simple implementation of early stopping. Hope this is helpful!</p>
Create a copy of a model along with loaded weights

username: jitesh
cooked: <p>I have trained a model and have saved weights(saving the state_dict) after training. Now, I want to create 2 copies of the same model with the weights loaded. I am using the following code for the purpose</p>
<pre><code class="lang-auto">    PATH_TO_WEIGHTS = 'path/model.pth'
    model = MyModel().cuda() 
    pretrained_weight = torch.load(PATH_TO_WEIGHTS)
    model.load_state_dict(pretrained_weight)
    modelCopy = model
</code></pre>
<p>I would like to know if <code>modelCopy</code> would also have the same weights which are loaded from my saved checkpoint or do I have to do something like <code>modelCopy = deepcopy(model)</code>  or If I have to again load the weights for <code>modelCopy</code>?</p>
<p>Any help would be appreciated. Thanks!</p>

username: ptrblck
cooked: <p><code>modelCopy</code> is referencing <code>model</code>, so that parameter changes will be reflected in both models.<br>
If you want to use the same <code>state_dict</code> in two independent models, you could use <code>deepcopy</code> or initialize a second model and load the <code>state_dict</code> again.</p>
<p>This code demonstrated the referencing:</p>
<pre><code class="lang-python"># Setup
model = nn.Linear(1, 1)
sd = model.state_dict()

# Load state_dict
modelA = nn.Linear(1, 1)
modelA.load_state_dict(sd)

modelB = modelA

# Check params
for pA, pB in zip(modelA.parameters(), modelB.parameters()):
    print((pA == pB).all())

&gt; tensor(True)
&gt; tensor(True)

# modelB is referencing modelA
with torch.no_grad():
    modelA.weight.fill_(100.)

print(modelA.weight)
&gt; Parameter containing:
tensor([[100.]], requires_grad=True)
print(modelB.weight)
&gt; Parameter containing:
tensor([[100.]], requires_grad=True)
</code></pre>

username: jitesh
cooked: <p>Thanks a lot. The explanation, especially with the help of the code snippets made it very clear.</p>
TypeError: Expected Ptr<cv::UMat> for argument '%s'

username: yibrah
cooked: <p>I am a beginner.<br>
in this code, I want to read multiple images and then denoise using median filtering. but I got this error ‘TypeError: Expected Ptr<a>cv::UMat</a> for argument ‘%s’’</p>
<p>import cv2<br>
import glob<br>
import matplotlib.pyplot as plt<br>
import numpy as np<br>
import random<br>
from PIL import Image<br>
image_list = []<br>
path = “K:/Dataset/All_images/resized_images/<em>.</em>”<br>
for file in glob.glob(path):<br>
a= cv2.imread(file)<br>
denoised_images = []<br>
a = Image.open(file)<br>
image_list.append(a)<br>
for image in image_list:<br>
image = cv2.cvtColor(image_list, cv2.COLOR_RGB2BGR)<br>
noisy = np.zeros(a.shape, np.uint8)<br>
p = 0.2<br>
for i in range(noisy.shape[0]):<br>
for j in range(noisy.shape[1]):<br>
r = random.random()<br>
if r &lt; p/2:<br>
noisy[i][j] = [0, 0, 0]<br>
elif r &lt; p:<br>
noisy[i][j] = [255, 255, 255]<br>
else:<br>
noisy[i][j] = image[i][j]<br>
denoised = cv2.medianBlur(noisy, 5)<br>
output = [image, denoised ]<br>
titles = [‘Original’, ‘Denoised’]<br>
for i in range(2):<br>
plt.subplot(1, 2, i+1)<br>
plt.imshow(output[i])<br>
plt.title(titles[i])<br>
plt.xticks([])<br>
plt.yticks([])<br>
plt.show()</p>
<p><span class="hashtag">#if</span> <strong>name</strong> == “<strong>main</strong>”:</p>
<h1>main()’’’</h1>

username: ptrblck
cooked: <p>This seems to be rather an OpenCV issue than a PyTorch one and searching for this error message yields a similar <a href="https://github.com/opencv/opencv/issues/15833">issue</a> in the OpenCV repo. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
Frozen Model is Affected by Optimizer

username: paarandika
cooked: <p>I have a trained a CRNN model containing LSTM, batch norm and convolutional layers. Using the trained model I’m trying to train the input of the model. (Trained model is loaded from the disk)</p>
<pre><code class="lang-auto">image = torch.rand(1, 1, input_size[0], input_size[1]).cuda()
loss_function = CTCLoss().to(device)
optimizer = optim.Adam([image.requires_grad_()], lr=lr)
</code></pre>
<p>Image is saved after training. This saved image generates a different output with a newly loaded model than the model used to train it. By using a single image I discovered that this difference happens only if <code>optimizer.step()</code> is called.<br>
Am I missing something here? Does optimizer change the model even this case? (I haven’t included model params in optimizer)</p>

username: yash1994
cooked: <p>Hi,</p>
<p>I am not getting what you are trying to achieve by “I’m trying to train the input of the model.”. When <code>optimizer.step()</code> is called, the gradients are accumulated to <code>image</code> tensor which changes the value of the tensor. So I am guessing that your input (<code>image</code> tensor) is getting changed and not your model.</p>

username: paarandika
cooked: <p>I’m trying to do something similar to this <a href="https://pytorch.org/tutorials/advanced/neural_style_tutorial.html#sphx-glr-advanced-neural-style-tutorial-py" rel="nofollow noopener">tutorial</a>.<br>
Basically my intention is to find the optimal image expected by the model, given a label.</p>
<p>" So I am guessing that your input ( <code>image</code>  tensor) is getting changed and not your model."<br>
This is exactly what I want. But it appears that it changes the model as well. That’s the problem I’m having.</p>

username: yash1994
cooked: <p>Hi,</p>
<p>I understood your problem, but it’s hard to tell with just three lines of code. You can debug your optimizer by checking <code>opt.param_groups</code>, it will give you all the parameter it tries to optimize. Hope this helps.</p>

username: paarandika
cooked: <p>Thanks. I will try that and post the results if I find anything weired in there.</p>

username: paarandika
cooked: <p>I figured out the issue here. CRNN model has two batchNorm layers. They behave differently in <code>train</code> and <code>eval</code> modes.</p>
add_module() missing 1 required positional argument: 'module'

username: CP995
cooked: <p>I use the same file in two Google colab, ones can run, but another get a TypeError, which is 'add_module() missing 1 required positional argument: ‘module’.<br>
The following code contains the add_module()</p>
<pre data-code-wrap="python"><code class="lang-python"># self.straight.add_module(nn.BatchNorm1d(num_features=out_channel))
self.straight.add_module(nn.ReLU())

length = duration - sum(kernels) + len(kernels)  # 卷积结束后作池差
self.straight.add_module(prefix+"Pooling",
                                 nn.MaxPool1d(kernel_size=length))
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/5/f5ba6c22ced2da65bdac1cd0446a19864ccf85fb.png" data-download-href="https://discuss.pytorch.org/uploads/default/f5ba6c22ced2da65bdac1cd0446a19864ccf85fb" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/5/f5ba6c22ced2da65bdac1cd0446a19864ccf85fb_2_636x500.png" alt="image" data-base62-sha1="z3Oo5668qSwa9N0ikz5Fo0A3xID" width="636" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/5/f5ba6c22ced2da65bdac1cd0446a19864ccf85fb_2_636x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/5/f5ba6c22ced2da65bdac1cd0446a19864ccf85fb_2_954x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/f/5/f5ba6c22ced2da65bdac1cd0446a19864ccf85fb_2_1272x1000.png 2x" data-dominant-color="3D3D3E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1318×1035 139 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p><code>.add_module</code> expects a <code>name</code> and <code>module</code> as the arguments. Could you add the name for the relu module and try it again?</p>

username: CP995
cooked: <p>Thanks, after adding the name, it works!</p>
Memory and speed issue (nn.conv3d and functional conv3d)

username: JayHan
cooked: <p>Hi, I am having trouble with some weird behavior of the Convolution layer… I couldn’t find sufficient information yet.</p>
<p>I found two strange things, and I will write small codes to test it below.</p>
<p>First thing is, (F is torch.nn.functional)</p>
<pre><code class="lang-auto">with torch.no_grad():
    test = torch.ones(1, 1, 200, 1200, 1900).cuda().float()
    weight = torch.ones(1, 1, 40, 40, 100).cuda().float()
    answer = F.conv3d(test, weight)
    print(answer.size())

with torch.no_grad():
    test = torch.ones(1, 1, 200, 1200, 1900).cuda().float()
    weight = torch.ones(1, 1, 40, 40, 100).cuda().float()
    answer = F.conv3d(test, weight)
    print(answer.size())
</code></pre>
<p>If I execute this code(two block), first block shows result in about 5 seconds as torch.Size([1, 1, 161, 1161, 1801]). However, second print doesn’t appear although several minutes passes. Result is like below. Also, until first print, GPU usage was about 28%(3GB/11GB) and after first print, GPU usage grows until 57%(6.3GB/11GB).</p>
<pre><code class="lang-auto">(base) D:\codes&gt;python util.py (deleted private path)
torch.Size([1, 1, 161, 1161, 1801])
</code></pre>
<p>.</p>
<p>The second issue is about kernel size… Here is the snippet.</p>
<pre><code class="lang-auto">with torch.no_grad():
    test = torch.ones(1, 1, 200, 1200, 1900).cuda().float()
    weight = torch.ones(1, 1, 40, 40, 40).cuda().float() / 100
    weight2 = torch.ones(1, 1, 3, 3, 3).cuda().float() / 100

    answer1 = F.conv3d(test, weight)
    answer2 = F.conv3d(test, weight2)
    print(answer1.size())
    print(answer2.size())
</code></pre>
<p>If I only calculate answer1, it consume 3.2GB of gpu memory. However if I calculate answer2, I meet CUDA out of memory, also saying tried to allocate 7.56GiB. I don’t know how this difference made.</p>
<p>Also, I think I need a model that can calculate the huge size of the input so I test with 200, 1200, 1900.</p>
<p>.</p>
<p>Please give me an idea. Thanks.</p>

username: albanD
cooked: <p>Hi,</p>
<p>I think the problem in both cases is that these would correspond to very large convolutions.<br>
Also there are known issues (<a href="https://github.com/pytorch/pytorch/issues/32370">https://github.com/pytorch/pytorch/issues/32370</a>) with cudnn conv3d so might want to try to disable it with <code>torch.backends.cudnn.enabled=False</code> or try the benchmark mode to allow it to discover better algorithms with <code>torch.backends.cudnn.benchmark=True</code>.</p>

username: JayHan
cooked: <p>Thanks!!!</p>
<p>There are still some ‘not understandable’ thing … and maybe I can test some situations and organize them.</p>
<p>Thank you!</p>
What is the best way to deploy object detection and segmentation model to the cloud

username: Bill_CX
cooked: <p>Hi guys,<br>
I have a model written in Pytorch, and I wonder what is the best way to deploy it on the cloud with simple GUI allows user to use it simply. If you have some resources, please send it my way.<br>
Please help,<br>
Thank you!<br>
Bill</p>
Best way to create std::vector from torch::Tensor

username: goksug
cooked: <p>I need to create <code>std::vector</code> containers from the rows of a 2D tensor. I can slice the tensor to get the rows, and I need to convert those rows to <code>std::vector</code>. One option is, of course, to access all elements one by one, but there must be a better way to do it.</p>
<p>P.S. I don’t need to take ownership of the data, though I would appreciate it if there’s specific to be done when we want to do a deep copy.</p>
How to put a model into production?

username: mgloria
cooked: <p>I started learning Pytorch after completing the fastai course. <a href="https://github.com/fastai/fastai" rel="nofollow noopener">Fastai</a> library is based on Pytorch. In fastai, there is a simple way to export the model as (.pkl) can be seen <a href="https://github.com/fastai/fastai2/blob/master/fastai2/learner.py" rel="nofollow noopener">here</a> and a nice <a href="https://course.fast.ai/deployment_render.html" rel="nofollow noopener">tutorial</a> to create with it a simple web interface.</p>
<p>I would like to use the same web interface but with my pytorch trained model, which now I have saved as a .pt file.</p>
<p>Any good resources for a complete beginner to get started? I am looking for an easy to follow example or tutorial. Thanks a lot for guiding me in this journey! <img src="https://discuss.pytorch.org/images/emoji/apple/smiling_face_with_three_hearts.png?v=9" title=":smiling_face_with_three_hearts:" class="emoji" alt=":smiling_face_with_three_hearts:"></p>
Empty_cache behavior for multiple GPUs

username: Rakshit_Kothari
cooked: <p>A few jobs of mine were permanently stalled at <code>cuda.empty_cache</code>. I have access to 2 GPUs (8 GB, P4s) per job. I was hoping someone could shed light (or a link) on how <code>empty_cache</code> works for a multi GPU operation.<br>
I had come across a few people facing similar issues but can’t find that thread anymore.</p>