akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Memory usage during training and validation

BarCodeReader: <p>Here I observe some difference and just want to make sure:</p>
<p>I am running my model on dual-GPU, during <strong>training</strong>, each GPU will be used for 4GB.<br>
But during <strong>validation</strong> (i.e. the training is not finished yet, but just switch to model.eval() to run validation and will switch back to model.train() after validation), I will observe GPU 1 memory usage become 7GB while GPU 2 is still the same.</p>
<p>Then after validation finished and change back to training, the memory usage for GPU1 is not back to 4GB…it locked to 7GB…</p>
<p>is this normal??? I would expect it will change back to 4GB after validation is done…</p>
<p>what is the process for model.eval()? will it clear the memory after it finished? how does PyTorch assign the memory when it switches from train() to eval()? I am so curious!</p>
<pre><code class="lang-auto">training epoch 1:
GPU1 ---4GB
GPU2 ---4GB

validation:
GPU1 ---7GB
GPU2 ---4GB

training epoch 2:
GPU1 ---7GB (???)
GPU2 ---4GB
</code></pre>

ptrblck: <p>This is expected, since PyTorch uses a caching allocator to reuse the memory instead of reallocating it (which would be slow).<br>
If you want to reset the memory, you would have to delete all tensors, which might still be alive from the validation run, and call <code>torch.cuda.empty_cache()</code>. Note that this is only necessary, if you want other applications to use the memory.</p>

BarCodeReader: <p>hi <a class="mention" href="/u/ptrblck">@ptrblck</a>,</p>
<p>Thanks for the answer! Appreciate!</p>
Getting all zeros in mAP in faster rcnn on custom dataset

Tupakula_Mallikarjun: <p>Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.000<br>
Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.000<br>
Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.000<br>
Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000<br>
Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000<br>
Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.000<br>
Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.000<br>
Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.000<br>
Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.000<br>
Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000<br>
Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000<br>
Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.000</p>
<p>I am getting these values in IoU in implementing faster rcnn on custom dataset</p>
Iterative solver within loop

k9_kit: <p>If I have a minimzation function which produces some features A,b from some input X for a linear system of equations, I would be able to use <a href="https://pytorch.org/docs/stable/generated/torch.solve.html" rel="noopener nofollow ugc">torch.solve</a><br>
to solve the system of equation. However, I have non-linear system of equation which needs to be solved iteratively in the middle. I can optimize this function(with autograd) indiviually using SGD. Is there a way I can combine the iterative loop to the main optimization function?</p>
<pre><code class="lang-auto">x = input
A,b = f_1(x)
z = iterative_solve(f_2(A,b))
loss = z' - z
</code></pre>
Conv2d set weights and output channels of different sizes

Levon: <p>Hi,</p>
<p>Please help me solve the following confusion:<br>
I have 4 filters of (4,4) size. The values of these filters I want to assign to my conv2d weights and then visualize them.<br>
Below I define my model for greyscale image. When I run the code - it works correctly.<br>
But when I change the out_channels from 4 to 5 or 2, I expect the model to stop working as I initialize weights for 4 filters, but in output expect 5 or 2. But my model still works, returns 4 layes and no error is thrown. Why this happens?</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.nn.functional as F

class Net(nn.Module):
    
    def __init__(self, weight):
        super(Net, self).__init__()
        # assumes there are 4 grayscale filters
        self.conv = nn.Conv2d(1, 4, kernel_size=(4, 4), bias=False)
        self.conv.weight = torch.nn.Parameter(weight)

    def forward(self, x):
        conv_x = self.conv(x)
        activated_x = F.relu(conv_x)
        return conv_x, activated_x
    
weight = torch.from_numpy(filters).unsqueeze(1).type(torch.FloatTensor)
model = Net(weight)
print(model)
</code></pre>

ptrblck: <p>Your code is not fully defined, as e.g. <code>filters</code> are missing.<br>
However, it seems to work fine and the output shape corresponds to the passed number of filters as seen here:</p>
<pre><code class="lang-python">class Net(nn.Module):
    def __init__(self, weight):
        super(Net, self).__init__()
        # assumes there are 4 grayscale filters
        self.conv = nn.Conv2d(1, 4, kernel_size=(4, 4), bias=False)
        self.conv.weight = torch.nn.Parameter(weight)

    def forward(self, x):
        conv_x = self.conv(x)
        activated_x = F.relu(conv_x)
        return conv_x, activated_x


weight = torch.randn(4, 1, 4, 4)
model = Net(weight)
x = torch.randn(1, 1, 24, 24)
out = model(x)
print(out[0].shape)
&gt; torch.Size([1, 4, 21, 21])

weight = torch.randn(5, 1, 4, 4)
model = Net(weight)
out = model(x)
print(out[0].shape)
&gt; torch.Size([1, 5, 21, 21])
</code></pre>

Levon: <p>Hi, Thanks for the answer.<br>
my point is that if I use  <code>nn.Conv2d(1, 5, kernel_size=(4, 4), bias=False)</code> , i.e. output channels 5, but still remain with lets say 4 filters <code> weight = torch.randn(4, 1, 4, 4)</code> ), it still works (I was assuming that I <strong>must</strong> provide 5 filters). Probaly it is  allowed to overwrite 5 output channels to 4 (please let me know whether I am right or wrong). It was very strange to me and this behaviour is prone to potential bugs.</p>

ptrblck: <p>Thanks for the follow-up. I assume you think it should be disallowed to pass a new <code>weight</code> kernel with a different number of filters, since the original layer was initialized with another number of kernels?<br>
If so, then note that it’s not disallowed to manipulate the underlying parameters (at your own risk).<br>
While this might be prone to potential bugs, I would consider this an advanced approach and don’t think we should disallow it as it could limit potentially valid use cases.</p>
Prevent copying when num_workers>1

G.M: <p>Hi, I’m loading my data using a DataLoader with <code>num_workers&gt;1</code>. As far as I know, Pytorch uses multiprocessing, which copies the data to different processes. Is there a way to prevent the tensors in my dataset from being copied? Would <code>Tensor().share_memory_()</code> work?<br>
Thanks</p>

ptrblck: <p>I’ve written some dummy code some time ago using <code>mp.Array</code> <a href="https://github.com/ptrblck/pytorch_misc/blob/master/shared_array.py">here</a>.<br>
Your approach might work as well, but I haven’t tried it yet.</p>

G.M: <p>Thanks, I’ll try it.</p>

jaks19: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> thanks for the code sample.</p>
<p>I wanted to ask a related question.</p>
<p>My dataset object does not contain a straightforward tensor that will be read in chunks by dataloader workers.</p>
<p>In fact, I have a custom python object which performs the role of a search tree and everytime a worker triggers the <code>__getitem__</code> method of the dataset, the search tree is used to get the appropriate data.</p>
<p>In general, how could I ensure that an arbitrary python object (neither pytorch tensor nor np array) is not copied when I used more than one worker in the dataloader? I have made sure I never write to the structure when loading data but I still get memory overhead due to copying.</p>
<p>Thanks for your help!</p>

ptrblck: <p>Since multiple workers are using multiple processes, you would have to use any approach so share the data between processes. The linked <code>mp.Array</code> object would be one way to use it.</p>
Installing PyTorch with CUDA for Python 2.7 on windows 10

CBMadsen: <p>I have now been stuck on this problem for 2 days and it seems to be a almost un-googlable problem. I know that pytorch does not support 2.7 since version 1.4 and I know that 2.7 has been deprecated, but I am trying to replicate the code from this repository: <a href="https://github.com/XiSHEN0220/ArtMiner" class="inline-onebox" rel="noopener nofollow ugc">GitHub - XiSHEN0220/ArtMiner: (CVPR 2019) Pytorch implementation of paper "Discovering Visual Patterns in Art Collections with Spatially-consistent Feature Learning"</a> which just so happen to use very outdated tools.</p>
<p>I am on a windows 10 PC with cuda 11.1 installed and I’ve tried countless “pip install” calls but I just cannot seem to find a torch version with cuda that support 2.7 on windows. This also made me attempt approaches with Docker and an Ubuntu VM in Virtualbox, but so far to no avail.</p>
<p>Has anyone been in the same situation and know how to solve it?</p>
<p>TL:DR<br>
I need a torch version with cuda that can be installed on python 2.7 for windows 10 and works with the cuda 11.1 toolkit.</p>
<p>Thank you all in advance!</p>

ptrblck: <p>As you’ve already mentioned, the current PyTorch binaries are not built anymore for Python2.7.<br>
I would guess the right approach would be to update the repository to use Python&gt;=3.6.<br>
Alternatively, you could try to build PyTorch from source using Python2.7 (with potentially some needed fixes) at your own risk.</p>

CBMadsen: <p>Thank you for the answer, <a class="mention" href="/u/ptrblck">@ptrblck</a>, but I gave up on trying to find a solution. I imagine building from source might work, but in my case it wasn’t worth it.</p>
CocoEval Evaluating Dataset

Dicko87: <p>Hi there,<br>
I have been using DETR on my own dataset and it works very well. I get a good mAP and Recall on the validation set. My question is, how to I run cocoEval to give me the same or similar results to what it got during model training. For example the model achieved and mAP of 0.89 on the validation set. I then decided to see if I could produce the same results again. I ran the model in eval mode on the dataset and set a confidence threshold &gt; 0.8 and saved the results in a json file. I then used cocoeval and gave the validation set json and my new resFile as inputs and the evaluation results gave me an mAP of 0.6, which isn’t right. How do I go about getting the same or similar results as to what the model achieved originally and consequently how do I adjust the confidence threshold and get the precision-recall curves for these different thresholds. Thank you.</p>
RuntimeError: mat1 and mat2 shapes cannot be multiplied (128x34400 and 41624x1376)

pgomez: <p>Hello! Im new using pytorch and I cant manage to find a proper solution to this error, could anybody help me?</p>
<p>class MaskCnnModel(ImageClassificationBase):<br>
def <strong>init</strong>(self):<br>
super().<strong>init</strong>()<br>
self.network = nn.Sequential(<br>
nn.Conv2d(3, 86, kernel_size=3, padding=1),<br>
nn.ReLU(),<br>
nn.Conv2d(86, 172, kernel_size=3, stride=1, padding=1),<br>
nn.ReLU(),<br>
nn.MaxPool2d(2, 2), # output: 172 x 43 x 43</p>
<pre><code>        nn.Conv2d(172, 258, kernel_size=3, stride=1, padding=1),
        nn.ReLU(),
        nn.Conv2d(258, 258, kernel_size=3, stride=1, padding=1),
        nn.ReLU(),
        nn.MaxPool2d(2, 2), # output: 344 x 21.5 x 21.5

        nn.Conv2d(258, 344, kernel_size=3, stride=1, padding=1),
        nn.ReLU(),
        nn.Conv2d(344, 344, kernel_size=3, stride=1, padding=1),
        nn.ReLU(),
        nn.MaxPool2d(2, 2), # output: 688 x 10.8 x 10.8

        nn.Flatten(), 
        nn.Linear(344*11*11, 1376), 
        nn.ReLU(),
        nn.Linear(1376, 688),
        nn.ReLU(),
        nn.Linear(688, 2))
    
def forward(self, xb):
    return self.network(xb)</code></pre>

user_123454321: <p>I think you need to replace</p>
<pre><code class="lang-auto">nn.Linear(344*11*11, 1376)
</code></pre>
<p>with</p>
<pre><code class="lang-auto">nn.Linear(344*10*10, 1376)
</code></pre>
<p>Note tha t in max pool the default is to have floored size (that is 21 instead of 21.5 and then 10 instead of 10.5). For example</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch
&gt;&gt;&gt; layer = torch.nn.MaxPool2d(2, 2)
&gt;&gt;&gt; a = torch.randn(1, 1, 5, 5)
&gt;&gt;&gt; b = layer(a)
&gt;&gt;&gt; b
tensor([[[[1.7285, 1.0654],
          [2.4715, 1.0278]]]])
&gt;&gt;&gt; b.shape
torch.Size([1, 1, 2, 2])
&gt;&gt;&gt;
</code></pre>
Get ALL indices of the maximum of a tensor

roxor: <p>Hi everyone,<br>
I am looking for a possibility to get ALL indices for the maximum for each row in a 2-D tensor.<br>
Let us define the following tensor:</p>
<pre><code class="lang-auto">a = torch.tensor([[5,8,9,6,9], [5,4,3,2,1]])
</code></pre>
<p>I already tried torch.max and torch.argmax, but I didn’t get the desired output:</p>
<pre><code class="lang-auto">torch.max(a, dim=1) # returns tensor([4,0])
max_val, idx = torch.max(a, dim=1, keepdim=True) # max_val is tensor([[9],[5]]) and idx is tensor([[4],[0]])
</code></pre>
<p>I am looking for a function which returns ALL indices of the max value for each column. So I expect the following tensor, since the 9 occurs twice in the first row and the 5 only one time in the second row.</p>
<pre><code class="lang-auto">torch.tensor([[2,4],[0]])
</code></pre>
<p>Anyone has an idea?<br>
Thanks in advance!</p>

KFrank: <p>Hi Roxor!</p>
<aside class="quote no-group" data-username="roxor" data-post="1" data-topic="109206" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/r/4da419/48.png" class="avatar"> roxor:</div>
<blockquote>
<p>I am looking for a possibility to get ALL indices for the maximum for each row in a 2-D tensor.<br>
Let us define the following tensor:</p>
<pre><code class="lang-auto">a = torch.tensor([[5,8,9,6,9], [5,4,3,2,1]])
</code></pre>
</blockquote>
</aside>
<p>Try this:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; torch.__version__
'1.7.1'
&gt;&gt;&gt; a = torch.tensor([[5,8,9,6,9], [5,4,3,2,1]])
&gt;&gt;&gt; torch.nonzero ((a == a.max (dim = 1, keepdim = True)[0]))
tensor([[0, 2],
        [0, 4],
        [1, 0]])
</code></pre>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">torch.tensor([[2,4],[0]])
</code></pre>
</blockquote>
</aside>
<p>Note, that pytorch does not support “ragged tensors” (tensors whose rows<br>
are not equal to one another in length), so you can’t get your result in a<br>
format like this.</p>
<p>Best.</p>
<p>K. Frank</p>
PyTorch Mobile speed_benchmark_torch crashes on mobile optimized model

windmaple: <p>I’m trying to benchmark MobileNet v2 but hit a crash. This is what I did:<br>
Compiled the benchmark binary off 4ed7f36ed using:</p>
<blockquote>
<p>BUILD_PYTORCH_MOBILE=1 ANDROID_ABI=arm64-v8a ./scripts/build_android.sh -DBUILD_BINARY=ON</p>
</blockquote>
<p>Export MobileNet v2 using:</p>
<blockquote>
<p>import torch<br>
from torch.utils.mobile_optimizer import optimize_for_mobile<br>
model = torch.hub.load(‘pytorch/vision:v0.6.0’, ‘mobilenet_v2’, pretrained=True)<br>
model.eval()</p>
<p>scriptedm = torch.jit.script(model)<br>
torch.jit.save(scriptedm, “mobilenet_v2_scripted.pt”)<br>
optedm = optimize_for_mobile(scriptedm)<br>
torch.jit.save(optedm, “mobilenet_v2_opted.pt”)</p>
</blockquote>
<blockquote>
<p>(python3.7) user1$ adb shell “/data/local/tmp/speed_benchmark_torch --model=/data/local/tmp/mobilenet_v2_opted.pt” --input_dims=“1,3,224,224” --input_type=“float”<br>
Starting benchmark.<br>
Running warmup runs.<br>
Main runs.<br>
Segmentation fault</p>
</blockquote>
<p>mobilenet_v2_scripted.pt however works fine. Device is Pixel 3a. Not sure what’s wrong.</p>

valerianG: <p>Same problem as long as I use float inputs. The benchmark on quantized networks seem to be more robust.</p>
Compute model gradients correctly?

iamgroot42: <p>I want to compute the gradients of a model with respect to certain inputs. This is how I think it should be, but PyTorch throws an error unless I include retain_graph=True, which I feel shouldn’t be the case? This is how I do it right now:</p>
<pre><code class="lang-auto">for x, y in loader:
  logits = model(x)
  loss = ce_loss(preds, y)

  grads = autograd.grad(loss, model.parameters()) # Throws an error unless I include retain_graph=True
</code></pre>
<p>What’s the right way of doing this (without using loss.backward(), since I will be performing similar operations above for input-optimization as well, so I really need to know how to do it using autograd)?</p>
Backpropagation failed because of 'UnsqueezeBackward0'

frwidi: <p>Dear all,<br>
currently I try to train a model in which I need to write a modified loss function. After some running time I the training is interrupted by an error in the backward propagation, because the Function ‘UnsqueezeBackward0’ returns nan values in its 0th output.</p>
<p>I tried to figure out what could generate this kind of error, as so far as I know, the function unsqueeze is just an formating function, thus it won’t affect the backward propagation. I am stuck in this debugging, as I cannot figure out what can caused this kind of errors.<br>
Any hint would be appreciated <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>Below I attached the errors that I got:</p>
<pre><code class="lang-auto">
  File ".\testnet_gpu_withsupervised_restructured.py", line 363, in &lt;module&gt;
    check_path = checkpoint_path

  File ".\testnet_gpu_withsupervised_restructured.py", line 131, in training_loop
    loss_all.backward()

  File ".\site-packages\torch\tensor.py", line 195, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)

  File ".\site-packages\torch\autograd\__init__.py", line 99, in backward
    allow_unreachable=True)  # allow_unreachable flag

RuntimeError: Function 'UnsqueezeBackward0' returned nan values in its 0th output.
</code></pre>
<p>Best Regard,</p>
Data Transformation a part of model graph

Ankita_Makker: <p>Can we make transformations a part of model graph in pytorch so that data preprocessing is not done at the client side too, just like we can with tensorflow (tfx)?</p>

Ankita_Makker: <p>To explain it in more detail:<br>
When we define transformations in pytorch which is basically a pre-process step.<br>
For example, lets say i have a dataset of images,<br>
colored images 512*768but i trained the model on b&amp;w, center cropped, resized images, so this is the transformation i chose.<br>
now at real time predictions, when you will write the client side code,<br>
you may need to define this transformation there too,<br>
since your model’s input layer will expect a specific shape wrt channel, size etc. so in tensorflow it is possible to define whatever transformation we did, as a part of the model graph</p>
<p>so at real time we don’t have to make any manipulations<br>
just send the raw data.<br>
while in pytorch i couldn’t find a way to offload this dependency.</p>

Ali_Haider: <p>If I am rightly understanding your question than you can use non-sequential modeling to do that.</p>
Value_loss: nan

shuaibiqbal: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/4/e4881f998b0b010ebc10708fee7b737fc51957c2.png" data-download-href="https://discuss.pytorch.org/uploads/default/e4881f998b0b010ebc10708fee7b737fc51957c2" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/4/e4881f998b0b010ebc10708fee7b737fc51957c2.png" alt="image" data-base62-sha1="wBGvowBPoDsJKIe8ZivhjpniKFs" width="690" height="168" data-dominant-color="F5F5F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">865×211 5.49 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
my model is giving value_loss to nan… how can i solve this problem is their anyone to help</p>

shuaibiqbal: <p>here my data<br>
splitting training an testing in this way<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/2/b227f93ee6a5a2859581c8f8a0787bef3b6b91c6.png" data-download-href="https://discuss.pytorch.org/uploads/default/b227f93ee6a5a2859581c8f8a0787bef3b6b91c6" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/2/b227f93ee6a5a2859581c8f8a0787bef3b6b91c6.png" alt="image" data-base62-sha1="pq2Dey5mny8fbCl3sctB75Px4AS" width="597" height="500" data-dominant-color="EFF1F1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">772×646 32.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>Could you post your model and training code so that we can have a look, please?</p>
Pass Torchscript from Python to C++ without serialization

edwinem: <p>I was wondering if there is any way to pass a Torchscript module from python to C++, without having to go through serialization.</p>
<p>I have my own custom <a href="https://github.com/Edwinem/ceres_python_bindings" rel="nofollow noopener">C++ library</a>(not just an extension) wrapped with Pybind11. The goal is to pass a torchscript module without too much overhead, so ideally I want to avoid passing through the filesystem, or strings of Python code.</p>
<p>I have currently tried creating a C++ function which accepts <em>torch::jit::script::Module</em>, but when I pass a <em>torch.ScriptModule</em> it fails as different type. My guess is because PyTorch doesn’t export the Pybind code for Module in torch/Extension.h, or I am doing something wrong with the linking.</p>
<p>Ideally it should look something like this.</p>
<pre><code class="lang-cpp">void ExampleFunction(torch::jit::script::Module&amp; module){
 // Do something
};

//Pybind11 code
m.def("CppExampleFunction",&amp;ExampleFunction);
</code></pre>
<pre><code class="lang-python">class MyModule(nn.Module):
    def __init__(self):
        super(MyModule, self).__init__()

    def forward(self, input):
        return 10 - input
my_script_module = torch.jit.script(MyModule())
CppExampleFunction(my_script_module)
</code></pre>

juliusbk: <p>I need the exact same thing. BUMP!</p>
<p>Did you find a solution?</p>
Is there a way I can call a function in the middle of a series of operations but ignore that function's gradient?

Sam_Lerman: <p>For example,</p>
<pre><code class="lang-auto">output_1 = model(inputs)
# ignore the grads of torch.exp()
middle_func_output = torch.exp(output_1)
loss = loss_func(middle_func_output)
loss.backward()
</code></pre>
<p>I would like to ignore the gradients from the middle_func_output and treat the backprop as if <code>torch.exp()</code> didn’t change the gradients in the backwards pass even though it’s still applied in the forwards pass.</p>
<p>I tried:</p>
<pre><code class="lang-auto">with torch.no_grad():
     middle_func_output = torch.exp(output_1)
</code></pre>
<p>But this returns the error: <code>RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn</code>.</p>
<p>My actual middle_func is not <code>torch.exp()</code>, but is multivariable.</p>

ptrblck: <p>You could use a similar approach as <a href="https://discuss.pytorch.org/t/relu-with-leaky-derivative/32818/2">this one</a> posted by <a class="mention" href="/u/tom">@tom</a>, where <code>x_backward</code> would correspond to the tensor, which should be used during the backprobagation and <code>x_forward</code> only in the forward pass.</p>
Removing layers from inception v3

DHYEY_PATEL: <p>i want my inception v3 till mixed 6e layer and i am using the output from mixed 6e layer as input to another network , but don’t know how to get exact that part of inception!</p>

ptrblck: <p>You could write a custom module by using the <code>inception_v3</code> model as the base class and override the <code>forward</code> method of your custom class to only call the desired modules.<br>
Alternatively, you could also use forward hooks to grab the intermediate activation (this would run the complete model nevertheless, so would come with a performance penalty).</p>
Model.eval error

M_emergency: <p>question:<br>
i installed a module with pip which works on a specific dataset.<br>
when I want to work with a different dataset, it gives the error :: :Expected more than 1 value per channel when training, got input size torch.Size([1, 15])</p>
<p>I understand that I can set the model to eval mode and run as the batch norm is creating the issue;<br>
how do I set the pytorch model to eval mode when the module downloaded gives the error:<br>
‘module’ object has no attribute ‘eval’</p>

M_emergency: <p>I downloaded the Contextualized Topic model package which works on a 20000 doc dataset.<br>
when I try this with a custom dataset, I get a 'Expected more than 1 value per channel when training, got input size torch.Size([1, 15]) error;</p>
<p>from contextualized_topic_models.models.ctm import CombinedTM<br>
ctm = CombinedTM(input_size=len(qt.vocab), bert_input_size=768, n_components=15, num_epochs=15)<br>
how do I set this to eval mode, so that batch norm doesn’t create an issue?</p>

ptrblck: <p>Most likely the <code>CombinedTM</code> wraps the actual model internally and is not a plain instance of <code>nn.Module</code>. I would recommend to check the source code of this model in the repository you’ve downloaded it from and search for the actual model, which would provide the <code>.eval()</code> method.</p>
Pytorch install import libcublas error

DanielWang: <p>For Pytorch install on conda with CUDA11, when import PyTorch, I got the following error.</p>
<blockquote>
<p>anaconda3/envs/pytorch/lib/python3.8/site-packages/torch/lib/…/…/…/…/libcublas.so.11: symbol free_gemm_select, version libcublasLt.so.11 not defined in file libcublasLt.so.11 with link time reference</p>
</blockquote>

ptrblck: <p>Could you share the install command and log if available?<br>
Also, could you create a new conda environment and install the binaries again in this clean env?</p>
Match pixels of an image with corresponding features after encoding

Li-Yun: <p>Assume that we have an input image, a tensor <strong>inputImg</strong> of shape B,C,W,H (in our case its B,3,256,128), and after encoding with pre-trained VGG, for a specific layer (let’s say for relu4_1), we got a feature tensor, <strong>featTens</strong> of shape ([B, 512, 16, 32]) . Is there a way to directly map each pixel of <strong>featTens</strong> with each corresponding patch or region of  <strong>inputImg</strong> ?</p>
<p>The main reason is that I want to find the corresponding image pixels for the features that seem to give big errors during training.</p>

ptrblck: <p>You could either try to calculate the receptive fields manually (might be cumbersome) or use a method from e.g. <a href="https://captum.ai/">Captum</a> which might have some methods to visualize (large) output activations in the input tensor.</p>
Weird memory Allocation issue. Pytorch tries to allocate 95GiB instead of 15GB

a_d: <p>Hello All,<br>
I have a model (let us call it <code>NN_Model</code>) that is made of two submodules let’s say <code>A</code> and <code>B</code>. The input first goes through A then through B and at last a couple of conv layers(2 to be precise).</p>
<pre><code class="lang-auto">output = A(input)
output = B(output)
output = conv(output)
return output
</code></pre>
<p>When I create an instance of model <code>A</code> and run the forward, it takes around 7.4GB on the GPU. Similarly, model B alone takes around 5GB on the GPU. The last convolutional layers do not take 700MB separately.</p>
<p>However when I try to run the forward pass of the ‘NN_Model’, pytorch tries to allocate 95GiB on the GPU with the following error message -<br>
<code>RuntimeError: CUDA out of memory. Tried to allocate 95.37 GiB (GPU 0; 7.80 GiB total capacity; 4.25 GiB already allocated; 2.59 GiB free; 4.31 GiB reserved in total by PyTorch)</code><br>
I am not able to understand the sudden surge in memory required. I expected a total of 15GB required, give or take a GB but 95 is a lot.<br>
Could anyone please tell me the possible reasons. The submodule <code>A</code> is a Resnet50 from torchvision and the submodule <code>B</code> contains  4 softmax’s as their learnable parameter and nothing else. The forward function mainly contains <code>bmm</code> calls which I suppose cannot add to the memory .</p>
<p>Could anyone please help me out me the possible reasons for the above aberration?</p>
<p>Edit - I noticed that 95GiB was trying to get allocated in the forward of the <code>B</code>(which works perfectly when run separately). what could be going wrong ?</p>
<p>TIA</p>

ptrblck: <p>Could you post the code of <code>NN_Model</code> so that we could try to reproduce this issue, please?</p>
How to make a self-made dataset for rnn

skerlet_flandorle: <p>I’m making my own dataset for rnn<br>
I don’t know where the internal state (time) of GRU starts<br>
The size is 1 batch = 100,256</p>
<p>In pytorch’s GRU, which do you start from right (deta [batch] [0]) or left (deta [batch] [100])?</p>

ptrblck: <p>RNNs expect an input in the shape <code>[seq_len, batch_size, features]</code> in the default setup.<br>
The temporal dimension is thus in <code>dim0</code> and the first time step would be at index 0.</p>
Unknown source of out of memory

sohaib_attaiki: <p>Hi,</p>
<p>I’m implementing a network, that in the forward loop does some computation with some intermediate variables, that are deleted after using <code>del</code>. However, during the training phase, even if I’m using a batch size of 1, I got an out of memory error, and I don’t know the cause of this, but I’m guessing it’s related to gradient, because in the validation phase, I don’t have any error.</p>
<p>My code looks like:</p>
<pre><code class="lang-python">C = my_subnetwork1(input_data)
# allocated memory 9 Gb | used memory 0.1 Gb
for i in range(5):
    D = some_arithmetic_computation(C, i)
    C = torch.softmax(D, 0)
    del D

return C
</code></pre>
<p>however, If I inspect the allocated memory and the used memory at the end of each step of the loop, the used memory keeps increasing if I’m in the training mode, it grows like this: 1 Gb -&gt; 1.97 -&gt; 3.02 -&gt; 4.2.</p>
<p>Can you point me to a direction to solve this?</p>
<p>Thank you in advance!</p>

ptrblck: <p>Depending on the used operations the intermediate tensors would be needed to calculate the gradients and you won’t be able to delete them unless you disable the gradient calculation e.g. via wrapping the code in a <code>with torch.no_grad()</code> block.</p>
DataLoader size limit in torch.load()?

ipostr08: <p>I’m attempting to load a 14 GB object using torch.load() under Windows 10 (64-bit), Python 3.8.5, 3090 RTX and PyTorch 1.7.1 and I’m getting an error:</p>
<pre><code class="lang-auto">Exception has occurred: TypeError
get_storage_from_record(): incompatible function arguments. The following argument types are supported:
    1. (self: torch._C.PyTorchFileReader, arg0: str, arg1: int, arg2: object) -&gt; at::Tensor

Invoked with: &lt;torch._C.PyTorchFileReader object at 0x00000248E74CC970&gt;, 'data/2588766380832', -785689216, torch.float32
  File "U:\endgame\datasetLoader.py", line 40, in loadOrCreateDatasets
    trainLoader = torch.load(savedFnameTrain)
</code></pre>
<p>The object was saved using torch.save(). On the other hand, loading a 3.5 GB object works fine. Is there a size limitation at play? Any recommended workaround?</p>
<p>I’ve attempted to load a Dataset (18 GB) and convert it into a DataLoader but this also fails:</p>
<pre><code class="lang-auto">Exception has occurred: RuntimeError
[enforce fail at ..\c10\core\CPUAllocator.cpp:48] ((ptrdiff_t)nbytes) &gt;= 0. alloc_cpu() seems to have been called with negative number: 18446744070566794752
  File "U:\endgame\datasetLoader.py", line 42, in loadOrCreateDatasets
    demoSetTrain = pickle.load(open(savedFnameTrainDS, "rb"))
</code></pre>
<p>A similar report from March 2020: <a href="https://discuss.pytorch.org/t/using-torch-save-and-torch-load-on-very-heavy-files/74338" class="inline-onebox">Using torch.save and torch.load on very heavy files</a></p>

InnovArul: <p>For large files, a typical workaround is to use HDF5 format.<br>
Check if it suits your needs.</p>
<aside class="onebox allowlistedgeneric">
  <header class="source">
      <img src="https://pypi.org/static/images/favicon.6a76275d.ico" class="site-icon" width="16" height="16">
      <a href="https://pypi.org/project/h5py/" target="_blank" rel="noopener">PyPI</a>
  </header>
  <article class="onebox-body">
    <img src="https://pypi.org/static/images/twitter.90915068.jpg" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://pypi.org/project/h5py/" target="_blank" rel="noopener">h5py</a></h3>

<p>Read and write HDF5 files from Python</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


ptrblck: <p>The issue might also be related to <a href="https://github.com/pytorch/pytorch/issues/12085">this one</a>.<br>
Could you try to suggestion by using pickle protocol 4?</p>
Validation and training loss per batch and epoch

over9k: <p>Hi,</p>
<p>I am currently keeping track of training and validation loss per epoch, which is pretty standard. However, what is the best way of going about keeping track of training and validation loss per batch/iteration?</p>
<p>For training loss, I could just keep a list of the loss after each training loop. But, validation loss is calculated after a whole epoch, so I’m not sure how to go about the validation loss per batch. The only thing I can think of is to run the whole validation step after each training batch and keeping track of those, but that seems overkill and a lot of computation.</p>
<p>For example, the training is like this:</p>
<pre><code class="lang-auto">for epoch in range(2):  # loop over the dataset multiple times

    running_loss = 0.0
    for i, data in enumerate(trainloader, 0):
        # get the inputs; data is a list of [inputs, labels]
        inputs, labels = data

        # zero the parameter gradients
        optimizer.zero_grad()

        # forward + backward + optimize
        outputs = net(inputs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

        # print statistics
        running_loss += loss.item()
</code></pre>
<p>And for validation loss:</p>
<pre><code class="lang-auto">with torch.no_grad():
    for data in testloader:
        images, labels = data
        outputs = net(images)
        _, predicted = torch.max(outputs.data, 1)
        total += labels.size(0)
        correct += (predicted == labels).sum().item()
        # validation loss
        batch_loss = error(outputs.float(), labels.long()).item()
        loss_test += batch_loss
    loss_test /= len(testloader)

</code></pre>
<p>The validation loss/test part is done per epoch. I’m looking for a way to get the validation loss per batch, which is my point above.</p>
<p>Any tips?</p>

ptrblck: <aside class="quote no-group" data-username="over9k" data-post="1" data-topic="108308">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/o/a87d85/48.png" class="avatar"> over9k:</div>
<blockquote>
<p>The only thing I can think of is to run the whole validation step after each training batch and keeping track of those, but that seems overkill and a lot of computation.</p>
</blockquote>
</aside>
<p>That’s right and is the reason, why the validation loss and metric is usually calculated once per epoch.<br>
The idea of calculating the validation loss is to get a signal of the model’s performance on “unseen” data. In the best case the validation loss should closely stick to the final test loss (note that you shouldn’t touch the test dataset until your training is done, as you would otherwise leak the test data information into the training). For this, it’s often not needed to closely track the validation loss after each batch/iteration.</p>
Making an Age detector, need some helps

Jia_Ming: <p>I am right now trying to create an age detector by referencing human faces. I decided to use pretrained model from resnet50. However, when I refer to other age detectors found from github, I found out they did import a pretrained model, but do not freeze the CNN layers.</p>
<p>I thought that since CNN layers are kind of feature extractors, it is reasonable to freeze those layers. If so, what are the reasons for not freezing those CNN layers??</p>
<p>I really need you guys’ help! Trying to learn everything from scratch.</p>

user_123454321: <p>Sometimes one needs to update the feature extractors too. That is the feature extractor for imagenet might not be optimal for age detection. If you go through fastai course, you will see that they first train the classifier by freezing the feature extractor in first stage. Then in the second stage they unfeeze the whole network and train all the layers.</p>
How to add an "epsilon" to the .log_prob method?

phan_phan: <p>Hi,</p>
<p>Doing predictions of 3d trajectories, I am using a LSTM to output the parameters of a distribution (3d gaussian). Then, computing the loss : <code>-dist.log_prob(ground_truth).mean()</code>, to maximize the likelihood of the ground truth values.</p>
<p>Instead of <code>log(prob)</code>, I would like the method to compute <strong><code>log(prob + espilon)</code></strong>. Is it possible ?</p>
<p>The issue is : <code>log_prob</code> worked on a <em>test</em> dataset, but on the real, noised dataset, the loss explodes after a while. I can see in some trajectories random points that are suddenly really far from the previous and next points. This likely causes some <code>prob</code> to be infinitesimal, which causes some <code>-log_prob()</code> to be really big, which causes the grads to explode.<br>
I think <code>log(prob + epsilon)</code> would solve my problem, but the method <code>.log_prob()</code> does not propose this.</p>
<p>PS : Another solution would be to clip the gradient, but as discussed in this <a href="https://github.com/pytorch/pytorch/issues/1407" rel="nofollow noopener">github issue</a>, it is not (yet) doable when using <code>nn.LSTM</code>… only when using <code>nn.LSTMCell</code>.<br>
=&gt; <code>log_prob(prob + eps)</code> would kind of “limit” the log to small values, and most importantly limit the grads to small values too.</p>

Dhawgupta: <p>I also need to do something similar, did you have any luck getting around this issue. I don’t want to apply gradient clipping as well and epsilon seems like a good way to keep gradients small.</p>
Using SciPy optimization methods in Pytorch

Pablo: <p>Hello everyone.</p>
<p>I’d like to ask if is there any way to use SciPy minimization methods as Pytorch optimizers?<br>
I would like to use Levenberg-Marquardt minimization from SciPy since my NN is not that big and this method shows really good results.</p>
<p>Thank you very much <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

FA_mn: <p>Hello</p>
<p>Could you please inform me whether you can implement Levenberg-Marquardt minimization from SciPy as Pytorch optimizers?<br>
I appreciate you sharing your knowledge on this issue.</p>
<p>Thanks a lot.</p>
Convert batch of index to mask

zyc: <p>I have a batch of indexes with a dimension of [k, 4], and I would like to create a mask ([k, h, w]).</p>
<pre><code class="lang-python">index = [[4, 4, 8, 8], [2, 2, 6, 6]]

mask = [[[0, 0, 0, 0, 0, 0, 0, 0],
         [0, 0, 0, 0, 0, 0, 0, 0],
         [0, 0, 0, 0, 0, 0, 0, 0],
         [0, 0, 0, 0, 0, 0, 0, 0],
         [0, 0, 0, 0, 1, 1, 1, 1],
         [0, 0, 0, 0, 1, 1, 1, 1],
         [0, 0, 0, 0, 1, 1, 1, 1],
         [0, 0, 0, 0, 1, 1, 1, 1],]
         [0, 0, 0, 0, 0, 0, 0, 0],
         [0, 0, 0, 0, 0, 0, 0, 0],
         [0, 0, 1, 1, 1, 1, 0, 0],
         [0, 0, 1, 1, 1, 1, 0, 0],
         [0, 0, 1, 1, 1, 1, 0, 0],
         [0, 0, 1, 1, 1, 1, 0, 0],
         [0, 0, 0, 0, 0, 0, 0, 0],
         [0, 0, 0, 0, 0, 0, 0, 0],]]
</code></pre>
<p>A <code>for</code> loop solution could be:</p>
<pre><code class="lang-python">mask = torch.zeros(index.shape[0], 128, 128)
for m, i in zip(mask, index):
    m[i[0]:i[2], i[1]:i[3]] = 1
</code></pre>

Eta_C: <p>Try</p>
<pre><code class="lang-python">k, h, w = 2, 8, 8
index = [[4, 4, 8, 8], [2, 2, 6, 6]]
hh, ww = torch.meshgrid(torch.arange(h), torch.arange(w))
hh, ww = hh.repeat(k, 1, 1), ww.repeat(k, 1, 1)
index = torch.tensor(index).long()
hh = torch.logical_and(index[:, 0, None, None] &lt;= hh, hh &lt; index[:, 2, None, None])
ww = torch.logical_and(index[:, 1, None, None] &lt;= ww, ww &lt; index[:, 3, None, None])
mask = torch.logical_and(hh, ww)
print(mask)
</code></pre>
<p>PS: i am not sure it is faster than for loop, test it by your self.</p>
Is there a way to avoid people who look inside of my model(Python,JIT in Java, C++, etcs, ONNX)?

JonathanSum: <p>Is there a way to avoid people who look inside of my model? (Python,JIT in Java, C++, etcs, ONNX)<br>
I am planning to deploy a few models for Python, Jit, or more, but I do not want people to use them and say it is their work or they designed it.<br>
Is there a way to avoid people looking into the inside part of the model? I will do things similar in super re-revolution, Gans, or NLP.</p>
Steps to be able to use my saved model on images in a flask server for classification

AnswersPls: <p>Hello, I am using a resent18 pretrained model for an image classification task.</p>
<p>I have saved this model using the model.state_dict() save method. I have saved it as a .pth file and i am now ready to deploy it and use it for predicting images that i send to it.</p>
<p>I am unsure how to go about this, in my training and testing all of the data came from folder in my windows machine that was local to the file so using datasets and dataloaders like this was fine</p>
<p>dataloaders[‘train’] = torch.utils.data.DataLoader(image_datasets[‘train’],<br>
batch_size=128,<br>
shuffle=True,<br>
num_workers=4)</p>
<p>How should i be preparing my data for classification when it is being used to actually predict things? I know that it must be a certain size etc for resnet but I am struggling to see how i would do this for images that come from post requests for example</p>

AnswersPls: <p>I should make clear that I am able to load it etc it is just when it is being called that there is a problem. Thanks in advance</p>
Instally Pytorch with Caffe2

leo-paz: <p>Hi Pytorch Community,</p>
<p>What I’m looking to do is probably a little unconventional but it’s for my fourth year project and I know it’s possible but just want to know the best way of going about it.</p>
<p>Basically, I have exported a pytorch model in caffe2 format that I now want to run inferences within an AWS Lambda function. I came across some articles online that said it is possible to install pytorch with caffe2 inside it that will work with python 3.x.</p>
<p>I see that the installer tool tells me to install “pip install torch==1.7.1+cpu torchvision==0.8.2+cpu torchaudio===0.7.2 -f <a href="https://download.pytorch.org/whl/torch_stable.html" rel="noopener nofollow ugc">https://download.pytorch.org/whl/torch_stable.html</a>” but does this come with caffe2? Once I ran this and then tried to install caffe2 I was getting errors correlating with pytorch not being installed properly but install pytorch through pip worked smoothly</p>