akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Torch.nn.functional API and Declarations.yaml

username: adampauls
cooked: <p>I am using Declarations.yaml to generate bindings for the torch API. One thing I’m a little confused by is that several operations seem to be specified in Declarations.yaml without default, but present in torch.nn.functional with defaults. For example, <a href="https://pytorch.org/docs/stable/generated/torch.nn.functional.poisson_nll_loss.html" rel="noopener nofollow ugc">poisson_nll_loss</a> has defaults here, but in Declarations.yaml, the schema string is <code>aten::poisson_nll_loss(Tensor input, Tensor target, bool log_input, bool full, float eps, int reduction) -&gt; Tensor</code>.</p>
<p>Is there a reason for this? Is there some subtle difference between the call in torch.nn.functional and the underlying op?</p>

username: tom
cooked: <p>The torch.nn.functional is a Python module that has no “automatic” equivalent in C++.<br>
That said, I don’t think there is a systematic reason native_functions.yaml (the origin for these) doesn’t declare default values but rather historical reasons, especially for not-as-often used functions that originate from torch7 (e.g. other loss functions do have defaults).<br>
If you are adventurous, you could try to provide a PR providing the defaults matching the Python API and see what people say.</p>
<p>Best regards</p>
<p>Thomas</p>

username: adampauls
cooked: <p>Thanks for the quick response!</p>

username: adampauls
cooked: <p>I put up <a href="https://github.com/pytorch/pytorch/pull/70156" rel="noopener nofollow ugc">a PR</a>.</p>
Custom loss function even when just handing down MSELoss: expected float, got double. how is that possible

username: Liquidmasl
cooked: <p>I am currently trying to get another loss function to work, but I always get the same error:</p>
<p><strong>Found dtype Double but expected Float</strong></p>
<p>But how is that possible?</p>
<pre><code class="lang-auto">class MSLELoss(nn.Module):
    def __init__(self):
          super().__init__()
          self.mse = nn.MSELoss()
        
      def forward(self, pred, actual):
          return self.mse(pred,actual)
</code></pre>
<p><code>criterion = nn.MSELoss()</code> works fine while<br>
<code>criterion = MSLELoss()</code> does not…</p>
<p>What am I missing?</p>
<p>Thanks in advance!</p>

username: ptrblck
cooked: <p>I cannot reproduce the issue and both approaches work as expected:</p>
<pre><code class="lang-python">criterion = nn.MSELoss()

x = torch.randn(10, 10)
y = torch.randn(10, 10).double()

loss = criterion(x, y)

class MSLELoss(nn.Module):
    def __init__(self):
        super().__init__()
        self.mse = nn.MSELoss()
        
    def forward(self, pred, actual):
        return self.mse(pred,actual)
      
my_criterion = MSLELoss()
loss = my_criterion(x, y)
</code></pre>

username: Liquidmasl
cooked: <p>oh, that is strange.<br>
I will investigate, thanks for checking! I will report results</p>

username: Liquidmasl
cooked: <p>Yeah that was totally my bad.<br>
I changed MSELoss to L1Loss at the same time as I removed a .float() somewhere.<br>
L! does not seam to care, MSE does.<br>
Also at the same time I tried to use MSLELoss…<br>
Yeah I should go to sleep at 5am.</p>
<p>thanks for showing me the obvious!</p>
<p>Guys: go to sleep and try again the next day before making forum posts <img src="https://discuss.pytorch.org/images/emoji/apple/stuck_out_tongue.png?v=10" title=":stuck_out_tongue:" class="emoji" alt=":stuck_out_tongue:"></p>
The computational complexity of torch.cat

username: DRISS_ELALAOUI
cooked: <p>how can I calculate the complexity O of torch.cat and torch.stack ?<br>
and thank you</p>

username: tom
cooked: <p>They allocate a target blob of the required size and then copy the values, so it is proportional to the total size. In degenerate cases (0-sized tensors) you would also have things that perform with the number of operands, for non-degenerate cases this is bounded by the total size.<br>
Strictly speaking, the number of dimensions also plays a role, but in terms of asymptotic analysis you would soon discover that there is a hard upper bound for the number of dimensions you can use with PyTorch anyway.</p>
<p>The most common pitfall to avoid is to grow tensors by using <code>growing_list = cat(growing_list, new_item)</code> in a loop because that will give quadratic complexity (in the iteration size from having effort proportional to <em>i</em> in the <em>i</em>-th loop), which becomes noticeable quite fast.</p>
<p>Best regards</p>
<p>Thomas</p>

username: DRISS_ELALAOUI
cooked: <p>For more clarity, I want to calculate the time complexity O of torch.cat and torch.stack through those examples<br>
Example 1 concatenation by torch.cat:</p>
<pre><code class="lang-auto">import torch 
T1 = torch.Tensor([1,2,3,4]) 
T2 = torch.Tensor([0,3,4,1]) 
T3 = torch.Tensor([4,3,2,5]) 
T = torch.cat((T1,T2,T3)) 
</code></pre>
<p>Example 2 of torch.stack:</p>
<pre><code class="lang-auto">import torch 
T1 = torch.Tensor([1,2,3,4]) 
T2 = torch.Tensor([0,3,4,1]) 
T3 = torch.Tensor([4,3,2,5])  
T = torch.stack((T1,T2,T3)) 
</code></pre>

username: tom
cooked: <p>I’m not sure what you are asking here.<br>
If you have fixed tensors, you’ll have a fixed amount of time this will take. You could measure it, e.g. using <a href="https://pytorch.org/tutorials/recipes/recipes/profiler_recipe.html" rel="noopener nofollow ugc">profiling</a>.<br>
When you have many / large tensors, it would be roughly scalingas above.</p>
<p>Best regards</p>
<p>Thomas</p>

username: DRISS_ELALAOUI
cooked: <p>No, I don’t have fixed tensors. But at each iteration, I have a number of tensors which I concatenate either by torch.cat, or by torch.stack. for your above explanation, I can’t understand it. Is the time complexity worth O (n) or O (n ^ 2)?</p>

username: tom
cooked: <p>Each operation is proportional to the size.<br>
If you have a loop that concatenates fixed-size tensors in each iteration, then you have “iters * fixed” which is linear in the iters.<br>
If you have a loop that grows a tensor linearly by repeated concatenation, you have something like fixed + 2 * fixed + 3 * fixed + … + iters * fixed = fixed * iters * (iters + 1) / 2.</p>

username: DRISS_ELALAOUI
cooked: <p>Thank you for these useful explanations</p>
How to obtain element wise loss in a batch?

username: karthikvjit
cooked: <p>Hello all, I have a requirement where I need to get an element wise loss in a batch of data and my model returns loss in training mode. What is the best way to obtain this?</p>
<p>I am doing the following way.<br>
loss0 = model(batch[0])<br>
loss1 = model(batch[1])<br>
loss2 = model(batch[2])<br>
loss3 = model(batch[3])</p>
<p>loss = w0<em>loss0 + w1</em>loss1 + w2<em>loss2 + w3</em>loss3</p>
<p>loss.backward()</p>
<p>my code executes perfectly alright without errors. Am I missing anything here or I am doing right in getting the weighted loss function with different weight to the loss corresponding to the each element of batch? Please suggest.</p>

username: mMagmer
cooked: <p>The way you were implementing it works but may have effect on performance.<br>
You can use <code>reduction = 'none'</code> in most of the <a href="https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html" rel="noopener nofollow ugc">loss</a> functions to get what you want.</p>

username: karthikvjit
cooked: <p>Thanks <a class="mention" href="/u/mmagmer">@mMagmer</a> for your reply. What aspects of performance will be impacted? Do you mean the GPU power is not utilised completely by the way I am implementing?</p>

username: mMagmer
cooked: <p>Yes, it has an effect on GPU utilization.<br>
But, some layers, like batch normalization, require large batch size for better performance in i.e. model accuracy.</p>

username: karthikvjit
cooked: <p>Thanks very much <a class="mention" href="/u/mmagmer">@mMagmer</a> for the clarification.</p>
Single class classification(image)

username: Sherzod_Bek
cooked: <p>Hi,<br>
I want to implement single class classification(output should be just single probability number). is there any example? or guide ?</p>
<p>I did multi-class classification, but I couldn’t modify it to single class… (if I use single class, training always shows loss: 0)</p>

username: ptrblck
cooked: <p>A single-class classification wouldn’t make much sense, since your model wouldn’t learn anything and will just output a high probability for the single class.<br>
A “perfect” classifier would thus be:</p>
<pre><code class="lang-python">def predict():
    return 0
</code></pre>
<p>as no other class indices are valid.</p>
<p>Assuming you are dealing with a binary classification, you could treat it as a 2-class multi-class classification (i.e. with <code>nn.CrossEntropyLoss</code> as the criterion) and use 2 output values or alternatively you could use a single output neuron and use <code>nn.BCEWithLogitsLoss</code>. The lower the output value of this neuron the more likely class0, the higher the more likely class1 was detected.</p>
Register_hook inside loop

username: tuero
cooked: <p>I’m working with a model which unrolls over a few steps, and I would like to scale the gradient at each of these steps. If I want to scale each unroll step by the same amount (say by 1/2), then this is straight forward:</p>
<pre><code class="lang-auto">...
float gradient_scale = 0.5;
for (int step = 0; step &lt; N; ++step) {
    loss_output = LossFnc(...);
    loss_output.register_hook([&amp;](torch::Tensor grad) {
        return grad * gradient_scale;
    });
}
</code></pre>
<p>Suppose now we want to scale each step independently. Let <code>gradient_scale</code> be a [batch_size, N] tensor where N is the number of unroll steps. Thus, for the <code>i</code>th unroll step, we would want to scale the gradient by <code>gradient_scale[:,i]</code>. How could one achieve this? I tried the following:</p>
<pre><code class="lang-auto">...
torch::Tensor gradient_scale;  // [batch_size, N]
for (int step = 0; step &lt; N; ++step) {
    loss_output = LossFnc(...);
    loss_output.register_hook([&amp;](torch::Tensor grad) {
        return grad * gradient_scale.index({Slice(), step});
    });
}
</code></pre>
<p>The issue with this is that when these hooks are called, the value of <code>step</code> is now <code>N</code>, which every hook will reference. This thus throws an index error, since accessing the <code>N</code>th column of <code>gradient_scale</code> is now out of bounds.</p>
<p>Any help is appreciated!</p>
What is the meaning of grad_output?

username: haoran-hash
cooked: <p>The code snippet is below.</p>
<pre><code class="lang-auto">import torch
from torch.autograd.function import Function

class MyCalc(Function):
    @staticmethod
    def forward(ctx, x):
        res = x * x + 2 * x
        ctx.res = res
        
        return res

    @staticmethod
    def backward(ctx, grad_output):
        print(grad_output)

        return grad_output

x = torch.tensor([2.], requires_grad=True)
res = MyCalc.apply(x)
print(res)

res.backward()
print(x.grad)
</code></pre>
<p><code>print(grad_output)=1</code>, I want to know what it means.</p>

username: mMagmer
cooked: <p>Suppose we have two sequential layers.<br>
Then grad_output is the gradient of the second layer output with respect to the input of itself (first layer output).<br>
in your case , you calling backward on a scaler (a tensor of one item) , and buy default grad is set to 1 for scaler output.</p>

username: haoran-hash
cooked: <p>Thank you.<br>
I get it.</p>
Is the calculation of optimizer.step() on CPU?

username: ddmm2020
cooked: <p>When I do profile on my train code, I found that there would be aten::copy_ and aten::to operations before  optimizer.step().So I wonder if  optimizer.step()  is calculated on cpus.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/8/583a4e52fe8e7b0aa51b43c85f6cacef25c7b19e.png" data-download-href="https://discuss.pytorch.org/uploads/default/583a4e52fe8e7b0aa51b43c85f6cacef25c7b19e" title="pfofilr"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/8/583a4e52fe8e7b0aa51b43c85f6cacef25c7b19e_2_689x223.png" alt="pfofilr" data-base62-sha1="cAuW75Tf4qtL9A8AjPEZZIVpPjM" width="689" height="223" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/8/583a4e52fe8e7b0aa51b43c85f6cacef25c7b19e_2_689x223.png, https://discuss.pytorch.org/uploads/default/optimized/3X/5/8/583a4e52fe8e7b0aa51b43c85f6cacef25c7b19e_2_1033x334.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/8/583a4e52fe8e7b0aa51b43c85f6cacef25c7b19e.png 2x" data-dominant-color="E4ECE8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">pfofilr</span><span class="informations">1206×391 14.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>If your model’s parameters are on the GPU the <code>step()</code> operation would also use the GPU to update the parameters. I don’t know which iteration you are profiling, but guess it might be the first one and you might see the initialization of some buffers <a href="https://github.com/pytorch/pytorch/blob/bd9983366bc1feaa3ffee5cdf62aeb268dbc60ce/torch/optim/adam.py#L101-L138">here</a>. The update can be seen <a href="https://github.com/pytorch/pytorch/blob/bd9983366bc1feaa3ffee5cdf62aeb268dbc60ce/torch/optim/_functional.py#L64-L110">here</a>.</p>
How to use multi-thread in pytorch dataloader with pybind release-gil functions?

username: FindDefinition
cooked: <p>Half of my project written in C++ with pybind nogil wrapper and unpickleable c++ class, so I can only use multi-thread loader instead of multi-process.<br>
Does pytorch implement multi-thread for loader? If not, why?</p>

username: ptrblck
cooked: <p>No, because of the Python GIL which would block the threads and thus wouldn’t yield any speedup.</p>

username: FindDefinition
cooked: <p>c++ function with pybind11::gil_scoped_release can avoid python gil problem and get ultra-fast multi-thread dataloader without pickle/IPC overhead if all bottleneck functions in dataloader code are written in c++. I think we should add a option to use thread workers.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="FindDefinition" data-post="3" data-topic="139432">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/finddefinition/48/41948_2.png" class="avatar"> FindDefinition:</div>
<blockquote>
<p>if all bottleneck functions in dataloader code are written in c++</p>
</blockquote>
</aside>
<p>Would this mean that the standard approach of using e.g. <code>ImageFolder</code> or a custom <code>Dataset</code> wouldn’t be suitable for this approach?<br>
If so, I think it might be a great extension (so feel free to create a feature request on GitHub for it and explain your interest in implementing it).</p>

username: ejguan
cooked: <p>General reason is multithreading is not fast in python. For the use case of DataLoader, most of users create Dataset using pure python code. Then, there won’t be any performance gain if we provide that.</p>
<p>But, we are working on new DataLoader-DataPipe, and hopefully we can provide multithreading DataLoader for you in the short future. And, you can use that to call a c++ function without python gil.</p>
When will pytorch be supported by python 3.10?

username: cijerezg
cooked: <p>I have python 3.10 and haven’t been able to install pytorch. Does anyone have a rough estimate when pytorch will be supported by python 3.10? I saw open issues on github on this, but they did not indicate any dates.</p>

username: cameronraysmith
cooked: <p>You can probably just subscribe to <a href="https://github.com/pytorch/pytorch/issues/66424" class="inline-onebox" rel="noopener nofollow ugc">Add Python-3.10 support · Issue #66424 · pytorch/pytorch · GitHub</a> for the latest information as indicated by your comment <a href="https://github.com/pytorch/pytorch/issues/66424#issuecomment-985148728" rel="noopener nofollow ugc">here</a>.</p>
The slicing operation of tensor affects the normal autograd

username: Linkang_Du
cooked: <p>Firstly, I put the slicing operation out of “with torch.no_grad()” scope. The loss of the classifier does not decrease and the prediction is wrong.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/b/ab961773c17ee3339b0ab91c7197f29b271db24b.png" data-download-href="https://discuss.pytorch.org/uploads/default/ab961773c17ee3339b0ab91c7197f29b271db24b" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/b/ab961773c17ee3339b0ab91c7197f29b271db24b_2_690x63.png" alt="image" data-base62-sha1="otVdt3Xh3KFFArp6UZuJpNHLZqP" width="690" height="63" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/b/ab961773c17ee3339b0ab91c7197f29b271db24b_2_690x63.png, https://discuss.pytorch.org/uploads/default/original/3X/a/b/ab961773c17ee3339b0ab91c7197f29b271db24b.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/a/b/ab961773c17ee3339b0ab91c7197f29b271db24b.png 2x" data-dominant-color="F0F1F6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1019×94 17 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Then, I move the slicing operation into “with torch.no_grad()”. And the classifier behaves normally.</p>
<p>Why does the classifier run abnormally in the first case?</p>

username: soulitzer
cooked: <p>It might matter if your data requires grad, but otherwise it shouldn’t. Do you have a small repro demonstrating the issue?</p>
Tensor and Sequence dataset

username: Oscar_Garcia
cooked: <p>Hi everyone!</p>
<p>I have a dataset which consists in pairs of embeddings and strings (sentences). The goal is to learn to map them, so after that I could be able to generate the string from a given embedding, and viceversa. How should I approach this? Any code reference?</p>
<p>Thank you all!</p>

username: sagsriv
cooked: <p>If there is a lookup table for a string to embedding generation and it is a bijective dict (every string maps to one embedding and vice versa), it is possible to just store them.</p>
<p>Short of that, you are trying to learn the inverse of a fn. Imagine the simplest case: the sentence embedding is a real vector and is generated by taking an average of token embeddings. So in the reverse task, given a sentence embedding S, you would like to find the tokens, whose embeddings, when averaged, would produce S. It seems really difficult unless the token embeddings have some specific properties.</p>
<p>This could be a multivariate regression problem, but I can’t imagine the usual constraints for that problem would be satisfied here.</p>

username: Oscar_Garcia
cooked: <aside class="quote no-group" data-username="sagsriv" data-post="2" data-topic="139620">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sagsriv/48/45018_2.png" class="avatar"> sagsriv:</div>
<blockquote>
<p>It seems really difficult unless the token embeddings have some specific properties.</p>
</blockquote>
</aside>
<p>Embeddings are embeddings of a word, and the sequence it’s that word’s definition in a real dictionary. Does it makes sense?</p>

username: Dace
cooked: <p>The best thing I can think of is using a sort of Sequence to Sequence Model</p>
<p>The training will be in two parts</p>
<ul>
<li>Training an encoder to convert strings into their respective encoding</li>
<li>Training a decoder to convert the trained encoders encoding into a string</li>
</ul>
MacOS conda install fails

username: reubens
cooked: <p>I have factory reset my iMac today. Installed latest Anaconda, created env, activated env, followed instructions <a href="https://pytorch.org/get-started/locally/" rel="noopener nofollow ugc">here</a>.</p>
<p>This is the error message I receive when I try to run any code with torch.</p>
<pre><code class="lang-auto">  File "torch.py", line 1, in &lt;module&gt;
    import torch
  File "/Users/reubenschmidt/code/python/nn/cxr_net/torch.py", line 3, in &lt;module&gt;
    x = torch.rand(5,3)
AttributeError: partially initialized module 'torch' has no attribute 'rand' (most likely due to a circular import)
</code></pre>
<p>Terminal <code>conda list | grep torch</code> result:</p>
<pre><code class="lang-auto">pytorch                   1.4.0                   py3.8_0    pytorch
torchvision               0.5.0                  py38_cpu    pytorch
</code></pre>
<p>This issue has been discussed previously here:</p><aside class="quote" data-post="9" data-topic="32391">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/minister_79/48/18315_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/attributeerror-module-torch-has-no-attribute-randn/32391/9">AttributeError: module 'torch' has no attribute 'randn'</a> 
  </div>
  <blockquote>
    Thank you. I’m surprised I did exactly that twice as you said (bash-3.2$ pip uninstall torch), but returns the following: 
Skipping torch as it is not installed. 
You are using pip version 10.0.1, however version 18.1 is available. 
You should consider upgrading via the ‘pip install --upgrade pip’ command. 
import torch doesn’t return error, but throws error when importing other packages.
  </blockquote>
</aside>

<p>Any thoughts would be much appreciated. Cheers.</p>

username: albanD
cooked: <p>Hi,</p>
<p>This most likely happens because you have a local file called <code>torch.py</code>. And so when you do <code>import torch</code>, it tries to import that local file instead of the torch library. You should not have a file (or folder) with the same name as a library you use locally to avoid issues <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: reubens
cooked: <p>Thank you so much for your help albanD. That was a really dumb mistake on my behalf!</p>
<p>I owe you one. Cheers.</p>

username: Echizen
cooked: <p>There are so many names I can have. Why I have to name it as torch.py?!!! Thank you for your questions. This helps me!</p>
Getting probability score

username: alpha2210
cooked: <p>I need to have probability score for each testing sample. What changes I should make in following code to get those probability score for each sample</p>

username: mMagmer
cooked: <pre><code class="lang-auto">eta = nn.functional.softmax(logits,-1)
</code></pre>
<p>this gives you probability of each class given sample <code>x</code> (P(i|x)).</p>

username: mMagmer
cooked: <p>this gives you probability of each class  for each sample.<br>
can you give me an example of what you want</p>

username: mMagmer
cooked: <aside class="quote no-group" data-username="mMagmer" data-post="2" data-topic="139616">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mmagmer/48/43283_2.png" class="avatar"> mMagmer:</div>
<blockquote>
<p><code>eta = nn.functional.softmax(logits,-1)</code></p>
</blockquote>
</aside>
<p>it should work.</p>
<p>replace</p>
<pre><code class="lang-auto">for imageName,realLabel,actualLabel,logitValue in zip(tested_file_name,real_label,actual_pred,logits):
</code></pre>
<p>with</p>
<pre><code class="lang-auto">for imageName,realLabel,actualLabel,logitValue in zip(tested_file_name,real_label,actual_pred,eta):
</code></pre>
<p>some where in your for loop print it.</p>

username: mMagmer
cooked: <p>something is off<br>
how many class do you have ?<br>
what is you batch size??</p>

username: mMagmer
cooked: <p>eta should be <code>NxC</code> and <code>eta.sum(-1) = 1</code></p>

username: alpha2210
cooked: <p>I have 9 classes and the total batch size is 16<br>
<a class="mention" href="/u/mmagmer">@mMagmer</a></p>

username: mMagmer
cooked: <p>can you give me eta shape.<br>
and what you’re printing in for loop</p>

username: alpha2210
cooked: <p><a class="mention" href="/u/mmagmer">@mMagmer</a><br>
eta.size is 9*9</p>
<p>Output of above print in for is above I have attached in previous commnets</p>

username: mMagmer
cooked: <p>in for loop you should print <code>logitValue</code> not <code>eta</code> itself.<br>
but still something is off,<br>
you should have eta shape as 16x9??<br>
also all of <code>tested_file_name,real_label,actual_pred,logits</code> should have 16 element or  have 16 row (if they’re matrices).</p>

username: alpha2210
cooked: <p>okay, i have printed logitvalue</p>

username: mMagmer
cooked: <p>it’s better now.<br>
i think you can report predicted label probability as confidence.<br>
in your example &gt;&gt;&gt;&gt; <code>0.1517</code></p>

username: alpha2210
cooked: <p>hey thanks <a class="mention" href="/u/mmagmer">@mMagmer</a>.</p>
Error with torch.nn.functional.normalize

username: codeflux
cooked: <p>Hi,<br>
when I try to normalize the input for my neural net, I receive the error:<br>
<code>IndexError: Dimension out of range (expected to be in range of [-1, 0], but got 1)</code></p>
<p>For the following code:</p>
<blockquote>
<p>input = torch.tensor(data=[1000, 4, 0.1, 0], dtype=torch.float32)</p>
<p>input_norm = torch.nn.functional.normalize(input)</p>
</blockquote>
<p>I cannot really make sense out of it, what is my mistake here?</p>
<p>Thanks in advance!</p>

username: mMagmer
cooked: <p><code>functional.normalize</code> expect input dimention greater than 1.<br>
maybe this fixes your issue.</p>
<pre><code class="lang-auto">input = torch.tensor(data=[[1000, 4, 0.1, 0]], dtype=torch.float32)

input_norm = torch.nn.functional.normalize(input)
</code></pre>

username: codeflux
cooked: <p>Thanks, that worked!<br>
The best way to “transform” it best would be to do this?</p>
<blockquote>
<p>input_norm.data = input_norm.data[0]</p>
</blockquote>
How to adjust SmoothL1Loss for particular need

username: Remote_Senser
cooked: <p>Hello,</p>
<p>I have been experimenting with L1 and MSE losses in Pytorch and noticing that L1Loss performs better because it is more robust to outliers. I discovered SmoothL1Loss which seems to be the best of both worlds. I understand that it behaves like MSELoss for error&lt;1 and like L1Loss otherwise. My dataset only contains values between 0 and 1. Therefore the largest possible error is 1. Does this mean the function behaves identical to MSELoss? Is it possible adjust the threshold in anyway to work better for my problem?</p>
<p>Regards,</p>

username: Apurv25
cooked: <p>Hi,</p>
<p>The boundary between “like MSELoss” and “L1Loss” is defined by the beta factor (to the best of my knowledge).</p>

username: mMagmer
cooked: <aside class="quote no-group" data-username="Remote_Senser" data-post="1" data-topic="69868">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/r/ea5d25/48.png" class="avatar"> Remote_Senser:</div>
<blockquote>
<p>My dataset only contains values between 0 and 1</p>
</blockquote>
</aside>
<p>first, you should check your model output, not your dataset.<br>
Second, l1 norm also computes sum of all difference in one dim. Although your data is 0-1 range, l1norm could be greater than one i.e. <code>L1([.5,.5,.5]) = 1.5</code><br>
also as <a class="mention" href="/u/apurv25">@Apurv25</a>  said, you can change beta .</p>
Tracing model issue

username: Rim
cooked: <p>Hello,</p>
<p>While trying to export a ppm_deepsup model, I got this error:</p>
<p>RuntimeError: Expected 4-dimensional input for 4-dimensional weight [512, 2048, 1, 1], but got 3-dimensional input of size [2048, 1, 1] instead.</p>
<p>The PPM_DEEPSUP is defined as follwos:</p>
<p>class PPMDeepsup(nn.Module):<br>
def <strong>init</strong>(self, num_class=150, fc_dim=4096,<br>
use_softmax=False, pool_scales=(1, 2, 3, 6)):<br>
super(PPMDeepsup, self).<strong>init</strong>()<br>
self.use_softmax = use_softmax<br>
self.ppm = []<br>
for scale in pool_scales:<br>
self.ppm.append(nn.Sequential(<br>
nn.AdaptiveAvgPool2d(scale),<br>
nn.Conv2d(fc_dim, 512, kernel_size=1, bias=False),<br>
BatchNorm2d(512),<br>
nn.ReLU(inplace=True)<br>
))<br>
self.ppm = nn.ModuleList(self.ppm)<br>
self.cbr_deepsup = conv3x3_bn_relu(fc_dim // 2, fc_dim // 4, 1)<br>
self.conv_last = nn.Sequential(<br>
nn.Conv2d(fc_dim+len(pool_scales)*512, 512,<br>
kernel_size=3, padding=1, bias=False),<br>
BatchNorm2d(512),<br>
nn.ReLU(inplace=True),<br>
nn.Dropout2d(0.1),<br>
nn.Conv2d(512, num_class, kernel_size=1)<br>
)<br>
self.conv_last_deepsup = nn.Conv2d(fc_dim // 4, num_class, 1, 1, 0)<br>
self.dropout_deepsup = nn.Dropout2d(0.1)<br>
def forward(self, conv_out, segSize=None):<br>
conv5 = conv_out[-1]<br>
input_size = conv5.size()<br>
ppm_out = [conv5]<br>
for pool_scale in self.ppm:<br>
ppm_out.append(nn.functional.interpolate(<br>
pool_scale(conv5),<br>
(input_size[2], input_size[3]),<br>
mode=‘bilinear’, align_corners=False))<br>
ppm_out = torch.cat(ppm_out, 1)<br>
x = self.conv_last(ppm_out)<br>
if self.use_softmax:  # is True during inference<br>
x = nn.functional.interpolate(<br>
x, size=segSize, mode=‘bilinear’, align_corners=False)<br>
x = nn.functional.softmax(x, dim=1)<br>
return x<br>
# deep sup<br>
conv4 = conv_out[-2]<br>
_ = self.cbr_deepsup(conv4)<br>
_ = self.dropout_deepsup(<em>)<br>
_ = self.conv_last_deepsup(</em>)<br>
x = nn.functional.log_softmax(x, dim=1)<br>
_ = nn.functional.log_softmax(_, dim=1)<br>
return (x, _)</p>
<p>and the export script is as follows :</p>
<p>model_decoder = ModelBuilder.build_decoder(“ppm_deepsup”, weights=checkpoint_decoder, use_softmax=True, num_class=150, fc_dim=2048)<br>
model_decoder.load_state_dict(model_decoder_checkpoints, strict=False)<br>
model_decoder.to(device)<br>
model_decoder.eval()<br>
example_enc = torch.zeros(1, 3, 450, 450).to(device)# torch.zeros(1, 3, 2048, 224,224).to(device)<br>
example_dec = torch.zeros(1, 3, 224,224).to(device)<br>
traced_encoder = torch.jit.trace(model_encoder, example_enc, strict=False)<br>
traced_decoder_script = torch.jit.trace(model_decoder, traced_encoder(example_enc), example_dec, strict=True)#, example</p>
<p>Any help please to solve this issue ?</p>
<p>Thanks</p>

username: googlebot
cooked: <p>your example tensors are 4d, it is inconsistent with 3d inputs to the model (no batch dim perhaps?)</p>

username: Rim
cooked: <p>Thank you for your reply. I pass torch.zeros(1, 3, …,… ) as Imentionned. Any other suggestion please ?</p>
Need Help,Could not install pytorch on windows

username: oneluxyou
cooked: <p>Could install pytorch,My pip is python 3.10<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/c/6cd4b1a6e1df0b19fc71d5fb1075e65eb8b862e1.png" data-download-href="https://discuss.pytorch.org/uploads/default/6cd4b1a6e1df0b19fc71d5fb1075e65eb8b862e1" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/c/6cd4b1a6e1df0b19fc71d5fb1075e65eb8b862e1.png" alt="image" data-base62-sha1="fwLgdcSBlfGTef4UuOeEpS00k4p" width="690" height="38" data-dominant-color="181818"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">847×47 1.22 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/4/7467926cd9a5181212da4a5af4e2f49af8dc23de.png" data-download-href="https://discuss.pytorch.org/uploads/default/7467926cd9a5181212da4a5af4e2f49af8dc23de" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/4/7467926cd9a5181212da4a5af4e2f49af8dc23de_2_690x58.png" alt="image" data-base62-sha1="gBLhT02JnGBza3jqJQxapc83Fp4" width="690" height="58" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/4/7467926cd9a5181212da4a5af4e2f49af8dc23de_2_690x58.png, https://discuss.pytorch.org/uploads/default/optimized/3X/7/4/7467926cd9a5181212da4a5af4e2f49af8dc23de_2_1035x87.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/7/4/7467926cd9a5181212da4a5af4e2f49af8dc23de.png 2x" data-dominant-color="333132"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1347×114 20.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>The pip wheels are built for Python 3.6 - 3.9 so you would need to downgrade to any of these versions.</p>

username: oneluxyou
cooked: <p>Many thanks for ur reply.It can be installed now  <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=10" title=":smiley:" class="emoji" alt=":smiley:"></p>
How can I mask on arrays instead of values?

username: fjwillemsen
cooked: <p>Hi all,</p>
<p>I am looking to mask a 2D tensor with a 1D tensor. The <code>masked_select</code> and <code>masked_scatter</code> etc. are intended to mask on values and return a 1D tensor while I need to retain the 2D format, so this will not work.<br>
I have used indexing to make it work:</p>
<pre><code class="lang-auto">test = torch.Tensor(parameter_space)
print(test.shape)
mask = torch.ones(len(test), dtype=torch.bool)
mask[1] = False
print(mask)
print(test[mask, :].shape)
</code></pre>
<p>This yields the following (correct) output, removing the second array:</p>
<pre><code class="lang-auto">torch.Size([7520, 8])
tensor([ True, False,  True,  ...,  True,  True,  True])
torch.Size([7519, 8])
</code></pre>
<p>However, I was wondering if this is the correct / most efficient way to do this?</p>

username: ptrblck
cooked: <p>Your approach certainly looks correct. Regarding efficiency: are you seeing a bottleneck in this operation? If so, there might be faster approaches (unsure yet), but I would generally not optimize code which won’t be visible in the entire model training later.</p>
Best way to normalize the input?

username: codeflux
cooked: <p>HI,<br>
not sure if normalize is the correct term here. The issue: The input for my neural network has different dimensions ranging from 1e-2 and 1e3.<br>
What is the best way normalizing it ( for the forward call as well)</p>
<p>I looked into <a href="https://pytorch.org/docs/stable/generated/torch.nn.BatchNorm1d.html" rel="noopener nofollow ugc">Batch Normalization</a>, which expects it to be several data sets.</p>
<p>I just have one “dimension” which I want to bring into the same range of values.</p>

username: ptrblck
cooked: <p>You could calculate the mean and std of the entire dataset for each feature and normalize each input by subtracting the mean and dividing by std. <code>torchvision.transforms.Normalize</code> applies the same normalization using the mean and std values for each channel.</p>
A small question about inplace error

username: fxsxjtu
cooked: <pre><code class="lang-auto">class Drop_attr3(nn.Module):
    def __init__(self):
        super(Drop_attr3, self).__init__()
    def forward(self,x,adj):
        row = x.size()[0]
        i = random.randint(0, row-1)
        x = make_zero_row(x,i)
        return x ,adj
def make_zero_row(x,row):
    torch.zero_(x[row])
    return x
</code></pre>
<p>here is my model.when I try to train it , the error is reported as “one of the variables needed for gradient computation has been modified by an inplace operation”, i want to know how to set one row of the tensor to ‘0’ without having inplace problems .<br>
I’d appreciate some help!</p>

username: mMagmer
cooked: <p>i think it’s working now. you only need to add <code> x = x + 0</code><br>
and I’ve changed some lines to add support for batched data.</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import random

class Drop_attr3(nn.Module):
    def __init__(self):
        super(Drop_attr3, self).__init__()
    def forward(self,x,adj):
        x = x + 0
        row = x.size()[1]
        i = random.randint(0, row-1)
        x[:,i].fill_(0)
        return x ,adj

# test the code
total = 256 * 56 * 4 
x = torch.arange(0, total).view(256,56 , 4).float()
x.requires_grad = True
m = Drop_attr3()
o= m(x,1)
o[0].mean().backward()
</code></pre>

username: fxsxjtu
cooked: <p>Thanks alot! It does work!</p>
How to change the tensor value in c++？

username: MartinZhang
cooked: <p>In Pytorch</p>
<pre><code class="lang-auto">import torch

temp_data = torch.tensor([1, 2, 3, 4])  

# temp_data = tensor([1, 2, 3, 4])

temp_data[0] = 5

# temp_data = tensor([5, 2, 3, 4])
</code></pre>
<p>So how can something similar be done in c++？？</p>
<p>Thanks a lot.</p>

username: ptrblck
cooked: <p><code>tensor.index_put_</code> would be the right approach as given in <a href="https://pytorch.org/cppdocs/notes/tensor_indexing.html#setter">these docs</a>.</p>

username: MartinZhang
cooked: <p>Thank you for your help <img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=10" title=":grinning:" class="emoji" alt=":grinning:"><a class="mention" href="/u/ptrblck">@ptrblck</a></p>
Why is the gradient multiplied by grad_output in this

username: The_duke
cooked: <p>There is something I don’t quite understand in this tutorial:<br>
<a href="https://pytorch.org/docs/stable/notes/extending.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/notes/extending.html</a></p>
<p>There is example that implements a simple linear function in pytorch:</p>
<h1>
<a name="inherit-from-function-1" class="anchor" href="#inherit-from-function-1"></a>Inherit from Function</h1>
<p>class LinearFunction(Function):</p>
<pre><code># Note that both forward and backward are @staticmethods
@staticmethod
# bias is an optional argument
def forward(ctx, input, weight, bias=None):
    ctx.save_for_backward(input, weight, bias)
    output = input.mm(weight.t())
    if bias is not None:
        output += bias.unsqueeze(0).expand_as(output)
    return output

# This function has only a single output, so it gets only one gradient
@staticmethod
def backward(ctx, grad_output):
    # This is a pattern that is very convenient - at the top of backward
    # unpack saved_tensors and initialize all gradients w.r.t. inputs to
    # None. Thanks to the fact that additional trailing Nones are
    # ignored, the return statement is simple even when the function has
    # optional inputs.
    input, weight, bias = ctx.saved_tensors
    grad_input = grad_weight = grad_bias = None

    # These needs_input_grad checks are optional and there only to
    # improve efficiency. If you want to make your code simpler, you can
    # skip them. Returning gradients for inputs that don't require it is
    # not an error.
    if ctx.needs_input_grad[0]:
        grad_input = grad_output.mm(weight)
    if ctx.needs_input_grad[1]:
        grad_weight = grad_output.t().mm(input)
    if bias is not None and ctx.needs_input_grad[2]:
        grad_bias = grad_output.sum(0)

    return grad_input, grad_weight, grad_bias
</code></pre>
<p>The question is probably simplistic, but if the backward is simply the “gradient formula”, I don’t understand why the gradient is multiplied by grad_output?</p>
<p>In my mind I have :</p>
<p>e.g. say x = input for clarity<br>
w= weight<br>
b = bias</p>
<p>f (x)= w* x + b</p>
<p>d f/ dx = w<br>
d f/ dw = x<br>
d f/ db = 1</p>
<p>What piece of the puzzle am I missing?</p>

username: ptrblck
cooked: <p>You are missing the use the gradient calculated by the loss (or the next layer). <a href="http://cs231n.stanford.edu/handouts/linear-backprop.pdf">Backpropagation for a Linear Layer</a> by Justin Johnson derives the formula for a simple case and was part of the CS231n lecture.</p>
How to convert CSV data into graph dataset?

username: vishalthengane
cooked: <p>Hello there,</p>
<p>I am working on an anomaly detection problem using a graph neural network. However, I am not sure which will be the best way to convert my CSV data into graph data. I have 115 different attributes for each timestamp. You can find the example dataset here; <a href="https://gist.github.com/vgthengane/0b20f1b018b7ea06799f3ae00e167a7c" class="inline-onebox" rel="noopener nofollow ugc">Example dataset for anomaly detection in IoT devices. This is only for PyTorch Discussion forum. · GitHub</a></p>
<p>I want to test each time stamp for an anomaly. Which will be the best way to convert CSV data into graph data?</p>
<p>I was thinking of considering each attribute as a node and connecting all of them with each other by the undirected edge (since I don’t know their connection to each other). Then I will have a single feature for each node and a lot of edges. To be specific <code>feature_shape == [115, 1]</code> and <code>edge_shape == [2, 6555]</code>. But I am not sure if this is a good way to model it, and what will be the best method to work on this on node embedding or graph embedding?</p>
<p>PS: I am using an autoencoder model to detect anomalies.</p>
<p>Thanks in advance. Hope I am clear about my question<br>
Vishal T</p>
Is there an automatic way to analyze the bottleneck of the model

username: tjk
cooked: <p>Hello!</p>
<p>This is a simple but general problem. Recently I was training a deep learning model using Pytorch, but it seems that no matter how I modified the layer number, channels or layer output size, the training loss stop improving after  hundreds of iterations. Changing the optimizing algorithm did not help much. I wonder is there an automatic way which can help me analyze which layer is the bottleneck that affects the decrease of training loss or model parameter updating?</p>
<p>Thanks for your help!</p>

username: Ofri_Masad
cooked: <p>Hi.<br>
The answer to your question is very complex. You are assuming that there is a layer which is the bottleneck of your model, but in fact many different factors can affect your training. here are just a few examples:</p>
<ul>
<li>Data - your dataset may hide a limit on the success of your model. if your dataset contains errors in labeling for example, or if it is very difficult your network will never be able to reach accuracy over the errors in the dataset.</li>
<li>Weight decay - if your optimizer is using weight decay, you may have set it too high. That means that you are limiting you model from learning the data better. Decreasing that wd could of course, cause overfitting, but that is another issue.</li>
<li>Learning rate - when you have reached the plato in loss, you may want do decrease you learning rate in order to allow your model to decent into lower minima. Consider using a learning rate scheduler and decrease the lr when you reach the plato.</li>
</ul>
<p>many other factors could affect the model training. if you are using a known model and dataset, look for the training recipe for this model, it will give you a good start.</p>

username: tjk
cooked: <p><a class="mention" href="/u/ofri_masad">@Ofri_Masad</a>  That’s very insightful. Is there a module in Pytorch that I can use it to monitor the weight decay in each layer?</p>
<p>PS: just out of curious, I might make the problem too complicated.</p>
Torch.jit.script

username: Ali-Arch808
cooked: <p>Hi, I’m trying to make a script for a model trained on cifar-10.</p>
<pre><code class="lang-auto">scripted_model = torch.jit.script(model, torch.rand(1,3,32,32))
</code></pre>
<p>I get the following error:<br>
TypeError:</p>
<pre><code class="lang-auto">'numpy.int64' object in attribute 'Conv2d.in_channels' is not a valid constant.
Valid constants are:
1. a nn.ModuleList
2. a value of type {bool, float, int, str, NoneType, torch.device, torch.layout, torch.dtype}
3. a list or tuple of (2)
</code></pre>
<p>I also tried using <code>torch.jit.ScriptModule</code> instead of <code>nn.Module</code> in the class definition of the model but it didn’t work.</p>

username: ptrblck
cooked: <p>The error message claims you are using <code>numpy.int64</code> to specify the <code>in_channels</code> argument in an <code>nn.Conv2d</code> layer as seen here:</p>
<pre><code class="lang-python"># works
conv = nn.Conv2d(3, 3, 3, 1, 1)
torch.jit.script(conv)

# fails
conv = nn.Conv2d(np.int64(3), 3, 3, 1, 1)
torch.jit.script(conv)
&gt; TypeError: 
 'numpy.int64' object in attribute 'Conv2d.in_channels' is not a valid constant.
  Valid constants are:
  1. a nn.ModuleList
  2. a value of type {bool, float, int, str, NoneType, torch.device, torch.layout, torch.dtype}
  3. a list or tuple of (2)
</code></pre>
<p>Pass the arguments as one of the mentioned supported types (e.g. as a Python <code>int</code>).</p>
PyTorch Mobile for Linux

username: onyx22574
cooked: <p><a href="https://pytorch.org/mobile/home/" rel="noopener nofollow ugc">PyTorch Mobile</a> mentions that is available for Linux platform.<br>
Is there available some workflow to build it from source for Linux? Further is there some Linux-based examples for pre-trained model deployment and if possible, some model optimization steps for effective infer?</p>
RuntimeError: vmap: Calling Tensor.as_strided is not supported

username: Adam_Conkey
cooked: <p>I am using <code>torch.autograd.functional.hessian</code> to compute the hessian in my optimization problem, but I keep getting the following error that I’m not finding many resources on:</p>
<pre><code class="lang-auto">RuntimeError: vmap: Calling Tensor.as_strided is not supported unless the batch dims being vmapped over are at the front of the tensor (in memory layout). When they are not at the front of the tensor this operation can be error prone so we actively discourage it; please file us a bug report and/or try to express the as_strided operation in terms of PyTorch view operations
</code></pre>
<p>My code is somewhat involved so it’s hard for me to post a minimal working example. I don’t even know what to be looking for though that could be causing this issue. Any pointers to help me out? I thought maybe it my usage of <code>view</code> in spots so I replaced those with <code>reshape</code> but it had no effect. I’m not sure where to go from here.</p>

username: tom
cooked: <p>as_strided does very wild things with the Tensor, so it’s not surprising that vmap doesn’t like it.<br>
If you are not using as_strided directly, you could see if you can get a traceback that says which function (or backward(?)) causes the error.</p>
<p>Best regards</p>
<p>Thomas</p>

username: Adam_Conkey
cooked: <p>Thanks for the suggestion. It turns out we were setting <code>vectorize=True</code> in the hessian function which sounds like it’s doing strange things with vmap and it’s indicated it’s an experimental feature. Once I changed it back to False all was good.</p>
Can't import ConstantLR scheduler

username: Davi_Magalhaes
cooked: <p>When I trie to use ConstantLR or some other schedulers I get the error: AttributeError: module ‘torch.optim.lr_scheduler’ has no attribute ‘ConstantLR’. I already tried to update pytorch version and nothing changed, I’m currently using version 1.6. Has anyone faced this issue? Any help is appreciated!</p>

username: googlebot
cooked: <p>it wasn’t present in 1.6, and that version is about a year old</p>

username: Davi_Magalhaes
cooked: <p>My mistake! I didn’t notice an error when I tried to update, sorry. Thanks!</p>