akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to do large-scale outer product efficiently

username: LilySnow
cooked: <p>I have two matrices A and B. A and B have the same number of rows (m), and different number of columns. A.shape = [m,M], B.shape = [m,N]. For each row of A and B, I want to do an outer product (torch.outer).</p>
<p>Is there a way not using loop, because m is very large and I need to do this operation many times.</p>
<p>Thanks.</p>

username: ZimoNitrome
cooked: <p><a href="https://pytorch.org/docs/stable/generated/torch.outer.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.outer.html</a></p>

username: LilySnow
cooked: <p>torch.outer only works on two vectors</p>

username: ZimoNitrome
cooked: <p>This seems to work if you want to just do it for 2D tensors.</p>
<aside class="quote" data-post="2" data-topic="4025">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/fmassa/48/58_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/batch-outer-product/4025/2">Batch Outer Product</a> 
  </div>
  <blockquote>
    No, there is no batch version for ger. But you can obtain the same results by appending 1 to the tensors. 
x = torch.rand(10, 3)
y = torch.rand(10, 3)

res = torch.bmm(x.unsqueeze(2), y.unsqueeze(1))
  </blockquote>
</aside>


username: AlphaBetaGamma96
cooked: <p>if you have a tensor <code>A</code> of shape <code>[m,M]</code> and another tensor <code>B</code> of shape <code>[m,N]</code> you can do an outer product via <code>torch.einsum</code>,</p>
<pre><code class="lang-auto">outer_product = torch.einsum("bi,bj-&gt;bij",A,B)
</code></pre>
Approximate heavy loss function with combination of 2 other

username: ZimoNitrome
cooked: <p>I have a complicated loss function <code>a(x, y)</code> that takes a long time to compute. But I have 2 similar loss functions <code>b(x, y)</code> and <code>c(x, y)</code> that each capture some aspects of <code>a(x, y)</code>.</p>
<p>How could I easiest find <code>f()</code> and <code>g()</code> such that <code>a(x, y) = f(b(x, y)) + g(c(x, y))</code>?</p>
<p>I can sample any data <code>x</code>, <code>y</code> to try these functions. Would I do some curve fitting or evolutionary coding to get somewhere? I can try guessing but that will only get me so far.</p>
<p>Here are some quick values from inserting semi-arbitrary values to <code>a</code>, <code>b</code>, and <code>c</code>:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/d/4d4c77126a8dc263b9061a35472db17cc6c488d7.png" alt="image" data-base62-sha1="b1OADNz6ouNcXfitqe7q9T16o51" width="449" height="310"></p>
Process finished with exit code 139 (interrupted by signal 11: SIGSEGV): convert torch::Tensor to std::vector!

username: dwwcqu
cooked: <p>When convert a torch::Tensor to std::vector, i got the problem <em>Process finished with exit code 139 (interrupted by signal 11: SIGSEGV)</em>.<br>
This is my convert code:<br>
<code>std::vector&lt;double&gt; conv_w(conv_weight.data_ptr&lt;double&gt;(),conv_weight.data_ptr&lt;double&gt;() + conv_weight.numel());</code></p>

username: dwwcqu
cooked: <p>The reason why arise this error is that my torch::Tensor is in c10::kCUDA, but i want copy its content to c10::CPU’s std::vector. So it cause <em>SIGSEGV</em> error.</p>
How does PyTorch pick cuda versions?

username: cjenkins5614
cooked: <p>I always use the standard download selection panel to match cuda versions (<a href="https://pytorch.org/" rel="noopener nofollow ugc">https://pytorch.org/</a>):</p>
<pre><code class="lang-auto">pip3 install torch==1.8.2+cu102 torchvision==0.9.2+cu102 torchaudio==0.8.2 -f https://download.pytorch.org/whl/lts/1.8/torch_lts.html
</code></pre>
<p>My two questions:</p>
<ol>
<li>Does PyTorch look at strictly <code>/usr/local/cuda</code>'s linkage and decide what directory to dig into? If I have <code>cuda</code> linked to <code>cuda-10.1</code>, but have <code>cuda-10.2</code> around, what would <code>torch==x.y.z+cu102</code> pick?</li>
</ol>
<pre><code class="lang-auto">--&gt; ls -hdlt /usr/local/cuda*
drwxr-xr-x 20 root root 4.0K Oct  9 13:00 /usr/local/cuda-10.2/
lrwxrwxrwx  1 root root   20 Jul 22 13:37 /usr/local/cuda -&gt; /usr/local/cuda-10.1/
drwxr-xr-x 25 root root 4.0K Jun 23 21:23 /usr/local/cuda-11.0/
drwxr-xr-x 19 root root 4.0K Apr 22  2021 /usr/local/cuda-10.1/
drwxr-xr-x 19 root root 4.0K Apr 22  2021 /usr/local/cuda-11.1/
</code></pre>
<ol start="2">
<li>I’ve found that even if there were only one <code>cuda-10.1</code> in <code>/usr/local</code>, and <code>torch==x.y.z+cu102</code> would still work, most of the time. When do I expect it not to work, and what’s the caveat of having mismatches, minor or major?</li>
</ol>

username: ptrblck
cooked: <p>Your local CUDA toolkit won’t be used unless you build PyTorch from source or a custom CUDA extension, since the pip wheels and conda binaries use their own CUDA runtime.<br>
You would thus only need to install a valid NVIDIA driver and can use the binaries directly.</p>

username: cjenkins5614
cooked: <p>That explains… So the choice really depends on whether the driver is compatible with the pytorch cuda version to be installed.</p>
Multiplication with itself along specific dimension

username: Kevin_Xue
cooked: <p>Hi,</p>
<p>i would like to ask how do i get the result of the elementwise multiplication of a matrix.</p>
<p>Assume we have a tensor <code>matrix</code> with the size of <code>[N, V]</code>. I would like to have the elementwise multiplication along dimension 1. The results should be <code>[V1*V2*V3*...*VN]</code>, which will have finally the shape <code>[V]</code>.</p>
<p>With best regards<br>
Kevin</p>

username: KFrank
cooked: <p>Hi Kevin!</p>
<aside class="quote no-group" data-username="Kevin_Xue" data-post="1" data-topic="137913" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kevin_xue/48/43239_2.png" class="avatar"> Kevin_Xue:</div>
<blockquote>
<p>The results should be <code>[V1*V2*V3*...*VN]</code>, which will have finally the shape <code>[V]</code>.</p>
</blockquote>
</aside>
<p>Does <code>matrix.prod (dim = 1)</code> do what you want?</p>
<p>Best.</p>
<p>K. Frank</p>
How to make parameters registered by register_parameter trained?

username: Ardeal
cooked: <p>Hi,</p>
<p>I registered my paramert in <code>nn.Module</code> sub-class by function <code>register_parameter</code>. The example code is followed. in my training code, I found that loss kept on changing, but acc kept unchanged.</p>
<p><strong>I register all parameters in top Module and use each parameter in each forward function of sub-module. I am considering whether the gradient of each parameter will be calculated according to the definition Module or the Module where the parameter is used?</strong></p>
<p>in addition, if I do only forward to test acc, the acc is good as expected. without any change, if I do training, the loss kept on small changes(such as from <code>loss = 0.6420461028972839 to  loss = 0.6429245240923738</code>), but acc kept exactly unchanged.</p>
<p>I tried to change lr, but almost no help.</p>
<p>My questions are:</p>
<ol>
<li>could those registered parameters trained/optimzed?</li>
<li>is there any special/additional settings I should change to make those parameters trained/optimized?</li>
</ol>
<pre><code class="lang-python">self.register_parameter(
    name=param_name.replace('.', '_'),
    param=Parameter(
        torch.from_numpy(param_data), requires_grad=not is_constant
    )
)

</code></pre>

username: ptrblck
cooked: <p><code>self.register_parameter</code> should work fine and you can verify that this parameter is trained by checking its gradients after the <code>backward</code> pass via <code>model.param_name.grad</code>. If this attribute doesn’t return <code>None</code> than gradients are calculated the the parameter. Assuming you’ve also passed this parameter to an optimizer via e.g. <code>optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)</code> then it’ll also be trained in the <code>optimizer.step()</code> call. You could additionally verify it by comparing its values before and after the <code>step()</code> call.</p>

username: Ardeal
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> ，<br>
Thank you！</p>
<p>I will try your ideas.</p>
<p>As <code>loss.backward()</code> will calculate gradient according to Chain Rule,  I am considering where the gradient is calculated.  that is what I am asking in this post:</p>
<p><strong>I register all parameters in top Module and use each parameter in each forward function of sub-module. I am considering whether the gradient of each parameter will be calculated according to the definition Module or the Module where the parameter is used?</strong></p>

username: ptrblck
cooked: <p>I’m not sure I understand the question completely, but during the forward pass the operations applied on the tensor will be tracked by Autograd and the gradient will be calculated using the computation graph. It doesn’t matter where the parameter is registered as its usage will be tracked.</p>
How do I make a multi-class classification using the BERT model?

username: parvector
cooked: <p>I have a set of Russian-language text and several classes for text in the form:</p>
<div class="md-table">
<table>
<thead>
<tr>
<th>Text</th>
<th>Class 1</th>
<th>Class 2</th>
<th>…</th>
<th>Class N</th>
</tr>
</thead>
<tbody>
<tr>
<td>text 1</td>
<td>0</td>
<td>1</td>
<td>…</td>
<td>0</td>
</tr>
<tr>
<td>text 2</td>
<td>1</td>
<td>0</td>
<td>…</td>
<td>1</td>
</tr>
<tr>
<td>text 3</td>
<td>0</td>
<td>1</td>
<td>…</td>
<td>1</td>
</tr>
</tbody>
</table>
</div><p>I make a classifier like in this article, only I change the number of output neurons:</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://towardsdatascience.com/multi-class-text-classification-with-deep-learning-using-bert-b59ca2f5c613">
  <header class="source">
      <img src="https://miro.medium.com/fit/c/256/256/1*ChFMdf--f5jbm-AYv6VdYA@2x.png" class="site-icon" width="256" height="256">

      <a href="https://towardsdatascience.com/multi-class-text-classification-with-deep-learning-using-bert-b59ca2f5c613" target="_blank" rel="noopener nofollow ugc" title="09:33PM - 02 August 2020">Medium – 2 Aug 20</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/459;"><img src="https://miro.medium.com/max/1200/1*vO75HqnqS4bz_hJqBmLEtA.jpeg" class="thumbnail" width="690" height="459"></div>

<h3><a href="https://towardsdatascience.com/multi-class-text-classification-with-deep-learning-using-bert-b59ca2f5c613" target="_blank" rel="noopener nofollow ugc">Multi Class Text Classification With Deep Learning Using BERT</a></h3>

  <p>Natural Language Processing, NLP, Hugging Face</p>

  <p>
    <span class="label1">Reading time: 4 min read</span>
  </p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>But BERT starts to work like a silly classifier, i.e. it always gives ones or zeros to some criterion.</p>
<p>I also tried using</p>
<pre><code>AutoModel.from_pretrained("DeepPavlov/rubert-base-cased-sentence")
</code></pre>
<p>with</p>
<pre><code>AutoTokenizer.from_pretrained("DeepPavlov/rubert-base-cased-sentence") 
</code></pre>
<p>tokenizer, but the result is the same.</p>
<p>What’s the problem? What am I doing wrong? Please help. I’m new to NLP, I’ve only done image processing before.</p>
How to increase the batch size but keep the gpu memory

username: airzostorm
cooked: <p>since the .backward will accumulate the gradient ,can I use loss.backward twice then optim.step?<br>
would it take double gpu memory or not ?<br>
would the effect is the same as using  batch size*2?</p>

username: ptrblck
cooked: <p>You could use a smaller batch size and accumulate the gradients. Then after a few iterations you could update the parameters using your optimizer.<br>
Have a look at the 2nd option in <a href="https://discuss.pytorch.org/t/why-do-we-need-to-set-the-gradients-manually-to-zero-in-pytorch/4903/20?u=ptrblck">this post</a>.</p>
<p>It would yield the same behavior regarding the gradients, but note that other layers like <code>BatchNorm</code> will behave differently, since they see smaller batches.<br>
If that’s problematic, e.g. when your batch size is really small, then you could change the momentum a bit or use other normalization layers, e.g. <code>GroupNorm</code> which should be more stable regarding smaller batch sizes.</p>

username: JamesDickens
cooked: <p>For <span class="hashtag">#2</span> from this post, do you need the flag retain_graph=True , i.e loss.backward(retain_graph=True)? It seems this is important?</p>

username: JamesDickens
cooked: <p>I think the question has been answered here:</p><aside class="onebox stackexchange" data-onebox-src="https://stackoverflow.com/questions/70119232/accumulating-gradients-for-a-larger-batch-size-with-pytorch/70119359#70119359">
  <header class="source">

      <a href="https://stackoverflow.com/questions/70119232/accumulating-gradients-for-a-larger-batch-size-with-pytorch/70119359#70119359" target="_blank" rel="noopener nofollow ugc">stackoverflow.com</a>
  </header>

  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/5987698/gooddeeds" target="_blank" rel="noopener nofollow ugc">
    <img alt="GoodDeeds" src="https://www.gravatar.com/avatar/1b26d6435dcb8897d1a57fa24b462da0?s=256&amp;d=identicon&amp;r=PG&amp;f=1" class="thumbnail onebox-avatar" width="256" height="256">
  </a>

<h4>
  <a href="https://stackoverflow.com/questions/70119232/accumulating-gradients-for-a-larger-batch-size-with-pytorch/70119359#70119359" target="_blank" rel="noopener nofollow ugc">Accumulating gradients for a larger batch size with PyTorch</a>
</h4>

<div class="tags">
  <strong>pytorch</strong>
</div>

<div class="date">
  
  answered by
  <a href="https://stackoverflow.com/users/5987698/gooddeeds" target="_blank" rel="noopener nofollow ugc">
    GoodDeeds
  </a>
  on <a href="https://stackoverflow.com/questions/70119232/accumulating-gradients-for-a-larger-batch-size-with-pytorch/70119359#70119359" target="_blank" rel="noopener nofollow ugc">03:39AM - 26 Nov 21 UTC</a>
</div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
Tying weight of attention matrix

username: Rini
cooked: <p>Hello,</p>
<p>Is there a way to tie the weights of certain parts of a (square) attention matrix?<br>
For example if I divide the matrix in 4 blocks (2 diagonal and 2 off-diagonal) and want to tie the weights of the two off diagonal blocks how could I do that?</p>
<p>Thanks!</p>

username: tumble-weed
cooked: <p>say full attention matrix size is <code>N</code> and block size is <code>block_N</code></p>
<pre><code class="lang-auto">attention_W  = torch.zeros(N,N)
attention_W_diag1 = torch.randn(block_N,block_N).requires_grad_(True)
attention_W_diag2 = torch.randn(block_N,block_N).requires_grad_(True)
attention_W_off_diag = torch.randn(block_N,block_N).requires_grad_(True)
attention_W[:block_N,:block_N] = attention_W_diag1
attention_W[block_N:,block_N:] = attention_W_diag2
attention_W[block_N:,:block_N] = attention_W_off_diag
attention_W[:block_N,block_N:] = attention_W_off_diag
</code></pre>

username: Rini
cooked: <p>Thank you for explaining with the example! I think this is what I was looking for</p>
Sequence to class with transformers question

username: ixombi
cooked: <p>I have sequences of sensor measurements and I need to classify what the sensors are measuring. It is a sequence to class kind of problem. I looked at using nn.Transformer but it expects the target to be of shape (N,T,E) and I only have a class number. Is there a work around to input only a class label or is nn.Transform the wrong solution? Thanks.</p>
Create new tensor from individual scalar values of existing one

username: Matthias_Moller
cooked: <p>Dear PyTorch community,</p>
<p>assume that I have a one-dimensional tensor</p>
<pre><code class="lang-auto">torch::Tensor base = torch::linspace(0,1,10);
</code></pre>
<p>as attribute of a class.</p>
<p>In a member function of this class I want to return a new tensor that is composed of individual entries of the <code>base</code> tensor, e.g.</p>
<pre><code class="lang-auto">torch::Tensor eval(int k)
{
   return { base[k], base[k+1] }; // This code does not compile, it's only meant for illustration purposes
}
</code></pre>
<p>This is just one example. In my real application I want to return 2d tensors, e.g., of the form</p>
<pre><code class="lang-auto">[ [ base[k] , base[k+1] , 0]
  [ 0       , base[k+2] , base[k+3] ] ]
</code></pre>
<p>Any help is appreciated.</p>

username: Matthias_Moller
cooked: <p>It seems that I found a solution which I am providing here in case someone else is interested</p>
<pre><code class="lang-cpp">return torch::reshape(torch::stack( { base[k], base[k+1] } ), {1,2});
</code></pre>
<p>In contrast to posts from some time ago, <code>torch::stack( ... ).view(1,2)</code> does not seem to work.</p>
<p>In an attempt to extend the above to the 2d tensor case, I figured out that</p>
<pre><code class="lang-cpp">return torch::reshape(torch::stack( { base[k], base[k+1], torch::zeros(1), 
                                      torch::zeros(1), base[k+2], base[k+3] } ), {2,3});
</code></pre>
<p>yields a segmentation fault. What works is to make</p>
<pre><code class="lang-cpp">torch::Tensor zero = torch::zeros(1);
</code></pre>
<p>a member attribute of the surrounding class and replace the above by</p>
<pre><code class="lang-cpp">return torch::reshape(torch::stack( { base[k], base[k+1], zero[0], 
                                      zero[0], base[k+2], base[k+3] } ), {2,3});
</code></pre>

username: Matthias_Moller
cooked: <p>I looked into the above code once more and found out that <code>torch::reshape</code> can be replaced by <code>view(...)</code>. The above two code snippets then look as follows:</p>
<pre><code class="lang-auto">return torch::stack( { base[k], base[k+1] } ).view( {1,2} );
</code></pre>
<p>and</p>
<pre><code class="lang-auto">torch::Tensor zero = torch::zeros(1);
return torch::stack( { base[k], base[k+1], zero[0], 
                       zero[0], base[k+2], base[k+3] } ).view( {2,3} );
</code></pre>
Convert 'tensor like' Objects to a Tensor without torch.tensor()

username: M_Mitchell
cooked: <p>Older implementations of Pytorch treat a TensorImage and Tensor as though it is the same in terms of doing operations (duck typing). The newest implementation will insist that the types have been specified even if the two objects are compatible (multiple dispatch I guess).</p>
<p>My question is how do I get old code to work i.e. convert a TensorImage or TensorMultiCategory to a plain a Tensor in a loss or forward function.</p>
<p>A quick fix is to just use torch.tensor(tensor_image) but this creates a warning:</p>
<pre><code class="lang-auto">UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
</code></pre>
<p>Using this recommendation or <code>torch.as_tensor</code> does not work, the object e.g. TensorImage does not get converted to a Tensor, it just gets cloned.</p>
<p>Is there a cleaner way of solving this?</p>
How to define a parameter which takes values within a given range?

username: ADONAI_TZEVAOT
cooked: <p>I want to define a n-dimensional parameter that requires grad and also make sure that its value lies within the range of [-a, b] at all times and when the parameter is initialized I want it to take values from the normal Gaussian distribution. Please help!</p>

username: MAsad
cooked: <p>I am unsure of exactly what you are looking for in this (if you can give some example, that will definitely help). One way to achieve this would be to initialise a parameter randomly and always apply a tanh activation function to it before consuming it. This way, you will ensure both the range (-1, 1) and updates from gradients.</p>
<p>Here is a simple example to demonstrate this:</p>
<pre><code class="lang-auto"># initialise randomly from normal distribution
&gt;&gt;&gt; myparameter = torch.randn((100, 100))
&gt;&gt;&gt; print(torch.unique(myparameter))
tensor([-3.7089, -3.5629, -3.4922,  ...,  3.2828,  3.4450,  3.5880])

#  normalise using tanh activation
&gt;&gt;&gt; myparameter_norm = torch.tanh(myparameter)
&gt;&gt;&gt; print(torch.unique(myparameter_norm))
tensor([-0.9988, -0.9984, -0.9981,  ...,  0.9972,  0.9980,  0.9985])

# proceed to use myparameter_norm as input to next compute
</code></pre>

username: ADONAI_TZEVAOT
cooked: <p>I want to use clamp with parameter but don’t know how to do that. Please help</p>

username: MAsad
cooked: <p>From <a href="https://pytorch.org/docs/stable/generated/torch.clamp.html" class="inline-onebox" rel="noopener nofollow ugc">torch.clamp — PyTorch 1.10.0 documentation</a></p>
<pre><code class="lang-auto">&gt;&gt;&gt; a = torch.randn(4)
&gt;&gt;&gt; a
tensor([-1.7120,  0.1734, -0.0478, -0.0922])
&gt;&gt;&gt; torch.clamp(a, min=-0.5, max=0.5)
tensor([-0.5000,  0.1734, -0.0478, -0.0922])

&gt;&gt;&gt; min = torch.linspace(-1, 1, steps=4)
&gt;&gt;&gt; torch.clamp(a, min=min)
tensor([-1.0000,  0.1734,  0.3333,  1.0000])
</code></pre>
Where I can see whether and operation autocasts or not?

username: tjoseph
cooked: <p>Hi,<br>
simple question: Can someone point me to the part of the documentation where I can see how/whether an operation autocasts to fp16 when run with AMP or whether it keeps using fp32. Or can this only be seen from the source code?</p>
<p>Best,<br>
Tim</p>

username: Azerus
cooked: <p>Hi,</p>
<p>See the doc on autocast about <a href="https://pytorch.org/docs/stable/amp.html#autocast-op-reference" rel="noopener nofollow ugc">autocast operation reference</a>, in particular the part about Op-Specific Behavior where there is such a list. Is it what you are looking for?</p>
<p>Best regards,<br>
Thomas</p>

username: tjoseph
cooked: <p>Thank you very much. That’s exactly it.</p>
Concatenating an expanded view of a tensor

username: marcelroed
cooked: <p>I’ve got a module (mostly) out of my control that I’d like to apply to a special tensor of inputs. The input tensor is of shape <code>(batch_size, n_points, 3 + context_dim)</code>, where each element of the batch of shape <code>(n_points, 3 + context_dim)</code> looks like</p>
<pre><code class="lang-auto">[x_1 | context],
[x_2 | context],
...
[x_i | context]
...
[x_n | context]
</code></pre>
<p>Here I’ve used <code>|</code> to indicate concatenation of vectors.</p>
<p>For reference, <code>context_dim</code> is 256, and <code>n_points</code> is 2048. Each <code>x_i</code> is a 3 dimensional vector, and context is the same vector within an element of the batch.</p>
<p>My issue is that this tensor takes up a lot of memory if it’s constructed naively.</p>
<p>I’m familiar with striding and know that it’s possible to expand the context vector to the right shape for concatenation without any extra allocation, but if I understand correctly, the concatenation operation will require a full allocation of this memory, duplicating the context vector <code>n_points</code> times.</p>
<p>From what I can tell, <a href="https://github.com/pytorch/pytorch/issues/22169" rel="noopener nofollow ugc">this RFC</a> might solve my problem, since I could combine the tensors into a NestedTensor, but it seems a long way off.</p>
<p>How can I reduce memory consumption in this situation with current versions of PyTorch?</p>

username: googlebot
cooked: <p>In a similar situation, as my module started with a fully-connected layer, I just used matmul distributivity to do: linear([x_1 | context]; w0) + bias = linear(x_1; w1) + linear(context; w2) + bias. You can actually do inplace summation, and output from “linear(context)”  is a smaller non-expanded tensor.</p>
<p>If your scenario is different, and you can’t process vector parts independently, I don’t this you can do anything about memory, as you have to intersperse data from two sources to pass to your module.</p>
Np.vectorize in pytorch

username: ZenoB
cooked: <p>Is there a way to vectorize this code in pytorch?</p>
<pre><code class="lang-auto">def myfunc(a, b):
    "Return a-b if a&gt;b, otherwise return a+b"
    if a &gt; b:
        return a - b
    else:
        return a + b
</code></pre>
<p>With <code>numpy</code> I can simply do</p>
<pre><code class="lang-auto">@np.vectorize
def myfunc(a, b):
    "Return a-b if a&gt;b, otherwise return a+b"
    if a &gt; b:
        return a - b
    else:
        return a + b
</code></pre>
<p>such that <code>myfunc(a=[1, 2, 3, 4], b=2)</code> returns <code>array([3, 4, 1, 2])</code>.</p>
<p>Is there a way to do the same in pytorch?</p>

username: ZenoB
cooked: <aside class="quote no-group" data-username="ZenoB" data-post="1" data-topic="137896">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zenob/48/37262_2.png" class="avatar"> ZenoB:</div>
<blockquote>
<p>[1, 2, 3, 4]</p>
</blockquote>
</aside>
<p>It seems <code>torch.where()</code> is the way to go:</p>
<pre><code class="lang-auto">a = torch.tensor([1, 2, 3, 4])
b = torch.tensor(2)
torch.where(a&gt;b, a-b, a+b)
</code></pre>
<p>the code above returns <code>tensor([3, 4, 1, 2])</code>.</p>
Gradient flow using Adam optimizer

username: cbd
cooked: <p>I used adam optimizer for training. I trained my model as below:<br>
(1) From scratch for 50 epochs<br>
(2) From scratch for 55 epochs<br>
(3) Load the save model of 50 epochs and run for 5 epochs ( 50+5=55)</p>
<p>For epoch-51, case(2) and case(3) the results are same but for 52 epoch results is different as initialization of first and second moment running average coefficient is different. For case(2), it is in continue while for case(3) it is zero.</p>
<p>Is it issue or model will be stable as training goes on?</p>

username: Azerus
cooked: <p>Hello,</p>
<p>When (1) is done, you should save the state_dict of the model AND optimizer. So when you start (3) you load the pre-trained model and the optimizer to restore its state, as adviced here: <a href="https://pytorch.org/tutorials/beginner/saving_loading_models.html#saving-loading-a-general-checkpoint-for-inference-and-or-resuming-training" rel="noopener nofollow ugc">Saving &amp; Loading a General Checkpoint for Inference and/or Resuming Training</a>.</p>
<p>Best regards,<br>
Thomas</p>
Should I stop or continue with more epochs?

username: Sumera_Rounaq
cooked: <p>I am training WGAN -GP to generate X-ray images. After 1240 epochs I came up with this image.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/6/463623daac69ab4cef71fe88f097937cf51b41bd.png" data-download-href="https://discuss.pytorch.org/uploads/default/463623daac69ab4cef71fe88f097937cf51b41bd" title="fake_images-1240"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/6/463623daac69ab4cef71fe88f097937cf51b41bd_2_690x345.png" alt="fake_images-1240" data-base62-sha1="a17pSJEVuCUjeOd20qyrdjkQhD7" width="690" height="345" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/6/463623daac69ab4cef71fe88f097937cf51b41bd_2_690x345.png, https://discuss.pytorch.org/uploads/default/original/3X/4/6/463623daac69ab4cef71fe88f097937cf51b41bd.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/4/6/463623daac69ab4cef71fe88f097937cf51b41bd.png 2x" data-dominant-color="2E2E2E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">fake_images-1240</span><span class="informations">1022×512 270 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Should I stop here and make some hyperparameters tunings or should I continue with more epochs?</p>
<p>In addition to it, I would also like to know on average how many epochs are usually required to train a model which is expected to generate images.</p>
<p>Any help shall be highly appreciated.</p>
TensorImageUtils.bitmapToFloat32Tensor Function time on android

username: user21
cooked: <p>The development environment is Android.</p>
<p>I am detecting the image object through the TensorImageUtils.bitmapToFloat32Tensor(resizedBitmap,PrePostProcessor.NO_MEAN_RGB,PrePostProcessor.NO_STD_RGB,MemoryFormat.CHANNELS_LAST) function for the bitmap I have.</p>
<p>However, since it takes about 2 seconds in the TensorImageUtils.bitmapToFloat32Tensor function, there is a delay in detection compared to the actual real-time image.</p>
<p>Is there a faster way to process the TensorImageUtils.bitmapToFloat32Tensor function?</p>
So much reserved memory in torch

username: 3yuan
cooked: <p>When using libtorch in my project, I find so much cuda memory reserved by Pytorch, which lead to CUDA out of memory.</p>
<p>what():  CUDA out of memory. Tried to allocate 1010.00 MiB (GPU 0; 10.92 GiB total capacity; 8.69 GiB already allocated; 978.38 MiB free; 8.79 GiB reserved in total by PyTorch)</p>
<p>Is it possible to disable the function? Thanks</p>

username: eqy
cooked: <p>Reserved memory is a superset of allocated memory (they are not separate features), and in this case the difference is only about ~100MiB due to fragmentation.</p>

username: 3yuan
cooked: <p>Any ideas to reduce it?</p>
What does 'torch._C._log_api_usage_once' do?

username: jsg
cooked: <p>For example it is used here,</p>
<pre><code class="lang-auto">def _log_api_usage(identifier: str):
    """
    Internal function used to log the usage of different detectron2 components
    inside facebook's infra.
    """
    torch._C._log_api_usage_once("detectron2." + identifier)
</code></pre>
<p>But why would I need to log these and where does it log and  what is the usage of these logs?</p>

username: tom
cooked: <p>This is strictly debugging and, as the comment indicates, quite likely most of the usage is inside facebook. If you set the environment variable</p>
<pre><code class="lang-auto">PYTORCH_API_USAGE_STDERR=1
</code></pre>
<p>you can see that PyTorch records “which parts of PyTorch are involved”, quite likely to give you a rough idea where to look (and whom to have look) if something unexpected happens.<br>
PyTorch has grown quite huge with lots of moving bits and sometimes unexpected interactions, so having a quick idea of how to narrow things down or compare high-level processing flow between versions is a debug help.</p>
<p>Bast regards</p>
<p>Thomas</p>

username: jsg
cooked: <p>Thank you for the great answer!</p>
Why cannot I call dataloader or model object twice?

username: Hongnan_G
cooked: <p>I have been dabbling in PyTorch for a while and noticed a weird behaviour, likely because of seeding/instantiating issue.</p>
<pre><code class="lang-python">    # Model, cost function and optimizer instancing
    model = models.CustomNeuralNet().to(device)
    if is_forward_pass:
        # Forward Sanity Check
        _forward_X, _forward_y = models.forward_pass(
            loader=train_loader, model=model
        )

    my_trainer: trainer.Trainer = trainer.Trainer(
        params=TRAIN_PARAMS,
        model=model,
        device=device,
        wandb_run=wandb_run,
    )

    curr_fold_best_checkpoint = my_trainer.fit(
        train_loader, valid_loader, fold
    )
</code></pre>
<p>I won’t be able to paste the whole code pipeline as it is pretty long, but it would be good to know why this is the case.</p>
<p>For example, if I set <code>is_forward_pass</code> to False, then my trainer will just start training, and say my loss is 0.1 for epoch 1. As a cautious move, I will always check if this particular loss changes when I modified the function (I value reproducibility a lot, though it is a black box to me). So when I switched on <code>is_forward_pass</code> to True, then my loss changed. Once I turned it off, the loss is the same value so I am sure it is this flag. My hunch is somehow the <code>DataLoader</code> or the <code>Model</code> is being called again? Has anyone faced this issue?</p>

username: ptrblck
cooked: <p>I don’t know which higher-level API you are using, but if I understand your issue correctly you are concerned about the determinism between two different runs using <code>is_forward_pass=True</code> and <code>=False</code>.<br>
If so, I would guess that the pseudorandom number generator is called additionally in the <code>if_forward_pass=True</code> case and you thus won’t get the same values.<br>
This could happen if you are using random transformations inside the <code>DataLoader</code>, shuffling, etc. or if the model itself is using random values e.g. in dropout layers and is not in evaluation mode.</p>

username: Hongnan_G
cooked: <p>I think you are right, I am using a wrapper I coded up for my projects, haha. So I was just wondering if this is expected behaviour.</p>
Saving Pytorch Models independent of the source code

username: astroboy
cooked: <p>I have a question regarding model saving in PyTorch. Let’s say you run a series of experiments and change the code gradually during the project. However, if I save the model state_dict, I would need the exact source code to use the previous pre_trained models. Is there any good way to save a PyTorch model with just enough information to load it even when the source code changes (without the need to go back to the exact git commit that generated the network)? Tensorflow has this option and saves the model and the graph at the same time.  I suppose this is possible with TorchScript, but sometimes TorchScript has limited functionality.</p>

username: ptrblck
cooked: <p>Yes, your assumption is correct and saving the scripted model would allow you to reload it afterwards without the need to keep the source code.<br>
However, as you’ve also mentioned you might need to add some changes to make the model “scriptable”. Depending on this effort, storing the source code for each experiment might be easier and would allow you to apply further experiments to this particular model afterwards.</p>
Roc multiclass problem

username: Giulia_Carvalhal
cooked: <p>Hey community,</p>
<p>I am trying to create a ROC (weighted and per class) in my multiclass segmentation model. But, in the code example of sklearn they binarize the y (before even train test split) to do this multiclass task. So my question is: how to do this binarize my model, which I have the label (y_true, a tensor Bx1xHxW) and the model prediction (y_pred, a tensor BxCxHxW).</p>
<p>Best,<br>
Giulia</p>

username: ptrblck
cooked: <p>If I understand <a href="https://scikit-learn.org/0.15/modules/generated/sklearn.preprocessing.label_binarize.html#sklearn.preprocessing.label_binarize">this example</a> correctly, <code>label_binarize</code> will create a one-hot encoded array for a multi-class target.<br>
Would flattening the target tensor before passing it to <code>label_binarize</code> work?<br>
This would create a flat tensor containing the one-hot-encoded targets for each sample in the batch.<br>
<code>roc_curve</code> also uses the <code>ravel()</code> operation, so it would flatten the arrays anyway.<br>
Let me know, if this would work.</p>
ImageNet-ResNet 34.tar

username: A_Shamsadini
cooked: <p>Hi<br>
How I can find ImageNet-ResNet 34.tar?<br>
Please help me</p>

username: ptrblck
cooked: <p>The pretrained <code>state_dict</code> for <code>ResNet34</code> can be found <a href="https://github.com/pytorch/vision/blob/18cf5ab1c1307e9d0e1b1e4d26ce9c45e04f8376/torchvision/models/resnet.py#L27">here</a>.</p>
Math difference between SGD and Adam

username: Samuel_Bachorik
cooked: <p>Hi, as far i know, SGD is doing:</p>
<pre><code class="lang-auto">x_new = x * learning_rate -gradient
</code></pre>
<p>When we take look at Adam what is Adam doing with gradient and learning rate ?</p>

username: ptrblck
cooked: <p>The <a href="https://pytorch.org/docs/stable/generated/torch.optim.Adam.html?highlight=adam#torch.optim.Adam">docs</a> show the applied formula and the <a href="https://github.com/pytorch/pytorch/blob/5fdcc20d8d96a6b42387f57c2ce331516ad94228/torch/optim/_functional.py#L64-L108">source code</a> might also be helpful to check how it’s implemented.</p>
Dataloader and transform on loaded dataframe

username: savery
cooked: <p>Hi,</p>
<p>I am extremely new to PyTorch and would need a little guidance/clarification.</p>
<p>The tutorial I am inspiring myself from to build my CNN is: <a href="https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html#train-the-network" class="inline-onebox" rel="noopener nofollow ugc">Training a Classifier — PyTorch Tutorials 1.10.0+cu102 documentation</a></p>
<p>I am trying to build my own CNN using a local dataset. The dataset consists of roughly 10,000 images of 56x56.</p>
<p>I have loaded the data using pickle as follows:</p>
<pre><code class="lang-auto">with open("./images_l.pkl", 'rb') as f: imgs = pickle.load(f)
with open("./labels_l.pkl", 'rb') as f: labels = pickle.load(f)
</code></pre>
<p>From the tutorial, it seems as though we want to wrap the dataset in a data loader. So I convert the NumPy array into a dataset and then wrap it in a dataloader as follows:</p>
<pre><code class="lang-auto">trainset = torch.utils.data.TensorDataset(torch.tensor(imgs), torch.tensor(labels))
trainloader = torch.utils.data.DataLoader(trainset, batch_size=4)
</code></pre>
<p>After doing what’s above, defining + instantiating the CNN and setting the appropriate loss function/optimizer, I attempt to train the network, but there seems to be a mismatch with the dimensions which, I assume, is because the transformation is not applied.</p>
<p>Error message:</p>
<pre><code class="lang-auto">Expected 4-dimensional input for 4-dimensional weight [6, 1, 5, 5], but got 3-dimensional input of size [4, 56, 56] instead
</code></pre>
<p>Any guidance would be greatly appreciated.</p>
<p>Thanks,</p>

username: ptrblck
cooked: <p>The channel dimension is missing in your inputs, so you would need to add them via <code>torch.from_numpy(imgs).unsqueeze(1)</code>.</p>

username: savery
cooked: <p>Thank you for your reply!</p>
<p>I was unaware that it is always recommended to use Custom Datasets when not using a PyTorch dataset.<br>
After creating the custom dataset class and instantiating it, I have maintained the desired dimensions, but the images are losing the 3 RGB channels and the images being converted to binary for some reason. I think it’s due to my transforms, but still figuring that out.</p>

username: ptrblck
cooked: <p>You don’t necessarily need a custom <code>Dataset</code>, if you unsqueeze the missing dimension before passing the tensor to <code>TensorDataset</code>.<br>
Based on the error it seems the loaded numpy array is already missing the channel dimension (i.e. the RGB channels) so you could check how these arrays are created.</p>