akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Neural Network with variable size input

username: Federico_Ottomano
cooked: <p>Hello,</p>
<p>I’m wondering if there is a pretty straightforward way (possibly this is trivial) to implement a neural network in PyTorch with a variable number of input features.</p>
<p>Many thanks,</p>
<p>Federico</p>

username: ptrblck
cooked: <p>It depends on the model architecture and which dimensions are variable.<br>
E.g. if you are working with a CNN the spatial size can be variable and you would usually use an adaptive pooling layer to create a defined activation shape before feeding it to the first linear layer. Linear layers accept a variable input shapes as <code>[batch_size, *, in_features]</code>, RNN layers accept a variable sequence length etc. so you would need to give a bi more information about your use case.</p>

username: Federico_Ottomano
cooked: <p>Thanks, what about a <em>standard</em> fully connected neural network?</p>

username: KFrank
cooked: <p>Hi Frederico!</p>
<aside class="quote no-group" data-username="Federico_Ottomano" data-post="3" data-topic="138590" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/federico_ottomano/48/43102_2.png" class="avatar"> Federico_Ottomano:</div>
<blockquote>
<p>Thanks, what about a <em>standard</em> fully connected neural network?</p>
</blockquote>
</aside>
<p>Focusing on your word “standard,” no, a fully-connected network will<br>
not be able to accept a variable number of input features.</p>
<p>Let’s say you have an input batch of shape <code>[nBatch, nFeatures]</code><br>
and the first network layer is <code>Linear (in_features, out_features)</code>.<br>
If <code>nFeatures != in_features</code> pytorch will complain about a dimension<br>
mismatch when your network tries to apply the weight matrix of your<br>
first <code>Linear</code> to the input batch.</p>
<p>If you want to do something like this you will have to pad your input<br>
feature vectors to all have the same length or similarly use “dummy”<br>
values for “missing” features in the shorter feature vectors.</p>
<p>Bear in mind that <code>feature[7]</code> in one sample has to have the same<br>
meaning as <code>feature[7]</code> in another sample – otherwise your network<br>
won’t be able to “learn” what <code>feature[7]</code> means.  Therefore any<br>
scheme you use to “pad” feature vectors to the same length has to<br>
make sure that features with the same meaning end up in the same<br>
location in the input vectors.</p>
<p>Best.</p>
<p>K. Frank</p>

username: Federico_Ottomano
cooked: <p>Many thanks to both!</p>
What % of peak performance can I expect from my GPU?

username: pascalm
cooked: <p>Hi,</p>
<p>I currently train my neural network on my home computer. I have a 1660 Super and the input size of my data is 880k entries which result in ~8gb. So there’s enough data to really get that 1600 Super working.</p>
<p>Now gpustat is currently telling me that I have an average workload of 58%. Looks a bit small to me but maybe that’s fine, I have zero experience with cuda.</p>
<p>Now if my parallel application running on a “cpu cluster” would achieve 60% of the peak performance, then that’d be beleivable. Usually you are limited by communication, file IO or whatever else but with cuda and gpu? What can be expected?</p>
<p>I’m running arch linux and didn’t configure much. I connected via SSH, there is nothing running on that machine that’s visual.</p>
<p>Is 58% good or should I expect more?</p>

username: my3bikaht
cooked: <p>For me it is usually 95 to 97%. You are probably losing some time on IO operations.</p>
Partial feedforward network

username: Ahmad_Pouramini
cooked: <p>How can I define a network like the following?<br>
I want two or more LSTM networks (here one has 3 and other 4 states), some of these states could be related (equal letters here). I want to connect them to a partial feedforward layer and finally output some outputs (here 5 equal to distinct letters).</p>
<pre><code class="lang-auto">A  --------------A
B  ----------|--B
C  -------|-----C  
        |     |
B-----|     |
C ---------|
D  -----------D
E  ---------- E
</code></pre>
<p>Could you please guide me, I am novice with Pytorch</p>
PyTorch for CPU with pip-compile

username: ar2
cooked: <p>Hello, I’m using <code>pip-compile</code> (from <a href="https://pypi.org/project/pip-tools/" rel="noopener nofollow ugc">pip-tools</a>) for managing versions of python packages. I put them in <code>requirements.in</code> and this command generates <code>requirements.txt</code> for me.</p>
<p>I want to use the CPU version of PyTorch. This could be done with <code>torch==1.10.0+cpu</code>. However, the CPU specification is provided in the version string, not in the name of the package. This also means that I’m requesting a specific release version (<code>1.10.0</code>).</p>
<p>How can I specify in <code>requirements.in</code> that I want to use <strong>the newest version</strong> of PyTorch, but specifically the one for <strong>CPU</strong>?</p>
<p>I’ve tried the following</p>
<pre><code class="lang-auto">--find-links https://download.pytorch.org/whl/cpu/torch_stable.html

torchvision==1.*+cpu
</code></pre>
<p>but it didn’t work.</p>
<p>So far I found a proper name for the <code>+cpu</code> part. It’s called a <a href="https://www.python.org/dev/peps/pep-0440/#local-version-identifiers" rel="noopener nofollow ugc">local version label</a>.</p>
How are Tensors on CPU created?

username: maxbit
cooked: <p>Hi,</p>
<p>I am trying to implement basic support for having Persistent Memory (Intel Optane PMem) instead of DRAM as the underlying storage of a Tensor. PMem memory regions are basically also just regular pointers.</p>
<p>However, I am currently not able to understand where the actual memory of a Tensor is allocated. Is there any documentation on that? I found that there are several libraries that are actually backing tensors (C10, ATen), but without a first introduction it’s hard to dive into that level of detail. For example, if I torch.load() a tensor from disk, how does the call stack work?</p>
<p>Thank you so much for giving me initial pointers.</p>
<p>Best,<br>
Maximilian</p>
Training a UNet model with Pytorch

username: Ella
cooked: <p>I am training a UNet model on a dataset of 100 images. The image size is 1024 x1024.<br>
I am using NVIDIA Tesla k80, 56 RAM. My batch size is two images due to GPU memory restrictions.<br>
The training process takes a lot of time, I am training four days so far, and it still doesn’t finish.<br>
I am not sure if it normal process due to the large image size, or maybe I do something wrong. If anyone encounters with training images 1024x1024 and how much time does it take?<br>
Thanks in advance <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Trying to use a Transformer for a simple contrived task doesn't work

username: yotam
cooked: <p>Hi,<br>
For some reason I tried feeding random binary sequences into a small transformer and training the model to predict the value at position k+1 given the transformer output at position k. The sequences are summed with learnt positional embedding, with the aim that the model learns to query the input sequence in the right position.</p>
<p>However this does not work at all and the model does not learn anything.<br>
I tried hunting for bugs with no success.<br>
I tried many hyperparams with no success.<br>
I tried predicting value at position k from outputs at position k and it works easily (residual connection probably)<br>
I tried replacing the Transformer with a feedforward net and it works easily.<br>
Can’f find my bug! or is this a difficult task for a transformer? sounds weird</p>
<p>The code can be found here: <a href="https://github.com/yotam-happy/TransformerBuffle/blob/main/train.py" class="inline-onebox" rel="noopener nofollow ugc">TransformerBuffle/train.py at main · yotam-happy/TransformerBuffle · GitHub</a><br>
Would greatly appreciate if anyone spots my bug or has some explanation <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: mMagmer
cooked: <p>Random binaries are  <a href="https://en.wikipedia.org/wiki/Independent_and_identically_distributed_random_variables" rel="noopener nofollow ugc">i.i.d.</a>s<br>
Hence, you cannot predict one, given other samples.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="yotam" data-post="1" data-topic="139368">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/y/c0e974/48.png" class="avatar"> yotam:</div>
<blockquote>
<p>I tried replacing the Transformer with a feedforward net and it works easily.</p>
</blockquote>
</aside>
<p>I’m not sure I understand the use case completely, but were you able to predict the next number in a sequence of random numbers without any signal using a feed-forward model?<br>
If so, I would check for a data leak, since as <a class="mention" href="/u/mmagmer">@mMagmer</a> described, I would expect all models to fail predicting random numbers (unless the PRNG is giving you some information about the sequence).</p>
Numerical problems with torch.nn.functional.kl_div still can be reproduced

username: kolkir
cooked: <p>I have problem with calculating <code>torch.nn.functional.kl_div</code> because I regularly got negative output values. I’ve thought that problem is in some numerical instability. I found that such problems were already addressed in the following issue <a href="https://github.com/pytorch/pytorch/issues/32520" class="inline-onebox" rel="noopener nofollow ugc">Numerical problems with torch.nn.functional.kl_div · Issue #32520 · pytorch/pytorch · GitHub</a> by adding <code>log_target</code> flag to the <code>kl_div</code> function. Unfortunately such solution didn’t resolve problems in my particular case. I will be very appreciate if someone could look at the code and give some advice.</p>
<p>Link to the data and code sample <a href="https://drive.google.com/drive/folders/1PvWaFPNG9WrVtaUAaHUcCi3HPNOkOxsj?usp=sharing" class="inline-onebox" rel="noopener nofollow ugc">kl_div_problem - Google Drive</a></p>
CUDA out of memory, but why?

username: C_J
cooked: <p>When I run my code, I get this runtime error below:</p>
<p>CUDA out of memory. Tried to allocate 88.00 MiB (GPU 0; 8.00 GiB total capacity; 6.04 GiB already allocated; 0 bytes free; 6.17 GiB reserved in total by PyTorch)</p>
<p>I don’t understand why it says 0 bytes free;<br>
Maybe I should have at least 6.17 - 6.04 = 0.13 GiB free?</p>
<p>I’m afraid if it’s not really a memory shortage problem.</p>

username: ptrblck
cooked: <p>You are trying to allocate 88MB. ~130MB are in the cache, but are not a contiguous block, so cannot be used to store the needed 88MB. 0B are free, which shows that the rest of your device memory is used to store the CUDA context, memory for other applications etc.</p>
Legality of adding new layers to built-in faster rcnn with forward hooks?

username: Arda_Tumay
cooked: <p>I am looking for a way to add new conv2d and relu layers in between the backbone and RPN of Faster R-CNN. I have already created a custom nn.module for new layers. Moreover, I discovered the existence of forward hooks that is called after each forward pass. So I thought that I can get the output activations of the last layer of the backbone of resnet and feed these activations to my new module and get activations from it so that I can append these new activations to input of RPN.</p>
<p>Is this a valid way to modify such kinds of networks?</p>
<p>If it is a valid way, does my new nn.module get gradient updates as usual?</p>
<p>Thank in advance for any help.</p>

username: Arda_Tumay
cooked: <p>Nock Nock. Is there anyone out there to have an idea about this?</p>

username: ptrblck
cooked: <p>Forward hooks would give you the activation output of a specific layer and you could feed them to any additional layers without breaking the computation graph. Based on your description I don’t know if this is your use case or if you want to manipulate the model internally and reuse other (later) layers.<br>
In this case, it might be easier to check the source code of the model and reuse it to create a custom model.</p>
Problem at the verification step

username: violet
cooked: <p>Hello,<br>
I have installed pytorch with anaconda and with cuda, and I made the verification proposed :</p>
<p>import torch<br>
x = torch.rand(5, 3)<br>
print(x)</p>
<p>Do I need to get the exact same numbers than the result shown, or is it just this structure that I have to get (I get the same structure but different results) ?<br>
Thank you !!</p>

username: ptrblck
cooked: <p>You shouldn’t expect to see the exact same results, since you are not seeding the code and even if so your setup might differ (e.g. you might be using a different PyTorch version etc.).<br>
If the code returns random values, this smoke test was successful.</p>
Runtime error in gradient computation

username: Rashmi_S_Murthy
cooked: <p>Hi,<br>
When I try to train a model, I face an error.<br>
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [20, 256, 1, 1]] is at version 3; expected version 2 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!</p>
<p>Any idea what is causing this?<br>
Thanks.</p>

username: ptrblck
cooked: <p>The error is usually raised by disallowed inplace operations, which would manipulate tensors which are needed to compute the gradients in the backward pass.<br>
Here is a small example:</p>
<pre><code class="lang-python"># works
w = nn.Parameter(torch.randn(1, 1))
x = torch.randn(1, 1)

y = w * x
y.backward() # works

# fails
w = nn.Parameter(torch.randn(1, 1))
x = torch.randn(1, 1)

y = w * x
x += 1 # x is needed to compute the gradient wrt w in the multiplication, so you can't manipulate it
y.backward()
# &gt; RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [1, 1]] is at version 2; expected version 1 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).
</code></pre>
<p>Check for these inplace manipulations and replace them with their out-of-place equivalents.</p>
LBFGS optim cant deal with multiple returns in closure

username: ricbrag
cooked: <p>I found an issue using LBFGS optimizer.<br>
I need to return some tensors computed inside closure along with the loss value. Adam and SGD work well but LBFGS can’t handle the extra tensors and returns:</p>
<p>TypeError: float() argument must be a string or a number, not ‘tuple’</p>
<p>Toy example:</p>
<pre><code class="lang-auto">import torch
X=torch.tensor([1,2,3,4], dtype=torch.float32)
Y=torch.tensor([2,4,6,8], dtype=torch.float32)
w=torch.tensor(0.0, dtype=torch.float32, requires_grad=True)

# model prediction
def forward(x):
    return w*x

print(f'Prediction before training: f(5) = {forward(5):.3f}')

# training
lr = 0.1
nit=100

loss= torch.nn.MSELoss()
#optimizer=torch.optim.SGD([w],lr=lr)
#optimizer=torch.optim.Adam([w],lr=lr)
optimizer=torch.optim.LBFGS([w],lr=lr)

def closure():

    optimizer.zero_grad()
    y_pred=forward(X)
    myloss=loss(Y,y_pred)
    myloss.backward()
    #print('ypred is:',pred)
    return myloss,y_pred

for epoch in range(nit):

    l,pred=optimizer.step(closure)
    
    if epoch %10 ==0:
        print(f'epoch {epoch+1}: w = {w:.3f},. loss = {l:.8f}')

print(f'Prediction after training: f(5) = {forward(5):.3f}')
print('ypred:',pred)
</code></pre>
<p>Example adapted form:<br>
<a href="https://github.com/python-engineer/pytorchTutorial/blob/master" class="onebox" target="_blank" rel="noopener nofollow ugc">https://github.com/python-engineer/pytorchTutorial/blob/master</a><br>
File:06_1_loss_and_optimizer.py</p>

username: mMagmer
cooked: <p>See <a href="https://pytorch.org/docs/stable/_modules/torch/optim/lbfgs.html#LBFGS.step" rel="noopener nofollow ugc">LBFGS.step</a>.<br>
There are  lines as follows:</p>
<pre><code class="lang-auto">orig_loss = closure()
loss = float(orig_loss)
</code></pre>
<p>It expects that output of closure to be a number.<br>
Other methods are different in implementation of step function.</p>
LBFGS on dataset larger than memory

username: ricbrag
cooked: <p>I want to perform optimization using LBFGS but my dataset is very large so I can only fit 1/32rd of it in memory.<br>
I’m planning to split the dataset in 32 batches.<br>
Unfortunately, with this approach LBFGS will get a different gradient every step but, I know that LBFGS requires a smooth gradient.<br>
My idea is then to accumulate the gradients of all batches before calling optimizer.step(), thus simulating a full batch optimization.</p>
<p>The problem is that LBFGS requires a closure function where the gradient must be zeroed every time.<br>
How can I accumulate the gradient and perform an optimizer.step only with the accumulated gradient?</p>
Class Embeddings in ViT

username: Malik7115
cooked: <p>I have been trying to implement a toy Vision Transformer from scratch. In the paper it is mentioned that there is an additional class embedding that is trained. My question is how can this be implemented in PyTorch. As I understand, nn.Embedding is a lookup table, if I use that it will expect me give an input to generate a tensor, which would not be possible during inference. What would be the correct way to implement the learnable class embedding?</p>
<p>Any help would be appreciated!</p>
Distributed training creates a lot of processes on other GPUs with memory 0

username: Elegant_Lin
cooked: <p>Hi, everyone,</p>
<p>I used <code>DistributedDataParallel</code> and <code>nccl</code> as backend in my code. I ran my code as <code>python3 -m torch.distributed.launch --nproc_per_node=8 train.py</code>. However, Pytorch creates a lot of redundant processes on other GPUs with memory 0 as shown below.</p>
<p>I used 8 * 3090 and Pytorch 1.7. It creates 8 processes on every single GPU.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/f/bf55a883e6d96b6dd7c0c3384d17269e27f3d1d3.png" data-download-href="https://discuss.pytorch.org/uploads/default/bf55a883e6d96b6dd7c0c3384d17269e27f3d1d3" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/f/bf55a883e6d96b6dd7c0c3384d17269e27f3d1d3_2_349x500.png" alt="image" data-base62-sha1="riCIzd0U5YGQrpNiiMCQ1zRS3SP" width="349" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/f/bf55a883e6d96b6dd7c0c3384d17269e27f3d1d3_2_349x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/b/f/bf55a883e6d96b6dd7c0c3384d17269e27f3d1d3_2_523x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/b/f/bf55a883e6d96b6dd7c0c3384d17269e27f3d1d3_2_698x1000.png 2x" data-dominant-color="3C4651"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1002×1432 204 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Does anyone know the reason? Thanks!</p>

username: mrshenli
cooked: <p>Looks like each process is using multiple GPUs. Is this expected? If not, can you try setting <code>CUDA_VISIBLE_DEVICES</code> env var properly for each process before creating any CUDA context?</p>

username: Elegant_Lin
cooked: <p>Hi, mrshenli,</p>
<p>Thanks for your reply. I did not know how to set <code>CUDA_VISIBLE_DEVICES</code> for each process. Could you please give me more information?</p>
<p>Thanks a lot! <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>

username: wqm
cooked: <p>Did you solve this problem?</p>
Can neural network outputs discrete values[0:255]?

username: dalseeroh
cooked: <p>Hello,</p>
<p>My network predicts the values of missing pixels of a grayscale image. However, since the image is normalized, the network gives the normalized values between 0 and 1 with float32 datatype. I guess if the network could output discrete values in [0,1,2,…,255], it would be much more simpler and faster. Should I just put the threshold function that discretizes the float32 output? Do you have any ideas?</p>

username: eqy
cooked: <p>Converting from “tensor” images in [0,1] to unsigned bytes “PIL” images is a fairly standard transformation (e.g., just multiply by 255 and cast) as in <code>to_pil_image</code>:<br>
<a href="https://pytorch.org/vision/main/_modules/torchvision/transforms/functional.html#to_pil_image" class="inline-onebox" rel="noopener nofollow ugc">torchvision.transforms.functional — Torchvision main documentation</a>.</p>
<p>Alternatively, if you wanted a “discrete” output, you could frame it as a 255-class classification problem per pixel.</p>
How can I write my own activation function with cuda C++?

username: yeungkong
cooked: <p>Hello, users of pytorch. The reason that I want to write my own activation function with cuda C++ is that I want to calculate my own backpropagation. I tried to find some tutorials but very limited resources can be found. A tutorial here <a href="https://pytorch.org/tutorials/advanced/cpp_extension.html" class="inline-onebox" rel="noopener nofollow ugc">Custom C++ and CUDA Extensions — PyTorch Tutorials 1.10.0+cu102 documentation</a> is useful. I think some modifications of present files of an activation function would be the easiest way for me to write my own one. Could someone please tell me where should I start or suggest other resources for my request? Thank you in advance.</p>

username: googlebot
cooked: <p><a href="https://github.com/pytorch/examples/blob/master/cpp/autograd/autograd.cpp" rel="noopener nofollow ugc">autograd examples</a> perhaps (NB: “activation” characteristic doesn’t imply anything for coding or autograd, you need just “a function”).</p>

username: yeungkong
cooked: <p>Thank you for this resources. I am trying to figure it out. The resources use auto gradient. Is there any tutorials that use no auto gradient and write their own backward functions?</p>

username: googlebot
cooked: <p>have you scrolled down to “Inherit from Function” section? that’s it - subclassing the Function class, almost the same as in Python.</p>
Transformers in O(log(n))

username: J_Johnson
cooked: <p>A recent paper by Google show it’s possible to greatly reduce the memory footprint of transformers.</p>
<p>If and when would something like this be implemented in Pytorch?</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://arxiv.org/abs/2112.05682">
  <header class="source">
      <img src="https://static.arxiv.org/static/browse/0.3.2.8/images/icons/favicon.ico" class="site-icon" width="32" height="32">

      <a href="https://arxiv.org/abs/2112.05682" target="_blank" rel="noopener nofollow ugc">arXiv.org</a>
  </header>

  <article class="onebox-body">
    <img src="https://static.arxiv.org/icons/twitter/arxiv-logo-twitter-square.png" class="thumbnail onebox-avatar" width="500" height="500">

<h3><a href="https://arxiv.org/abs/2112.05682" target="_blank" rel="noopener nofollow ugc">Self-attention Does Not Need $O(n^2)$ Memory</a></h3>

  <p>We present a very simple algorithm for attention that requires $O(1)$ memory
with respect to sequence length and an extension to self-attention that
requires $O(\log n)$ memory. This is in contrast with the frequently stated
belief that...</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
Network initialization with nn.module

username: amirsepehr_aminian
cooked: <p>I am trying to use an inner class as my nn but it seems that the superclass does not recognize the name. Does anybody have any solutions for this?</p>
<p>this is the snippet of code that is getting an error:</p>
<pre><code class="lang-auto">class FashionMNISTmodel:
    class FashionNet(nn.Module):
        def __init__(self, activationfcn):
            super(FashionNet,self).__init__()
</code></pre>
<p>This is where I initialize the inner class:</p>
<pre><code class="lang-auto">        self.net = self.FashionNet(activationfcn = self.actfcn).to(device)
</code></pre>
<p>this is the error i’m getting:<br>
NameError: name ‘FashionNet’ is not defined</p>

username: ptrblck
cooked: <p>This works for me:</p>
<pre><code class="lang-python">class FashionMNISTmodel:  
    def __init__(self):    
        self.net = self.FashionNet(torch.relu)
        
    class FashionNet(nn.Module):
        def __init__(self, activationfcn):
            super().__init__()
    
    
model = FashionMNISTmodel()
print(model)
print(model.net)
</code></pre>
Variance or Confidence Interval for outputs

username: sadulaev
cooked: <p>This question might be trivial, but I am wondering if there is any variance associated with the logits in the final fully connected layer. In particular, is there a confidence interval for each of logit in the final layer? The documentation says that the weights are intialised from a uniform distribution, U(-\sqrt{k},\sqrt{k}), so would the variance be that of the uniform?</p>
<p>Thanks</p>

username: tom
cooked: <p>Typically you only use one copy of the network, so the variance of the initialization doesn’t give you much information here.<br>
One approach to this is to run the same input through a network several times while dropout is enabled.<br>
A classic in this regard is <a href="https://arxiv.org/abs/1506.02142" rel="nofollow noopener">Gal and Ghahramani: Dropout as a Bayesian Approximation</a>.</p>
<p>Best regards</p>
<p>Thomas</p>
Comparison operators

username: Sia_Rezaei
cooked: <p>The comparison operator <code>&gt;</code> seems to return a <code>at::Tensor</code> type.<br>
How do I convert it into a boolean to use in an <code>if</code> statement.<br>
Or does it just return 0 or 1 for false and true?</p>
<p>In general, am I missing something or the documentation for the C++ API is very poor?</p>

username: ptrblck
cooked: <p><code>tensor.item&lt;bool&gt;();</code> should work.<br>
Contributions to improve the docs are more than welcome <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=10" title=":wink:" class="emoji" alt=":wink:"></p>

username: Sia_Rezaei
cooked: <p>Thanks!<br>
I have contributed to the Python docs before, and I would definitely contribute to the C++ docs if my C++ was so rusty <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=10" title=":smiley:" class="emoji" alt=":smiley:"></p>
How to calculate gradient with respect to input

username: Bahar_Nikpour
cooked: <p>Hi,</p>
<p>I want to find the gradient of a network with respect to one of its inputs, not the weights. I was wondering how should I do so. Here is a snippet of my code:</p>
<pre><code>            manager.train()
            worker.eval()
            goal1 = manager(ds)
            goal2 = manager(dq)
            goal2 = Variable(goal2,requires_grad=True ).float().to('cuda')
            e1, CO1, mask1, width, height = initialize_the_state(num_masks, ds, size=84)
            e2, CO2, mask2, width, height = initialize_the_state(num_masks, dq, size=84)
            for st in range(2):
                mask1, e1, _, CO1 = take_one_step(worker, goal1, ds, CO1.to('cuda'), e1, mask1, width, 
                         height, num_masks)
                mask2, e2, pr, CO2 = take_one_step(worker, goal2, dq, CO2.to('cuda'), e2, mask2, width, 
                          height, num_masks)
                data_shot = e1
               data_query = e2
               with torch.no_grad():
                 proto = model(data_shot)
               logits = euclidean_metric(model(data_query), proto)
               pred = torch.argmax(logits, dim=1)
               acc = count_acc(logits, label)
               reward = torch.tensor(acc)
               m_loss = (torch.sum(torch.mul(pr.reshape([1]).to('cuda'), 
                           Variable(reward).mul(-1).to('cuda')))).to(
                           'cuda')
              optimizer_m.zero_grad()
              optimizer_w.zero_grad()
              m_loss.backward()
              optimizer_m.step()
</code></pre>
<p>manager and worker are both CNNs. The output of manager is given to the worker as the input, and the output of the worker is used to calculate the manager’s loss. The gradient of manager loss is calculated based on the gradient of pr which is the worker’s output w.r.t goal2, which is the input of worker (worker gets goal2, dq and CO2 as the input). Would you please help me with how to do that?<br>
Many thanks for your help</p>
CUDA memory builds up linearly with batches on privacy engine

username: divyat09
cooked: <p>I am training on MNIST images using resnet-18 architecture on a 16 GB GPU memory machine. Everything works perfectly if I do not attach the privacy engine to my optimizer, each batch takes about 1-2 GB GPU memory, which is flushed out after processing each batch, and hence the total GPU memory consumption stays around 2 GB throughout the process of training.</p>
<p>However, when I attach the privacy engine to my optimizer, the used GPU memory keeps accumulating with each batch. Hence, with the processing of each batch, the used GPU memory increases, and soon I receive CUDA out of memory error.</p>
<p>I am not able to understand why GPU memory is not being freed at the end of every batch upon the inclusion of the privacy engine.</p>

username: alexandresablayrolle
cooked: <p>Do you observe memory leak during training or test? Also, are you running the latest version of Opacus?</p>

username: divyat09
cooked: <p>I observe this during training, and I am using the opacus version 0.13.0</p>

username: alexandresablayrolle
cooked: <p>I see. Leaks can happen if you do more forward passes than backward passes as the activations do not get deleted. Is that the case for you? Also is it possible to get a minimal reproducing code sample?</p>

username: divyat09
cooked: <p>Thanks! I think that was the issue, there was an extra forward pass. However, I am not sure why this does not cause any issues when I train without the privacy engine.</p>

username: RevoGen
cooked: <p>I was facing a similar problem when I was using <code>GradSampleModule</code>. The problem was that I needed to call <code>model.zero_grad()</code> instead of <code>optim.zero_grad()</code> to clear the accumulated <code>grad_sample</code> attributes.</p>
Binary Tree lookup with PyTorch?

username: pumplerod
cooked: <p>I am attempting to construct a binary tree lookup table and would like to be able to do so using torch Tensors.  Is there a known process for accomplishing this?  For example, I’m trying to learn about the torch.nn.Embedding class now and I suspect there may be a way of using this but I’m struggling a bit.</p>
<p>My current implementation relies on creating a class object with either a set of pointers to the branches in this node or a value to be returned.  This is then called recursively until I reach a value, but it is quite slow.  Intuition tells me there should be a way to generate a tensor which could be used as a lookup if I supply the indices.  I’m rather a novice though and there is so much I don’t know about what is possible here.</p>
How to freeze a model?

username: Ahmad_Pouramini
cooked: <p>I use</p>
<pre><code class="lang-auto">        for p in model.parameters():
            p.requires_grad = False
</code></pre>
<p>to freeze a T5 model, but when I print parameters that require grad, there is still one parameter with the size <code>32000x512</code>. What is this? Is it embeddings  matrix? Should I freeze it too? It seems backward gradients affect this one remaining parameter</p>

username: Ahmad_Pouramini
cooked: <p>It seems I have called <code>model.resize_token_embeddings(len(tokenizer))</code> after freezing parameters, and it can affect the embeddings require_grad</p>
Indexing with slice and tuple of tensors

username: AreTor
cooked: <p>The following indexing</p>
<pre><code class="lang-auto">t = torch.rand(50, 2, 20, 20, ...)  # suppose to have n + 2 dimensions
ids = torch.meshgrid(*[torch.arange(10) for _ in range(n)])
print(t[:, :, ids])
</code></pre>
<p>with <code>n = 2</code> gives this error</p>
<pre><code class="lang-auto">TypeError: only integer tensors of a single element can be converted to an index
</code></pre>
<p>I am searching for a working solution with a generic <code>n </code>. The same indexing can be instead performed with NumPy.</p>
<p>Is there a way to perform it?</p>

username: AreTor
cooked: <p>I figured out a (not elegant) solution:</p>
<pre><code class="lang-auto">t = torch.rand(50, 2, 20, 20, ...)  # suppose to have n + 2 dimensions
ids = torch.meshgrid(*[torch.arange(10) for _ in range(n)])
ids = (slice(None),) * 2 + ids 
print(t[ids])
</code></pre>
How can I calculate all cross-terms in pytorch?

username: Masashi_Tsubaki
cooked: <p>I would like to calculate all cross-terms of each vector in a matrix.<br>
For example, consider the following matrix:</p>
<pre><code class="lang-auto">X = tensor([[1, 2, 3],
		    [4, 5, 6]]),
</code></pre>
<p>and I would like to obtain all cross-terms of each vector in this matrix as:</p>
<pre><code class="lang-auto">Y = [[1*1, 1*2, 1*3, 2*2, 2*3, 3*3],
	 [4*4, 4*5, 4*6, 5*5, 5*6, 6*6]].
  = [[1, 2, 3, 4, 6, 9],
     [16, 20, 24, 25, 30, 36]].
</code></pre>
<p>That is, this is the all combination values of the vector elements and I believe that this can be calculated using torch.combinations; however, torch.combinations does not provide the batch implementation and I couldn’t produce the above result in pytorch.</p>
<p>How can I calculate all cross-terms in pytorch? I assume that X is the batch of feature vectors and I would like to implement a regression model with the cross terms of the feature vectors Y as input.</p>

username: googlebot
cooked: <p>should be doable if you make combinations of indexes instead (i.e. of elements from arange(K,dtype=int64)), and use the resulting tensor for “advanced indexing” on X. so you’ll get value pairs and prod(dim=x) reduction will finally yield the desired output.</p>

username: J_Johnson
cooked: <pre><code class="lang-auto">Y = torch.cat([(X[0:1,:].T@X[0:1,:]), (X[1:2,:].T@X[1:2,:])]).unique().view(2, -1)
</code></pre>
<p>Typed on my phone so please excuse any typos. But should be something like this.</p>
How to count elements in a tensor given a certain condition?

username: Saleh_qaryan
cooked: <p>Hello,</p>
<p>I am trying to find the accuracy of a regression model with a 10% error margin. So what I am doing to calculate accuracy is this:<br>
abs(model output - target) &lt;= 0.1*target</p>
<p>how can I do that in tensors?</p>
<p>I subtracted the two tensors and got the absolute value using torch.abs and torch.sub() now I want to count how many elements are under the 10% margin. How do I do that?</p>
<p>Thank you!</p>

username: ptrblck
cooked: <p>Your code would work and you call <code>mean()</code> to get the ratio (or <code>sum()</code> to get the number of elements meeting the condition).<br>
Something like this might work:</p>
<pre><code class="lang-python">output = torch.randn(10, 10)
target = torch.randn(10, 10)
res = torch.abs(output - target) &lt;= 0.1*target
res.float().mean()
res.float().sum()
</code></pre>
Backward only for a batch, local variable 'step_size_min' referenced before assignment

username: bastien_batardiere
cooked: <p>Hello everyone,</p>
<p>I am trying to maximize a real function witt n arguments:</p>
<p>$$<br>
f : \mathbb R ^n \mapsto R<br>
f(X) = \frac 1 n \sum ^n f_i(X_i), X_i \in \mathbb R<br>
$$</p>
<p>However, I want  to maximize it in a batch-spirit, meaning that I want to maximize the batch_size first components, then the other batch_size…</p>
<p>Basically, I want to do this :</p>
<pre><code class="lang-auto">import torch 
X =  100+torch.arange(30, dtype = torch.float32)
X.requires_grad_(True)

def get_batches(X): 
    for i in range(3):
        yield (X[i*10:(i+1)*10])

optim = torch.optim.Rprop([X], lr = 0.1)
for j in range(3): 
    for batch in get_batches(X):
        l = torch.norm(batch)
        l.backward()
        optim.step()
        optim.zero_grad()
</code></pre>
<p>Here, I want to set all the components of X to 0, but I want to do so in a batch-manner, meaning that I want to take the batch_size first components of X, doing a gradient ascent only on those first components, then taking the batch_size others, doing a gradient ascent only on those etc.<br>
rprop is storing step sizes for the n components. For each batch, I want rprop to update the step sizes only on the right components, i.e. the batch_size components I take. Here I am doing so, but I think Rprop does not update the step sizes correctly. Is there a way to tell him which step sizes to update ?</p>
<p>Basically, I want that the script above does the same as the script below :</p>
<pre><code class="lang-auto">X =  100+torch.arange(30, dtype = torch.float32)
X.requires_grad_(True)
optim = torch.optim.Rprop([X], lr = 0.1)

for j in range(3): 
    l = torch.norm(X)
    l.backward()
    optim.step()
    optim.zero_grad()
</code></pre>
<p>If you print X, you can see that the non-batch gradient ascent is faster, since rprop is updating correctly the step sizes.<br>
I have already tried to set requires_gradient = False to the right components, but it throw me an error :</p>
<pre><code class="lang-auto">local variable 'step_size_min' referenced before assignment
</code></pre>
<p>Thanks for your answer !</p>
Should model be in train mode when saving it?

username: Kalle
cooked: <p>When saving a model like the following snip of code, does it then matter if the model is in train() or eval() mode? Does train() and eval() change the model.state_dict()? And if so, should model.train() be called before saving the model?</p>
<pre><code class="lang-auto">for epoch in range(5):
    # prepare model for training
    model.train()
    for i, (local_batch, local_labels) in enumerate(training_generator):
        ....

    # prepare model for evaluation
    model.eval()
    for i, (local_batch, local_labels) in enumerate(validation_generator):
        ....
    
    torch.save(model.state_dict(),  os.path.join('models', f'model_{epoch+1:03}.pt'))    
</code></pre>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Kalle" data-post="1" data-topic="139337">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/aeb1de/48.png" class="avatar"> Kalle:</div>
<blockquote>
<p>Does train() and eval() change the model.state_dict()?</p>
</blockquote>
</aside>
<p>No, it doesn’t. Calling <code>model.train()</code> or <code>model.eval()</code> will change the internal <code>self.training</code> flag to <code>True</code> or <code>False</code>, respectively. This flag is then used in some modules (e.g. <code>nn.Dropout</code> and batchnorm layers) so switch the behavior in their <code>forward</code> method.</p>

username: Kalle
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="139337">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>This flag is then used in some modules (e.g. <code>nn.Dropout</code> and batchnorm layers) so switch the behavior in their <code>forward</code> method.</p>
</blockquote>
</aside>
<p>Do you mean that I should put train() in this function?</p>
<pre><code class="lang-auto">def forward(self, x):
    return self.model(x)
</code></pre>

username: ptrblck
cooked: <p>No, you would call <code>model.train()</code> and <code>model.eval()</code> before starting the training and evaluation once.</p>
GPU memory shoot up while using cuda11.3

username: Azarudeen
cooked: <p>We are using NER model. Our model was trained using torch 1.6. It took around 1.9 GB of RAM. We have moved from Tesla V100 to A100. As A100 can be run only on Cuda&gt;=11.0 we have been forced to install the latest PyTorch with 11.3 support. While doing this, the memory needed for our model increased from 1.9 GB to 3.2 GB.</p>
<p>Could anyone shed some idea here on why such a huge difference and what can be done to reduce this?  Will retraining model using torch 1.10 reduce memory usage?</p>
<p>I am attaching stats on different version and their memory usage. Thanks for your help.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/4/d49496aece4b3d48c90e943638f8e742319aeabc.png" data-download-href="https://discuss.pytorch.org/uploads/default/d49496aece4b3d48c90e943638f8e742319aeabc" title="Screenshot 2021-12-09 at 5.40.25 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/4/d49496aece4b3d48c90e943638f8e742319aeabc_2_440x500.png" alt="Screenshot 2021-12-09 at 5.40.25 PM" data-base62-sha1="ukzzxY1ZceK4f9D6qRDCjQ0RMPa" width="440" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/4/d49496aece4b3d48c90e943638f8e742319aeabc_2_440x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/d/4/d49496aece4b3d48c90e943638f8e742319aeabc_2_660x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/d/4/d49496aece4b3d48c90e943638f8e742319aeabc_2_880x1000.png 2x" data-dominant-color="FBF9F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2021-12-09 at 5.40.25 PM</span><span class="informations">1380×1568 34.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>I don’t know how you’ve measured the GPU memory usage and also don’t know if you are facing any issues (e.g. if you had to lower the batch size or change the training in any way).<br>
Some expected reasons for a different memory usage:</p>
<ul>
<li>the CUDA context size is expected to increase due to the additional SMs on A100 compared to V100</li>
<li>besides that, also the libraries increased in size (CUDA, cuDNN etc.)</li>
<li>If you are checking the peak memory, note that cuDNN could pick kernels with a larger workspace on A100, if they are not yielding OOM errors and are giving a speedup (especially if <code>cudnn.benchmark=True</code> is used). You can limit the cuDNN workspace via <code>CUDNN_CONV_WSCAP_DBG</code>
</li>
</ul>

username: Azarudeen
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>Thanks for the reply. I took what amount of GPU was occupied after loading the model. I don’t face any issues other than the difference in memory consumption.</p>
Incorrect loss in a fully-connected NN

username: Saleh_qaryan
cooked: <p>Hello,</p>
<p>I am working on a project to predict soccer player values from a set of inputs. The data consists of about 19,000 rows and 8 columns (7 columns for input and 1 column for the target) all of numerical values.</p>
<p>I am using a fully connected Neural Network for the prediction but the problem is the loss is not decreasing as it should.</p>
<p>The loss is very large (1e+13) and doesn’t decrease as it should, it just fluctuates.</p>
<p>This is the code I am using to run the model:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/8/a8211d289528e2732c89c40aa6e4db933d6a051f.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/a8211d289528e2732c89c40aa6e4db933d6a051f" title="gradient"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/8/a8211d289528e2732c89c40aa6e4db933d6a051f_2_690x386.jpeg" alt="gradient" data-base62-sha1="nZl9ZO152m8NV5XC0NV6qTZdnEb" width="690" height="386" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/8/a8211d289528e2732c89c40aa6e4db933d6a051f_2_690x386.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/a/8/a8211d289528e2732c89c40aa6e4db933d6a051f.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/a/8/a8211d289528e2732c89c40aa6e4db933d6a051f.jpeg 2x" data-dominant-color="EEEFEE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">gradient</span><span class="informations">964×540 74.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The model is fully connected neural network with 4 hidden layers, each with 7 neurons. input layer has 7 neurons and output has 1. I am using MSE for loss function. I tried changing the learning rate but it is still bad.</p>
<p>What could be the reason behind this?<br>
Thank you!</p>

username: ptrblck
cooked: <p>Could you check the <code>logits</code> range and compare it to the <code>outputs</code>? Since the loss is extremely large I would guess that <code>outputs</code> (i.e. your targets) might also contain large values? If so, try to normalize them during training and un-normalize the model output during validation to get the expected value ranges again.</p>
How can i make feature selection in ensemble class?

username: mathwseg
cooked: <p>I need help in making feature selection in the result I got from merging two features. does a selection technique mean removing the zeros numbers from the result file and does it possibly make the result better ? I used the ensemble class for concatenation … is there any example for feature selection or how can I find one ?</p>

username: mathwseg
cooked: <p>is there any help please ?</p>

username: ptrblck
cooked: <p>For general feature selection approaches, take a look at <a href="https://scikit-learn.org/stable/modules/feature_selection.html">sklearn’s feature selection</a>. While these techniques are mostly targeting the input features, they might still be useful in case you want to adapt them to your use case. I don’t know any standard approach in selecting features from different models before concatenating them, but someone else might <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=10" title=":wink:" class="emoji" alt=":wink:"></p>