akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Gradient Checkpointing not backpropping in GNN

Daniel_Murnane: <p>I’m trying to apply gradient checkpointing to a graph neural network, with each iteration of message passing being checkpointed, but the backwards pass is completely not working. I have tried the ideas in other forum questions, such as dummy variables and forcing require_grad_(True), to no avail. The basic structure is:</p>
<pre><code class="lang-auto">class CheckGNN(nn.Module):
    """
    Segment classification graph neural network model.
    Consists of an input network, an edge network, and a node network.
    """
    def __init__(self, in_channels=3, hidden_dim=8, n_graph_iters=3,
                 hidden_activation=torch.nn.Tanh, layer_norm=True):
        super(CheckGNN, self).__init__()
        self.n_graph_iters = n_graph_iters
        # Setup the input network
        self.input_network = mlp(...)
        # Setup the edge network
        self.edge_network = EdgeNetwork(...)
        # Setup the node layers
        self.node_network = NodeNetwork(...)
        # Attempt at dummy input fix
        self.dummy_tensor = torch.ones(1, dtype=torch.float32, requires_grad=True)
        
    def custom_forward(self, inputs):
        # Apply edge network
        e = self.edge_network((inputs[0], inputs[1]))
        e = torch.sigmoid(e)
        # Apply node network
        x = self.node_network((inputs[0], e, inputs[1]))
       
        return x
    
    def forward(self, x, edge_index):
        """Apply forward pass of the model"""
        input_x = x
        x = self.input_network(x)
        # Shortcut connect the inputs onto the hidden representation
        x = torch.cat([x, input_x], dim=-1)
        
        # Loop over iterations of edge and node networks
        for i in range(self.n_graph_iters):
            
            x_inital = x
            x = checkpoint(self.custom_forward, (x, edge_index, self.dummy_tensor))
        
            # Shortcut connect the inputs onto the hidden representation
            x = torch.cat([x, input_x], dim=-1)  
            
            x = x_inital + x
            
        e = self.edge_network((x, edge_index))
        
        return e
</code></pre>
<p>Printing <code>loss.grad</code> gives <code>None</code>, the model weights are empty for the <code>NodeNetwork</code>. They exist for the <code>EdgeNetwork</code> but only because of the final, uncheckpointed run when it outputs the edge scores. The dummy input doesn’t change this. Am I missing something fundamental here?</p>
<p>P.S. I believe the details of the Edge and Node networks are irrelevant, but I can include them if needed.</p>

LBlaabjerg: <p>Hi Daniel, I am facing the same problem. Did you find a solution?</p>
Some questions regarding pruning

vainaijr: <ol>
<li>
<p>what is the difference between randomunstructured pruning and dropout?</p>
</li>
<li>
<p>why would one not want to prune weights based on decreasing l1 norm, that is make weights that have high l1 norm zero, (current l1structured pruning does it based on making weights with lowest l1 norm 0)?</p>
</li>
<li>
<p>what should be distribution of weights after applying pruning?</p>
</li>
</ol>

Michela: <ol>
<li>
<p>Dropout stochastically drops connections at every pass, but then reinstates them at the following pass, while dropping new ones. It’s a regularization technique. After training, at evaluation time, all connections are present again. Random pruning (structured or unstructured) permanently removes units or connections from the network and permanently sets them to zero.</p>
</li>
<li>
<p>You can, if you want to. Usually the “low magnitude = less important” convention is used because it makes more logical sense and has been shown to have some solid empirical and theoretical foundations. But <a href="https://arxiv.org/pdf/1905.01067.pdf" rel="nofollow noopener">people</a> have tried pruning high magnitude weights too, so you can definitely give it a shot if you think that it makes sense.</p>
</li>
<li>
<p>It depends on what pruning technique you use, what sparsity level you achieve, what initialization distribution you started out with, what task you’re solving, what optimization strategy you use, and more. So there is no real answer to this. Of course, if you prune out all connections with low synaptic weight in absolute value, then you will expect only weights with high absolute magnitude to remain.</p>
</li>
</ol>
Storing Quasi-Constants in a Namespace

rslz: <p>I want to store a small set of useful-to-have constants in my model:</p>
<pre><code class="lang-python">ZERO = torch.tensor( float(0.0) )
ONE = torch.tensor( float(0.0) )
INF = torch.tensor( float('inf') )
NAN = torch.tensor( float('nan') )
TRUE = torch.tensor( bool(True) )
</code></pre>
<p>These are often needed when e.g. perfoming operations like  <code>torch.where(mask, tensor, ZERO)</code></p>
<p>What is the best way to store these kinds of constants in my model? I have the follwoing desiderata:</p>
<ol>
<li>They are cast to the correct device/dtype when using <code>model.to</code>
</li>
<li>They are collected in a namespace <code>model.constants</code>
</li>
<li>I do not need to copy-paste the whole code in each model every time</li>
<li>The list is extendable on a per-model basis</li>
<li>It is fully compatible with JIT</li>
</ol>
<p>Currently, I am doing</p>
<pre><code class="lang-python">class model(nn.Module):
    ZERO: torch.Tensor
   
    def __init__(self):
          self.register_buffer('ZERO', torch.tensor(0.0))
</code></pre>
<p>Which satisfies (1, 4, 5) but violates (2, 3). Any better ideas?</p>
Tracing sliced Tensors during forward function

FreddyGump: <p>I’m trying to convert a <a href="https://github.com/mit-han-lab/temporal-shift-module" rel="noopener nofollow ugc">Temporal Shift Module</a> model for action recognition in video into an ONNX format. I have worked with ONNX before and successfully converted models to the format. However, the model that I want to convert now performs some slicing operations during the forward pass.</p>
<p>Basically the chunk of code that is giving the headaches is the following:</p>
<pre><code class="lang-auto">out = torch.zeros_like(x)
out[:, :-1, :fold] = x[:, 1:, :fold]  # shift left
out[:, 1:, fold: 2 * fold] = x[:, :-1, fold: 2 * fold]  # shift right
out[:, :, 2 * fold:] = x[:, :, 2 * fold:]  # not shift
</code></pre>
<ul>
<li>I have read and tested the different suggestions in the github issues of TSM, generally somebody reacts saying “it works for me”. I can assure you, that is a lie. None of the suggestions work.</li>
<li>Secondly, I came across using the <code>ScriptModule</code> of jit, but that doesn’t work either since TemporalShift objects are not callable.</li>
<li>Thirdly, I came across this <a href="https://github.com/onnx/onnx/blob/master/docs/Operators.md#Slice" rel="noopener nofollow ugc">onnx “slice” module</a> but cannot seem to get this one working. The documentation is also not very clear in my opinion.</li>
</ul>
<p>Anyway, I’m really hitting a wall here and I’d appreciate it if somebody could push me in the right direction.</p>
Problem when I convert Pytorch to ONNX

PytorchLearning: <p>Hi all</p>
<p>I have a question concerning when I convert Pytorch to ONNX.</p>
<p>When I use BatchNorm2d in my model for example BatchNorm2d(32) then I convert to ONNX then I test with javascript I get not error.<br>
Howerver when I use nn.GroupNorm in my model for example nn.GroupNorm(2,32) then I convert to ONNX then I test with javascript (I launched the chrome web browser then I check when I select “Inspecter”) I get the following error:</p>
<p>Uncaught (in promise) TypeError: cannot resolve operator ‘Shape’ with opsets: ai.onnx v9</p>
<p>Can someone tell me please how to solve this problem. The problem is when I use nn.GroupNorm.</p>
<p>Thanks a lot in advance</p>

PytorchLearning: <p>Hi all,</p>
<p>Does anyone have an idea how to solve this problem.</p>
<p>I have another question: What is the difference between ONNX, Torchscript and Torchserver?</p>
<p>Thanks a lot for your help.</p>
Get range of value in tensor

shashimalcse: <p>i have a tensor which shape is [4,1,224,224]. This is a 4 grayscale image batch. I want to get certain range values of each image. For example, the image grayscale range is 0 to 255, I want to get 30 to 40 values from the image and other pixels should be zero. is there a good way to do it?</p>

ptrblck: <p><code>torch.where</code> should work:</p>
<pre><code class="lang-python">x = torch.randint(0, 256, (4, 1, 224, 224))
print(x.min(), x.max())
&gt; tensor(0) tensor(255)
out = torch.where((30&lt;=x) &amp; (x&lt;=40), x, 0)
print(out.unique())
&gt; tensor([ 0, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40])
</code></pre>
Training Ratio Parameter

ChiTseHuang: <p>In my work, I need to train a parameter to represent the ratio in [0,1].<br>
I have tried to pass this parameter through sigmoid, tanh, clamp functions.</p>
<pre><code class="lang-python">...
self.register_parameter('ratio',nn.Parameter(torch.FloatTensor([ratio_init])))
...
ratio = getattr(self, 'ratio')
ratio = F.sigmoid(ratio)
parameter = ratio*some_parameter_1 + (1-ratio)*some_parameter_2
</code></pre>
<p>I train this network and hope to get the best ratio.<br>
I want to ask whether my method has some problems?<br>
Or pytorch has other method to restrict the trainable ratio between [0,1]</p>
CUDA error about stream

craaaab: <p>When I’m trying to bind the pytorch model to cuda tensor, I encountered this problem: <strong>RuntimeError: CUDA error: operation would make the legacy stream depend on a capturing blocking stream</strong>,and the code cause this error is simply like <code>model.to(device)</code>, has someone met the same problem? How can I deal with it?</p>

ptrblck: <p>I’m not sure what “trying to bind the pytorch model to cuda tensor” means, so could you explain your use case a bit more, please?</p>

craaaab: <p>my codes are as follows:</p>
<pre><code class="lang-python">device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
model = torchvision.models.resnet34(pretrained=True).to(device)
</code></pre>
<p>and the call of <code>to(..)</code> method causes the problem, will it be much more clear?</p>

ptrblck: <p>Thanks for the update! The error sounds like a setup issue, as it seems to be created during the first CUDA operation.<br>
Try to restart your machine (especially after e.g. a driver update) and rerun the code.<br>
If it’s still failing, reinstall the driver as well as the PyTorch binaries, and recheck it again.</p>
<p>Before doing these steps, you could also try to run your workload in a docker container and check, if this CUDA setup would work.</p>

craaaab: <p>Thanks a lot! The problem has been solved by reinstalling PyTorch with a previous version.</p>
Loop Optimization in Pytorch

KruskalLin: <p>Hi, everyone! I have an array [a0, a1, …, an] and now I want to create a new array [b0, b1, …, bn] where</p>
<blockquote>
<p>b0 = a0<br>
b1 = a0 + a1<br>
b2 = a0 + a1 + a2<br>
…<br>
bn = a0 + a1 + a2 + … + an</p>
</blockquote>
<p>And the code I have written is</p>
<pre><code class="lang-auto">roll = len(A)
B = torch.zeros(2 * roll, device=A.device, dtype=A.dtype)
for i in range(0, roll):
    B[i:i + roll] += A
B = B[:roll]
</code></pre>
<p>which still costs lots of time. Any idea for further optimizing?</p>

Kaga: <p>How about</p>
<pre><code>B=A.clone()
for i in range(1, len(A)):
    B[i]+=B[i-1]</code></pre>

Kaga: <p>Actually, no need to use python loop, torch has a built in function for that <a href="https://pytorch.org/docs/stable/generated/torch.cumsum.html" rel="noopener nofollow ugc">torch.cumsum</a>:</p>
<pre><code>B = torch.cumsum(A, dim=0)</code></pre>

KruskalLin: <p>Thank you so much! Fixed my problem.</p>
Vector output summing to one, best loss function?

Kaga: <p>I know that in classification problems CrossEntropyLoss can be used. However, CrossEntroyLoss expects class index as input, not a one-hot encoded vector. While one-hot encoded vectors and class indices can be mapped to each other one-to-one, in my case I have vector outputs (in training data) that do not necessarily assign 100% of the weight to one element and 0% of the weight to all other elements. Instead the weights can be arbitrarily distributed into the vector elements. This makes it impossible to map the output into integer index without loss of information and so CrossEntropyLoss cannot be used. I thought of simply using MSEloss, but it seems to me that this does not sufficiently take into account the property that all vector entries in the outputs must sum up to 1 (softmax output layer). Is there a loss function in pytorch that would be more appropriate in this case?</p>

ptrblck: <p><a class="mention" href="/u/kfrank">@KFrank</a> shared an implementation for “soft-labels” <a href="https://discuss.pytorch.org/t/soft-cross-entropy-loss-tf-has-it-does-pytorch-have-it/69501/2">here</a>.</p>
Implementation of Meta-Learning for Reinforcement Learning

jortuzar: <p>I’m trying to implement the following algorithm (<a href="https://arxiv.org/abs/1710.03463" class="inline-onebox" rel="noopener nofollow ugc">[1710.03463] Learning to Generalize: Meta-Learning for Domain Generalization</a>)</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/b/db79371722593d3ee61b95dff1162099101f2c5d.png" alt="algo" data-base62-sha1="vjygDGLyylWNX5HdjQCe2Q4jpud" width="494" height="480"></p>
<p>The best approach I was able to come up with was the following pseudocode:<br>
STEP:</p>
<pre><code class="lang-auto">	# 1. Copy the train model
	model_meta_test.load_state_dict(model_train.state_dict())
	optimizer_meta_test.load_state_dict(optimizer_train.state_dict())
	
	# Caluclate loss on the meta test model (A)
	loss_meta_test = self.get_loss(model_meta_test)
	
	# Optimize the meta test model
	self.optimize_network(loss_meta_test, model_meta_test, optimizer_meta_test)
	
	# Now I'm at ϴ' which means I can calculate the loss of my META TEST 
	loss_meta_test = self.get_loss(model_meta_test)
	
	# Now I need to recalculate the loss on the Train Model (B). 
	# Note that the loss at (A) and (B) should be the same.
	loss_train = self.get_loss(model_train)
	
	# Now I calculate the sum of both losses. 
	tot_loss_train = loss_train + self.BETA * loss_meta_test.detach()

	# Finally Optimize the Train Model
	self.optimize_network(tot_loss_train, model_train, optimizer_train)
</code></pre>
<p>I have a few doubts.</p>
<ol>
<li>Is the implementation is correct.</li>
<li>What would be a more efficient way of doing it? This is an overkill of calculations as I have to backup models and calculate the same loss twice for different models. With my limited knowledge, this was the best I could come up with. If anyone can enlighten me, I would greatly appreciate it.</li>
</ol>

jortuzar: <p>Solved this by looking at an example. <a href="https://github.com/Pulkit-Khandelwal/mldg" class="inline-onebox" rel="noopener nofollow ugc">GitHub - Pulkit-Khandelwal/mldg</a></p>
Loading saved models on host with different byteorder

andrewsi-z: <p>Hi all,</p>
<p>I am attempting to load a pre-trained model (LE) on a big endian platform. The model is <a href="https://github.com/ultralytics/yolov5" class="inline-onebox" rel="noopener nofollow ugc">GitHub - ultralytics/yolov5: YOLOv5 🚀 in PyTorch &gt; ONNX &gt; CoreML &gt; TFLite</a></p>
<p>I am seeing that pytorch tensor objects are not converted to the appropriate byteorder. I opened an issue using a simpler model here: <a href="https://github.com/pytorch/pytorch/issues/65300" class="inline-onebox" rel="noopener nofollow ugc">Saved model parameters are not portable to systems with different host byteorder · Issue #65300 · pytorch/pytorch · GitHub</a></p>
<p>In the interim, I am trying to write a python tool to load and convert the tensors. I am hoping the community can provide a bit of guidance on how to (somewhat generically) loop through and find all of the tensor objects and bytes wap them.</p>
<p>So far, I would plan to loop and convert:</p>
<ul>
<li>state_dict</li>
<li>optimizer_dict (I need to investigate this more to see if needed).</li>
</ul>
<p>Are there other parameters we should watch for (generically)?<br>
The quick example I have is the yolo model - in addition to the state_dict, there is a model level stride parameter (also tensor object) that needs to be adjust.</p>
<p>Has anyone experience anything similar? Any thought appreciated.<br>
If there are similar model navigation tools (whether for optimization or other) that you are aware of, that would be great info too.</p>
<p>Thank you for any info you can provide!</p>

googlebot: <p>my understanding is that only on-disk format is wrong, so you should convert what you deserialize. perhaps use numpy’s facilities - <a href="https://numpy.org/doc/stable/user/basics.byteswapping.html" class="inline-onebox" rel="noopener nofollow ugc">Byte-swapping — NumPy v1.21 Manual</a></p>

andrewsi-z: <p>Thanks Alex, that’s what I am utilizing and am able to convert the constants. Now researching other potentially related issues!</p>
Model.zero_grad() or optimizer.zero_grad()?

kunasiramesh: <p>Hi everyone,</p>
<p>I have confusion when to use model.zero_grad() and optimizer.zero_grad()? I have seen some examples they are using model.zero_grad() in some examples and optimizer.zero_grad() in some other example. Is there any specific case for using any one of these?</p>

kunasiramesh: <p>I am training a network on speech data.</p>

ritchieng: <p>If you’re referring to:</p>
<pre><code class="lang-auto">optimizer = optim.SGD(net.parameters())
</code></pre>
<p>They’re the same.</p>

kunasiramesh: <p>I am using :<br>
optimizer = optim.Adam(model.parameters())</p>
<p>In this case what should I use?</p>

ritchieng: <p>Same; whether you use SGD, Adam, RMSProp etc.</p>
<p>Typically I use <code>optimizer.zero_grad()</code>.</p>

dmitryako: <p>model.zero_grad() and optimizer.zero_grad() are the same IF all your model parameters are in that optimizer. I found it is safer to call model.zero_grad() to make sure all grads are zero, e.g. if you have two or more optimizers for one model.</p>

gabrieldernbach: <p>According to<br>
<a href="https://tigress-web.princeton.edu/~jdh4/PyTorchPerformanceTuningGuide_GTC2021.pdf" rel="noopener nofollow ugc">https://tigress-web.princeton.edu/~jdh4/PyTorchPerformanceTuningGuide_GTC2021.pdf</a> (Slide 6)<br>
it seems preferable to call <code>model.zero_grad()</code> for performance reasons (pytorch&gt;=1.7).</p>
<p>EDIT: thanks to ptrblck, I should have put <code>model.zero_grad(set_to_none=True)</code>.</p>

ptrblck: <p>The performance gains mentioned in the tuning guide are coming from setting the <code>.grad</code> attributes to <code>None</code> either manually or via the <code>set_to_none=True</code> argument, which would avoid a memset + read/write operation in the next backward.</p>
Parameterized cross product of matrices?

legoh: <p>Hi folks, I wish to do a cross product of 2 stacks of vectors, but I’d like this operation to be parameterized. For example, I could achieve a <code>n x n</code> matrix if I did <code>torch.matmul(h, k.permute(0, 2, 1))</code>, where <code>h</code> is of <code>size(batch, n, emb)</code> and <code>k</code> is of <code>size(batch, n, emb)</code>.</p>
<p>I’m wondering if there’s a way to do this such that I have a MLP module that takes in the pairs of <code>h</code> and <code>k</code> and projects them? For now, what I’ve done is this</p>
<pre><code class="lang-auto">h = h.repeat_interleave(n, dim=1)
k = k.repeat(1, n, 1)

hk = torch.cat((h, k), dim=1)

j = MLP(hk)
</code></pre>
<p>But this blows up the GPU memory. Is there a more efficient way of doing this?</p>
NamedTuples support in pytorch

Alex_Finkelshtein: <p>Hi,<br>
I am trying to figure out the general approach of the pytorch framework towards namedtuples.<br>
From what I was able to find online, it seems like its partially supported and more and more of pytorch capabilities are extended to support the python named tuples.</p>
<p>I am trying to understand what is the vision of pytorch regarding this, is it some kind of a secondary dtype to be supported that will mostly be lagging behind new features? Or instead this would become a standard for future features and optimizations?</p>
<p>I am at an intersection, before committing to the namedtuples (which have great value for me) in my code, I want to understand where do the winds blow and what to expect.<br>
Is there some official pytorch representative that can answer such a question?</p>
<p>thanks,<br>
Alex</p>

Alex_Finkelshtein: <p>any hint will be highly appreciated, anyone?</p>

googlebot: <p>IMHO, your question makes no sense - named tuples are ad-hoc containers and you’re free to use them as you want while programming models, but there is very little use for them in ML framework interfaces (e.g. sort() returns named tuples, but then - so what).</p>
<p>if you rather want to replace tensors with tensor groups, there is experimental <a href="https://github.com/pytorch/nestedtensor" rel="noopener nofollow ugc">nestedtensor</a> project, can’t comment on its state or usefulness.</p>

Alex_Finkelshtein: <aside class="quote no-group" data-username="googlebot" data-post="3" data-topic="131520">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/g/ba8739/48.png" class="avatar"> googlebot:</div>
<blockquote>
<p>want to replace tensors with tensor groups, there</p>
</blockquote>
</aside>
<p>Thanks for the answer, however it doesn’t answer the question. We can discuss the usefulness of named tuples. However, I am not asking about that.<br>
thanks for the input anyway</p>

googlebot: <p>you need to better explain what you want then, perhaps with a code snippet</p>
Do a 3x3 Kernel search on image/feature and get max value?

soulslicer: <p>Lets say I have a tensor of [B, 1, H, W].</p>
<p>I want to do a 3x3 kernel search on the image, and return the max value found in that kernel so possibly [B, 1, H-1, W-1]</p>
<p>Similarly given [B, 3, H, W], perhaps to do the same 3x3 kernel search, but compute a special function across the 3 channels. For example, to calculate some distance functions across the 3 channels, and return the max value one.</p>
<p>Is there a smart tensorized way that one can achieve this?</p>

ptrblck: <p>The first operation sounds like a max pooling op.<br>
The latter is quite undefined and I would guess you might be able to <code>unfold</code> the input and perform your operations via a matmul and additional operations, if needed.</p>
Effect of tensor modification in forward pass on gradient calculation

thepurpleowl: <p>Consider the forward pass</p>
<pre><code class="lang-auto">def forward(self, x):
  x = self.pool(F.relu(self.conv1(x)))
  x = self.pool(F.relu(self.conv2(x)))
  x = x.view(-1, 16 * 5 * 5)
  x = F.relu(self.fc1(x))
  x = F.relu(self.fc2(x))
  x = self.fc3(x)
  return x
</code></pre>
<p>In model forward passes, if we use tensor transformations such as view(as shown in the example), transpose, permute,reshape - does this effect grad values?</p>
<p>If no, how the mapping between the original and transformed tensor are maintained?</p>

ptrblck: <p>Yes, all the mentioned operations will be tracked by Autocrat such that the gradient will be passed to the corresponding values during the backward pass.</p>

drevicko: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> As a matter of interest, are there any operations that AREN’T tracked by Autocrat that we should keep an eye out for? I work under the assumption that ANY function/processing bundled in pytorch is tracked - is that a reasonable assumption?</p>

ptrblck: <p>Yes, I think this is generally a valid assumption and should be true as long as you don’t use e.g. the <code>.data</code> attribute of a tensor. If the operation is not differentiable or not implemented, Autograd would raise an error.</p>
<aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="132332">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>will be tracked by Autocrat</p>
</blockquote>
</aside>
<aside class="quote no-group" data-username="drevicko" data-post="3" data-topic="132332">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/drevicko/48/14854_2.png" class="avatar"> drevicko:</div>
<blockquote>
<p>AREN’T tracked by Autocrat</p>
</blockquote>
</aside>
<p>Also, interesting autocorrect when trying to type “Autograd”, but I think in the end my smartphone might know who has the absolute power in PyTorch land. <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=12" title=":smiley:" class="emoji" alt=":smiley:" loading="lazy" width="20" height="20"></p>
When I use hook to get output param , but get Proxy() , Why?

MartinZhang: <p>when I use the hook to get output param, but the param’s type is Proxy.</p>
<pre><code class="lang-auto">def hook_get_output(module,  input, output):
    print(output)

for name, layer in module.named_modules():
     layer.register_forward_hook(hook_get_output)
</code></pre>
<p>notes: the module is quantization_module</p>

Vasiliy_Kuznetsov: <p>Are you using FX graph mode quantization to symbolically trace through the model?  If yes, then you are expected to see <code>Proxy</code> objects in the process of symbolic tracing.</p>
<p>If not, could you provide some additional context on what you are feeding through the model after you set up the hook?</p>

MartinZhang: <p>Thanks, m friend.  I made a mistake <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=10" title=":sweat_smile:" class="emoji" alt=":sweat_smile:"></p>
Dropout pytorch GAN

fllci: <p>Hi everyone!</p>
<p>I’ve been trying to add dropout in my discriminator network. But somehow, I keep getting invalid syntax error in my console and I have no idea why. For instance, when I add</p>
<blockquote>
<p>nn.Dropout(0.8)</p>
</blockquote>
<p>after any LeakyRelu, I get this error:</p>
<blockquote>
<p>nn.Conv1d(64, 128, 4, 2, 1, bias=False),<br>
^<br>
SyntaxError: invalid syntax</p>
</blockquote>
<p>Here is my discriminator:</p>
<blockquote>
<p>class Discriminator(nn.Module):<br>
def <strong>init</strong>(self, ngpu):<br>
super(Discriminator, self).<strong>init</strong>()<br>
self.ngpu = ngpu<br>
self.main = nn.Sequential(<br>
# input is (1 x 1 x 4096)<br>
nn.Conv1d(1, 32, 4, 2, 1, bias=False),<br>
nn.LeakyReLU(0.2, inplace=True),<br>
# state size. (1 x 32 x 2048)<br>
nn.Conv1d(32, 64, 4, 2, 1, bias=False),<br>
nn.BatchNorm1d(64),<br>
nn.LeakyReLU(0.2, inplace=True)<br>
# state size. (1 x 64 x 1024)<br>
nn.Conv1d(64, 128, 4, 2, 1, bias=False),<br>
nn.BatchNorm1d(128),<br>
nn.LeakyReLU(0.2, inplace=True),<br>
# state size. (1 x 128 x 512)<br>
nn.Conv1d(128, 256, 4, 2, 1, bias=False),<br>
nn.BatchNorm1d(256),<br>
nn.LeakyReLU(0.2, inplace=True),<br>
# state size. (1 x 256 x 256)<br>
nn.Conv1d(256, 1, 256, 1, 0, bias=False),<br>
nn.Sigmoid()<br>
# state size. (1 x 1 x 1)<br>
)<br>
def forward(self, input):<br>
return self.main(input)</p>
</blockquote>

fllci: <p>My apologies… I am soo noob. I forgot to add a comma after the dropout!!!<br>
I’ve been dealing with this error for 2 entire days lol.</p>
How to calculate matrix trace in 3d tensor

Navpreet_Singh: <p>How to calculate the trace of a matrix in 3d tensor where first dimension is the batch size?<br>
torch.trace only works on 2d input but I want it to run on my batch tensor to get output of shape (batch_size,1)</p>

Rahul_Chand: <p><a class="mention" href="/u/navpreet_singh">@Navpreet_Singh</a></p>
<p>You can define a mask (a matrix of only 0 &amp; 1). Multiply the mask with the matrix you have &amp; then take the sum. Example:</p>
<pre><code class="lang-auto">input_matrix  = torch.ones((4, 5, 5)) #batch_size is 4
mask = torch.zeros((4, 5, 5))
mask[:, torch.arange(0,5), torch.arange(0,5) ] = 1.0 #This will mask all non-diagonal values.

output = input_matrix * mask
output = torch.sum(output,axis=(1,2))  #output will be of dimension (batch) &amp; will contain the trace of each 5 x 5 matrix.

</code></pre>
<p><img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji only-emoji" alt=":slight_smile:"></p>

AlphaBetaGamma96: <p>If you have a 3D tensor where the first dimension is the batch dimension the following will work</p>
<pre><code class="lang-auto">B=1000 #batch size
A=16    #nrows/ncols
matrices = torch.randn(B,A,A) #batch of matrices
trace = matrices.diagonal(offset=0, dim1=-2, dim2=-1).sum(dim=-1) #grab diagonal along last 2 dims, and sum
</code></pre>
Book introducing deep learning

Sylvain_Ard: <p>Hello all !<br>
I want to acquire the basics on deep learning ! A book to advise to me ?<br>
Thank you !<br>
Best regards</p>

ptrblck: <p>Is the recommendation from your <a href="https://discuss.pytorch.org/t/readings-for-improve-my-knowledge-in-deep-learning/131189">other topic</a> not sufficient or are you looking for something else?</p>

Sylvain_Ard: <p>I am looking for a book presenting the concepts of deep learning not especially on pytorch</p>

ptrblck: <p>Ah OK. In that case I would recommend to take a look at <a href="https://www.deeplearningbook.org/">Goodfellow et. al - Deep Learning</a>.</p>

Sylvain_Ard: <p>thank you I will look this</p>
How to fix RuntimeError: size mismatch?

Asya: <p>I receive an error:</p>
<blockquote>
<p>RuntimeError: size mismatch, m1: [3072 x 32], m2: [3072 x 32] at /opt/conda/conda-bld/pytorch_1587428091666/work/aten/src/THC/generic/THCTensorMathBlas.cu:283</p>
</blockquote>
<p>when I’m trying to add classifier to autoencoder:</p>
<pre><code class="lang-auto">class ConvAutoencoder(nn.Module):
    def __init__(self):
        super(ConvAutoencoder, self).__init__()

        # Encoder
        self.conv1 = nn.Conv2d(3, 16, 3, padding=1)
        self.conv2 = nn.Conv2d(16, 4, 3, padding=1)
        self.pool = nn.MaxPool2d(2, 2)

        # Decoder
        self.t_conv1 = nn.ConvTranspose2d(4, 16, 2, stride=2)
        self.t_conv2 = nn.ConvTranspose2d(16, 3, 2, stride=2)

        # Classifier
        self.classifier = nn.Sequential(nn.Linear(32 * 32 * 3, 32))


    def forward(self, x):
        x = F.relu(self.conv1(x))
        x = self.pool(x)
        x = F.relu(self.conv2(x))
        x = self.pool(x)
        x = F.relu(self.t_conv1(x))
        x = F.sigmoid(self.t_conv2(x))
        print(x.shape)
        out = self.classifier(x)

        return x, F.log_softmax(out)
</code></pre>
<p>I changed shapes in <code>nn.Sequential(nn.Linear(32 * 32 * 3, 32))</code> so it would be the same size, but still getting mismatch error. why?</p>

ptrblck: <p>Based on the error message, the activation input has <code>32</code> features so you would need to use <code>in_features=32</code> in the linear layer:</p>
<pre><code class="lang-python">lin = nn.Linear(32 * 32 * 3, 32)
x = torch.randn(32 * 32 * 3, 32)
out = lin(x)
&gt; RuntimeError: mat1 and mat2 shapes cannot be multiplied (3072x32 and 3072x32)


# works
lin = nn.Linear(32, 32)
out = lin(x)
</code></pre>
<p>However, it also seems you are not flattening the activation before feeding it to the linear layer, so make sure this is indeed the wanted behavior.</p>
How does GridSample handle overlap of same pixels?

soulslicer: <p>Lets say I have have a repeat of a number of pixels that i want to transform a bunch of values to. But those values of course are of different sizes.</p>
<p>Does it always overwrite / select the larger value, or is it random? (esp on GPU)</p>

ptrblck: <p>Based on this warning from the docs:</p>
<blockquote>
<p>When using the CUDA backend, this operation may induce nondeterministic behaviour in its backward pass that is not easily switched off. Please see the notes on <a href="https://pytorch.org/docs/stable/notes/randomness.html">Reproducibility</a> for background.</p>
</blockquote>
<p>I would assume it’s randomly overwriting the values.</p>
Possible bug? Linear layer does not throw error on incorrect input size when on GPU

Ada: <p>My network threw an error during backprop because layer shapes did not match, and I wondered how/why the network was able to do the forward pass without throwing an error.  After investigating, I’ve discovered some weird behavior.</p>
<p>If the model is on the CPU, all is well:</p>
<pre><code class="lang-auto">model = nn.Linear(100,1)
x = torch.randn(1,200)
model(x)
</code></pre>
<p>throws <code>RuntimeError: mat1 and mat2 shapes cannot be multiplied (1x200 and 100x1)</code>, as expected.</p>
<p>However, if I put it on the GPU:</p>
<pre><code class="lang-auto">device = 'cuda:0'
model.to(device)
model(x.to(device))
</code></pre>
<p>it happily computes without throwing an error.</p>
<p>Is this expected behavior?</p>

KFrank: <p>Hi Ada!</p>
<aside class="quote no-group quote-modified" data-username="Ada" data-post="1" data-topic="132432" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ada/48/32203_2.png" class="avatar"> Ada:</div>
<blockquote>
<p>throws <code>RuntimeError: mat1 and mat2 shapes cannot be multiplied (1x200 and 100x1)</code>, as expected.</p>
<p>However, if I put it on the GPU:<br>
…<br>
it happily computes without throwing an error.</p>
</blockquote>
</aside>
<p>This appears to be a known bug that has recently been fixed.  See<br>
this thread:</p>
<aside class="quote quote-modified" data-post="1" data-topic="130664">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/matthewleigh/48/60035_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/pytorch-is-allowing-incorrect-matrix-multiplication-when-using-cuda/130664">Pytorch is allowing incorrect matrix multiplication when using cuda</a> 
  </div>
  <blockquote>
    Hey everyone, 
I have a minor issue and I am not sure if it is a bug or I am simply not understanding something. But I noticed that in my project I was getting some strange results for certain configurations and error messages not popping up when expected. 
To boil it down, I am allowed to multiply matrices of incompatible sizes so long as it takes place on the gpu. 
When running the following: 
input = torch.randn(5, 10)
network = nn.Linear(3, 3)

output = network(input) ## This should fail

I …
  </blockquote>
</aside>

<p>Best.</p>
<p>K. Frank</p>

Ada: <p>Thanks for linking that post!  I had not succeeded in choosing the right keywords to find it!</p>
Slicing a 3D tensor

David_Ireland: <p>Suppose I have a tensor <code>X</code> of shape <code>(B, N, 1)</code> and a tensor of indices that I want to slice the first dimension of <code>X</code> by, call this <code>Y</code> and is of shape <code>(B, 1)</code> (optionally could be squeezed to be of shape <code>B</code>).</p>
<p>Suppose that the first element of <code>Y</code> is <code>j</code>. How do I slice <code>X</code> so that I am returned a tensor <code>Z</code> of shape <code>(B, 1)</code> where the first element of <code>Z</code> is the <code>j</code>th element of <code>X</code> along the first dimension.</p>
<p>An example would be if <code>X</code> is <code>[[[1], [2]], [[3], [4]]]</code> and <code>Y</code> is <code>[[0], [1]]</code> then I would want <code>Z</code> to be <code>[[1], [4]]</code>.</p>

Rahul_Chand: <p><a class="mention" href="/u/david_ireland">@David_Ireland</a></p>
<p>You can use <code>torch.gather(...)</code>. Link : <a href="https://pytorch.org/docs/stable/generated/torch.gather.html" rel="noopener nofollow ugc">torch.gather — PyTorch 1.9.1 documentation</a></p>
Why next_functions[0][0] is called with no loss on a branch

amsword: <p>In the following code, there is no loss on dup_x, but dup_x_acc is called? I assume the next function will be called only if the grad_fn is called. However, in this case, dup_x.grad_fn is not called if I’m correct.</p>
<pre><code class="lang-python">      x = torch.randn(2, requires_grad=True)                                                                                                                                                                
      l = x.sum()

      dup_x = x.expand_as(x)                                                                                                                                                                                
      dup_x_acc = dup_x.grad_fn.next_functions[0][0]
      def hook(*args, **kwargs):
          logging.info('called')
      dup_x_acc.register_hook(hook)                                                                                                                                                                   
      l.backward()
</code></pre>

Varal7: <p>You can get better intuition of what’s happening by using torchviz:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/6/c616746451928c8d5a166563c7a778f26feaaad7.png" data-download-href="https://discuss.pytorch.org/uploads/default/c616746451928c8d5a166563c7a778f26feaaad7" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/6/c616746451928c8d5a166563c7a778f26feaaad7_2_220x500.png" alt="image" data-base62-sha1="sgmEb0Y7J8BLtPLctH8Ixpxa0F9" width="220" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/6/c616746451928c8d5a166563c7a778f26feaaad7_2_220x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/c/6/c616746451928c8d5a166563c7a778f26feaaad7_2_330x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/c/6/c616746451928c8d5a166563c7a778f26feaaad7_2_440x1000.png 2x" data-dominant-color="F0F2EF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">624×1416 38.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The node <code>dup_x.grad_fn.next_functions[0][0]</code> is the <code>AccumulateGrad</code> that you see in the first figure, which corresponds exactly to the  <code>AccumulateGrad</code> node you see in the second figure.</p>
<p>You can verify this with:</p>
<p><code>assert(id(dup_x.grad_fn.next_functions[0][0]) = id(l.grad_fn.next_functions[0][0]))</code></p>
Force gradients computation even in inference mode

aliutkus: <p>hi,</p>
<p>for some reason, I am using gradient descent as part of the forward function of my model.</p>
<p>Hence, I wonder if it would be possible to enforce gradient computations in all cases, even if there is some <code>no_grad</code> context manager that is activated ?</p>
<p>thanks a lot</p>

Varal7: <p>Hi Antoine,</p>
<p>You can use <code>torch.set_grad_enabled(True)</code> to activate gradient tracking even under a <code>no_grad</code> context.</p>
<p>For instance:</p>
<pre><code class="lang-python">import torch

x = torch.ones(5, requires_grad=True)
with torch.no_grad():
  torch.set_grad_enabled(True)
  y = x * x

y.sum().backward()
print(x.grad)  # works fine
</code></pre>
<p>However, this does not work if you disabled gradients with the <code>torch.inference_mode</code> context:</p>
<pre><code class="lang-python">x = torch.ones(5, requires_grad=True)
with torch.inference_mode():
  torch.set_grad_enabled(True)
  y = x * x

y.sum().backward()  # fails
</code></pre>

aliutkus: <p>Hi Victor,<br>
ok, great, thanks a lot for the answer.</p>
<p>just another question then:<br>
is there a way to detect inference mode to issue some warning ?</p>

Varal7: <p>Yes, you can detect if we <code>torch.is_inference_mode_enabled()</code></p>
Efficient way to convert an `nn.module` into a single tensor

omarfoq: <p>Hello,</p>
<p>What is the best way in order to gather the parameters of a neural network (given as an <code>nn.Module</code> object) into a single tensor (vector) and vice-versa?</p>
<p>The simplest option seems to be looping over the <code>self.parameters()</code>, but I am not sure how efficient it is especially when we want to reconstruct the model from a tensor.</p>

ptrblck: <p>Yes, iterating the <code>.parameters()</code> and <code>.bufers()</code> (and flattening the tensors, if desired) sounds like the right approach.</p>
Pytorch, CUDA, and NCCL

ndronen: <p>I’d like to upgrade NCCL on my system to <a href="https://docs.nvidia.com/deeplearning/nccl/release-notes/rel_2-10-3.html" rel="noopener nofollow ugc">2.10.3</a>; it supports bfloat16, which I’d like to use. I don’t know the dependency relationships among Pytorch, CUDA, and NCCL. Does Pytorch have bindings to particular versions of NCCL, as suggested by <a href="https://github.com/pytorch/pytorch/pull/41608" rel="noopener nofollow ugc">this issue</a>? Can I choose to use a newer version of NCCL without upgrading either Pytorch or CUDA?</p>

ptrblck: <p>The PyTorch binaries ship with a statically linked NCCL using the <a href="https://github.com/pytorch/pytorch/tree/master/third_party">NCCL submodule</a>. The current CUDA11.3 nightly binary uses NCCL 2.10.3 already, so you could use it.<br>
On the other hand, if you want to use a specific NCCL version, which isn’t shipped in a binary release, you could build from source and use your locally installed NCCL via:</p>
<pre><code class="lang-auto">NCCL_INCLUDE_DIR="/usr/include/" \
    NCCL_LIB_DIR="/usr/lib/" \
    USE_SYSTEM_NCCL=1 \
    python setup.py install
</code></pre>

ndronen: <p>Thanks. Do you know when the current CUDA 11.3 nightly will become official?</p>

ptrblck: <p>We are currently targeting PyTorch <code>1.10.0</code> as the stable release using the CUDA11.3 runtime.</p>

ndronen: <p>Thank you, ptrblock.</p>
CUDA Out of Memory; but enough is free

Xristos: <p>Hello, I’m running a transformer model from the huggingface library and I am getting an out of memory issue for CUDA as follows:</p>
<p><code>RuntimeError: CUDA out of memory. Tried to allocate 48.00 MiB (GPU 0; 3.95 GiB total capacity; 2.58 GiB already allocated; 80.56 MiB free; 2.71 GiB reserved in total by PyTorch)</code></p>
<p>How does this error make any sense if it need to allocate 48 MiB but there are still 80.56MiB free?</p>
<p>Here is my setup:</p>
<p>Ubuntu 20.04<br>
MX150</p>

googlebot: <p>its because of fragmentation, if you’re using like 90% device memory, it will fail to find big contiguous free blocks.</p>
<p>you can try to explicitly do python’s garbage collection and torch.cuda.empty_cache(), but this only helps in some cases. another thing is to try to avoid allocating tensors of varying sizes (e.g. varying batch sizes).</p>
Memory leak in custom cuda extension

hos-b: <p>Hi,</p>
<p>I have a relatively large memory leak in my training script. The leaked memory is not allocated in the GPU but rather in RAM. It only occurs when I use a network with a custom cuda kernel. Another version of the network without the cuda extension does not have this issue. As far as I know, this extension does not have any inherent memory and simply calculates the forward/backward pass for a mathematical equation.</p>
<p>I have tried the following:<br>
<strong>1) Check the memory in the garbage collector</strong><br>
I used an edited version of <a href="https://gist.github.com/Stonesjtu/368ddf5d9eb56669269ecdf9b0d21cbe" rel="noopener nofollow ugc">this gist</a> to analyze tensors in RAM. I did not find any irregularities there.</p>
<p><strong>2) Use <code>tracemalloc</code></strong><br>
I again could not find the source of the leak. I did however notice the custom module allocating ~50 MBs of RAM after an epoch, despite the fact that the encompassing network and all of its input were on GPU. At that snapshot, the leak was ~5 GBs large.</p>
<p>The leak can take up to 60GBs in a day and given that <code>tracemalloc</code> cannot find anything, I assume the allocation is happening outside the python interpreter (e.g. inside the extension). Is there any way to pinpoint the problem further?</p>
<p>Thanks in advance</p>
How to transpose 3D matrix in PyTorch v0.2

maggie: <p>I want to transpose a 3D matrix, it can be done by .t() in PyTorch v0.1. However, I update my PyTorch version and I can’t do it this way.<br>
The error log is like this:<br>
raise RuntimeError(“t() expects a 2D Variable, but self is {}D”.format(self.dim()))<br>
RuntimeError: t() expects a 2D Variable, but self is 3D</p>

miguelvr: <p>use <code>x.permute()</code> -&gt; <a href="http://pytorch.org/docs/master/tensors.html?highlight=permute#torch.Tensor.permute" rel="nofollow noopener">http://pytorch.org/docs/master/tensors.html?highlight=permute#torch.Tensor.permute</a></p>

maggie: <p>You are right. Thank you.</p>

mirzaim: <p>If you just want to reverse the dimension, you could use <code>x.T</code> → <a href="https://pytorch.org/docs/master/tensors.html#torch.Tensor.T" rel="noopener nofollow ugc">https://pytorch.org/docs/master/tensors.html#torch.Tensor.T</a></p>
Load .png into PyTorch with different folders as labels

Reddie23: <p>Hi, everyone!</p>
<p>I’m pretty new on PyTorch, and until now I’ve done several tutorials in which I think they download the data from torchvision but at the moment I never loaded my own data.</p>
<p>I am trying to do a classification task with three labels, and I have several png’s distributed in three different folders depending on the label.</p>
<p>I have the folder ‘dataset’ in the same folder where the .py file I’m working on is, and the path from here is ‘/dataset/train/each label’.</p>
<p>So there are two questions:</p>
<p>How do I get to the png files, upload them and transform them to tensors? It seems that it’s not looking inside the same folder where my PyTorch file is…</p>
<p>And second, once I get to the files, how do I label the files in each folder? Usually, as long as I made models, label and sample are in the same file…</p>
<p>The code I have for now is this:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torchvision
import torchvision.transforms as transforms


train_path = './dataset/train/Covid'

train_dataset = torchvision.datasets.ImageFolder(
    root=train_path,
    transform=transforms.ToTensor()
)
</code></pre>
<p>It doesn’t even find the directory…</p>
<p>Thanks in advance!</p>
Recovering from Dataloader Error / Exception with utils.data.Subset

aksg87: <p>I recently came up with an approach to recover from a DataLoader issue using a Subset as an alternative to <a href="https://github.com/pytorch/pytorch/issues/1137#issuecomment-618286571" class="inline-onebox" rel="noopener nofollow ugc">add exception handler for DataLoader when reading a damaged image file · Issue #1137 · pytorch/pytorch · GitHub</a></p>
<p>Curious about one aspect –  the subset doesn’t inherit the original methods from the dataset, so I have to save a copy of the original dataset. Any way of getting around this? Feels a bit messy.</p>
<p>Overview of approach:</p>
<pre><code class="lang-auto">    from torch.utils.data import Dataset, DataLoader, Subset

    inference_ds_copy = inference_ds   # Do I require original copy?

    selected = list(range(len(inference_ds_copy)))
    inference_ds = Subset(inference_ds_copy, selected)  # type: ignore
    loader = DataLoader(
        inference_ds,
        batch_size=batch_size,
        num_workers=0,
    )

    total_batches = len(loader)
    loop = iter(tqdm(loader, total=total_batches))

    for batch_idx in range(total_batches):
        try:
            batch = next(loop)

            batch_pred = inference_step(
                ...
            )

            inference_ds_copy.save_prediction_batch(
                ...
            )

        except Exception as e:
            log.info(f"Iteration {batch_idx} | Exception: {e}")

            # rebuild dataloader with remaining subset
            selected = list(range(batch_idx, len(inference_ds_copy)))
            inference_ds = Subset( inference_ds_copy, selected)  # type: ignore
            loader = DataLoader(
                inference_ds,
                batch_size=batch_size,
                num_workers=0,
            )

            # restore progress bar
            pbar = tqdm(loader, total=total_batches)
            pbar.n = batch_idx + 1
            pbar.refresh()
            loop = iter(pbar)
</code></pre>