akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to get all registerd buffer by self.register_buffer()

dragen: <p>Hi, we can get all required gradient tensor by <code>self.parameters()</code> method,<br>
but how to get all non-required gradient tensor created by <code>self.register_buffer()</code> method?</p>

ptrblck: <p>You could filter the <code>state_dict</code>, which holds all buffers and parameters.</p>
<pre><code class="lang-auto">dict(filter(lambda v: (v[1].requires_grad==False), model.state_dict().iteritems()))
</code></pre>

dragen: <p>nn.Parameters Could also be <code>requires_grad=False</code>, So by filter no gradients may not work, I guess?.</p>

ptrblck: <p>Yeah, you are right. Besides that, it seems all parameters stored in <code>state_dict</code> have <code>requires_grad=False</code> by default.</p>
<p>You could try to call the internal buffer dict, but it’s not recommended, since it’s an internal method:</p>
<pre><code class="lang-auto">model._buffers
</code></pre>
<p>Could you explain a bit about your use case, i.e. why you need all buffers without their specific name?</p>

dragen: <p>in some case, i need a class buffers list as <code>self.params=[tensor() for i in range(n)]</code>. Since the buffer is a list and its number is determined by run-time parameters <code>n</code>, I prefer to access these buffers using <code>self.params[i]</code> instead of giving each tensor an name.<br>
<a class="mention" href="/u/ptrblck">@ptrblck</a></p>

dashesy: <p>Same question. Is there a canonical way to register a list of buffers (not just a single buffer)?</p>
<p>I currently do this:</p>
<pre><code class="lang-auto">self.params = []

for ii in range(3):
    pname = 'param_{}'.format(ii)
    self.register_buffer(pname, torch.tensor(9))
    self.params.append(getattr(self, pname))
</code></pre>
<p>Also, I use <code>register_buffer</code> to get the auto conversions when <code>model.cuda()</code> is called, but I really do not care if it goes to <code>state_dict</code></p>

karl-schulz: <p>Hey guys!</p>
<p>I think what you are looking for is now available as <code>Module.buffers()</code>. It returns an iterator over the registered buffers.</p>

JohnHerry: <p>Good news.  Is there any support for dict? I want to store a dict in the buffers, so that I can access data by Key.</p>
Prunning + Adam

SofiaCP: <p>Hi,</p>
<p>I am pruning the last (linear) layer of my model. However, since I am using Adam, the parameters that were pruned still get updated with momentum. After pruning the weight matrix of my<code> nn.Linear()</code> layer, a percentage of the parameters in the matrix are zero. I tried setting those zero weights to “None” to avoid updates but then I get an error <code>can't assign a NoneType to a torch.cuda.FloatTensor</code>.</p>
<p>How can I avoid this behavior?</p>
<p>Thanks</p>

KFrank: <p>Hi Sofia!</p>
<aside class="quote no-group" data-username="SofiaCP" data-post="1" data-topic="153486" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sofiacp/48/44302_2.png" class="avatar"> SofiaCP:</div>
<blockquote>
<p>the parameters that were pruned still get updated with momentum.</p>
</blockquote>
</aside>
<p>As I understand it, you have pruned some elements of your <code>weight</code><br>
tensor, which is to say, you have set those elements to zero.  Now<br>
you want to “freeze” those elements at zero, even as you continue<br>
to train.</p>
<p>The safest way to freeze just certain elements of a tensor is to restore<br>
them to their desired values after performing the optimization step.  In<br>
your case, you could multiply them inplace by a mask tensor:</p>
<pre data-code-wrap="python"><code class="lang-python"># perform pruning
mask = linear_layer.weight != 0   # pruned elements are flagged with False
# run training forward pass
opt.zero_grad()
loss.backward()
opt.step()   # updates all elements of linear_layer.weight
with torch.no_grad():
    linear_layer.weight.mul_ (mask)   # restores pruned elements to zero
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

SofiaCP: <p>Dear KFrank,</p>
<p>Thanks for your reply. That is exactly the solution I came up with. I was just hoping there was some better solution because I am unsure about the behavior of this option with adam.</p>
<p>Thank you!</p>
RuntimeError: view size is not compatible with input tensor's size

mathwseg: <p>I tried to concatenate two models using the following code but got<br>
<code>RuntimeError: view size is not compatible with input tensor's size and stride (at least one dimension spans across two contiguous subspaces). Use .reshape(...) instead.</code><br>
the code is</p>
<pre><code class="lang-auto">class MyEnsemble(nn.Module):
    def __init__(self, modelA, modelB):
        super(MyEnsemble, self).__init__()
        self.modelA = modelA
        self.modelB = modelB
        self.modelA.head = nn.Identity()
        self.modelB.head = nn.Identity()        
    def forward(self, x):
        x1 = self.modelA(x.clone()) 
        x1 = x1.view(x1.size(0), -1)
        x2 = self.modelB(x)
        x2 = x2.view(x2.size(0), -1)
        x = torch.cat((x1, x2), dim=1)
        gc.collect()
        x=torch.cuda.empty_cache()           
        return x
</code></pre>
<p>i tried to replace <code>view</code> with <code>reshape</code> but got another error in the method i created to extract the features in</p>
<pre><code class="lang-auto">    vec1=vec.detach().cpu().numpy()
AttributeError: 'NoneType' object has no attribute 'detach'
</code></pre>
<p>the code is</p>
<pre><code class="lang-auto"> image= load_img(filename, target_size=(224, 224,3))
 vec=model(preprocess(image).unsqueeze(0).cuda())
 vec1=vec.detach().cpu().numpy()
</code></pre>

eqy: <p>I think the issue is that <code>x</code> should be the output of the model in the <code>forward</code> method rather than the return value of <code>torch.cuda.empty_cache()</code> (which isn’t expected to return anything).</p>
Doubts about nn.Transformer() module

SHRAMAN_PRAMANICK: <p>Hi, I am using nn.Transformer() for the first time, and I have a basic doubt about it. I understand that both the src and tgt parameters have shape (batch_size, sequence_length, model_dim). I am working on a video frame scoring task, where I want to score every frame of a video using a transformer. I am extracting the frame-level features using a pre-trained ResNet module. Hence, for my case, the src has shape (batch_size, sequence_length, 1024) and the tgt has shape (batch_size, sequence_length, 1). Should I use two separate linear layers for src and tgt to transform {1024 to d_model} and {1 to d_model} respectively, or the src and tgt can have different dimensions?</p>
How to load large datasets that cannot fit into memory

kyy: <p>I have read this tutorial regarding how to directly load images from files: <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Writing Custom Datasets, DataLoaders and Transforms — PyTorch Tutorials 1.11.0+cu102 documentation</a>, but my use case is not exactly the same, in that each image can be easily saved as an individual file, while for my dataset (which is a 4D tensor of shape 1000000 * 100 * 10 * 50) it is not feasible to save every single data point to a file.  It is possible to chunk the data into several (e.g. 10) files, but it is difficult to do “full shuffling” in this way (“full shuffling” means randomly selecting data points from entire dataset, not just a few chunks).  I am wondering if there are any examples to deal with this use case?</p>
<p>Thank you!</p>

JuanFMontesinos: <p>Just use numpy memory map arrays.</p>

ejguan: <p>You might want to explore <a href="https://github.com/pytorch/data" rel="noopener nofollow ugc">TorchData</a> using iterative DataPipe to achieve loading data lazily. We have build in <code>shuffle</code> operation, which you can specify shuffle_buffer for the streaming case.<br>
We are going to release 0.4.0 with pytorch 1.12 in the coming weeks. Within this release, DataPipe becomes fully BC with DataLoader in terms of shuffle determinism and automatic sharding</p>
Parallel optimization sub-task

spyroot: <p>Hi Folks,</p>
<p>I have never done it. If you call another torch module, let’s say that module computes forward and backward passes, and it has its own optimization sub-routine. Let’s say that a task can be seen as a parallel task. How can you do that becase? Becase secondary backward step, I think that will trigger the entire backward pass.</p>
<p>For example, in my case, inside a training loop, I have a secondary computation task solver Ax=b (oversimplified example). So for the main optimization, i.e., inside the training loop, (for the sake of simplicity) we solve Ax=b, and it does not influence gradient flow back to the main optimizer.</p>
<p>Pseudocode.</p>
<p>y=model(x)<br>
loss() →  let say inside a loss, it computes backward() somewhere here.-&gt; the oput of that not s.t<br>
backward()</p>
Understanding MultiheadAttention parameters

Robert_Sim: <p>I’m converting some homegrown Keras code for attention to pytorch.  In principle I can just use the MultiheadAttention module.  The Keras code explicitly defines the weight matrices K, Q, and V. In the torch module, there are member attributes k_proj_weight, q_proj_weight, etc but these are initialized to None, and if I iterate through named_parameters() there are only in_proj_weight and out_proj_weight.  What is the relationship between these matrices and K/Q/V?</p>

ptrblck: <p>I think it’s an optimization as can be seen <a href="https://github.com/pytorch/pytorch/blob/1f819ee965894b8332cb364a67c91855c91c9dcc/torch/nn/modules/activation.py#L948-L965">here</a>:</p>
<pre><code class="lang-python">        self._qkv_same_embed_dim = self.kdim == embed_dim and self.vdim == embed_dim

        self.num_heads = num_heads
        self.dropout = dropout
        self.batch_first = batch_first
        self.head_dim = embed_dim // num_heads
        assert self.head_dim * num_heads == self.embed_dim, "embed_dim must be divisible by num_heads"

        if not self._qkv_same_embed_dim:
            self.q_proj_weight = Parameter(torch.empty((embed_dim, embed_dim), **factory_kwargs))
            self.k_proj_weight = Parameter(torch.empty((embed_dim, self.kdim), **factory_kwargs))
            self.v_proj_weight = Parameter(torch.empty((embed_dim, self.vdim), **factory_kwargs))
            self.register_parameter('in_proj_weight', None)
        else:
            self.in_proj_weight = Parameter(torch.empty((3 * embed_dim, embed_dim), **factory_kwargs))
            self.register_parameter('q_proj_weight', None)
            self.register_parameter('k_proj_weight', None)
            self.register_parameter('v_proj_weight', None)
</code></pre>
<p>If the <code>kdim</code> and <code>vdim</code> are equal to <code>embed_dim</code>, then <code>in_proj_weight</code> will increase its size by <code>3</code> and should then contain these parameters if I’m reading the code right.</p>
How to calculate the Jacobian Matrix of a GAN?

zwj: <p>I am wondering if I have a  pretrained generative, how to calculate the jacobian matrix of the output w.r.t. the input?</p>

AlphaBetaGamma96: <p>Have you read <a href="https://pytorch.org/docs/stable/generated/torch.autograd.functional.jacobian.html" class="inline-onebox" rel="noopener nofollow ugc">torch.autograd.functional.jacobian — PyTorch 1.11.0 documentation</a> ?</p>
How to choose the architecture of a GAN model?

Dalal: <p>Hello everyone,<br>
I designed a GAN architecture for a specific problem (dermoscopy images) and it is not working. How to choose the right architecture? why kind of architectures you suggest? I tried simple CNN for both the generator and discriminator. I also tried dense blocks for the generator and Patch-GAN for the discrimnator.</p>
<p>Thanks in advance</p>
Visual trace for nn.Module forward call

Vladimir_Blagojevic: <p>Hi everyone,</p>
<p>Do you know of any visualizers/debuggers that display the visual trace of all forward calls in a module along with their results (e.g. result tensor shape for each forward call)? Consider a complex instance of nn.Module like BertModel. I’d love to pass input to this module and get a visual trace (a simple printout would be a great start) of all forward calls executed and their corresponding tensor shapes.</p>
<p>I can imagine someone already made a similar debugger using hooks - i.e. using register_forward_hook recursively attached to every layer in the module.</p>