akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Model only fits for small dataset

username: nicozhou
cooked: <p>I am doing a regression problem, and right now facing a problem where my model can only be trained with small amount of data, but when I increase the amount of data, the network does not learn(loss does not decreasing).</p>
<p>Does anyone ever meet the same problem as me?</p>

username: thecho7
cooked: <p>I don’t know how small the change of loss but we can typically observe that with a small model.<br>
It means your model is not enough big to fit the larger dataset.</p>
<p>Scale up your model and show how it changes</p>

username: nicozhou
cooked: <p>the loss I used is L1 Loss, and it started from around 0.4 and can only decrease to 0.3</p>

username: thecho7
cooked: <p>How small the loss value cannot exactly explain how good your model is.<br>
If you use <code>nn.L1Loss</code>, check whether you are averaging the loss by the size of batch.</p>
<p>Anyway, scale up your model first.</p>

username: nicozhou
cooked: <p>yeah, I mean it is supposed to be decreasing to 0.01. But it could not. By the way, scaling model is like adding more parameters?</p>

username: thecho7
cooked: <p>Yes, stacking more layers or somehow you already know</p>
Batch normalization does not what it states

username: AndreasH
cooked: <p>Hey everybody,</p>
<p>I have a question.<br>
I wanted to understand what exactly is done within batch normalization. In the documentation it is stated, that BatchNorm1d performs batch normalization as stated in this paper</p><aside class="onebox pdf" data-onebox-src="https://arxiv.org/pdf/1502.03167.pdf">
  <header class="source">

      <a href="https://arxiv.org/pdf/1502.03167.pdf" target="_blank" rel="noopener nofollow ugc">arxiv.org</a>
  </header>

  <article class="onebox-body">
    <a href="https://arxiv.org/pdf/1502.03167.pdf" target="_blank" rel="noopener nofollow ugc"><span class="pdf-onebox-logo"></span></a>

<h3><a href="https://arxiv.org/pdf/1502.03167.pdf" target="_blank" rel="noopener nofollow ugc">1502.03167.pdf</a></h3>

  <p class="filesize">169.48 KB</p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p>
I do, however, think that this is not true.<br>
This is what BatchNorm1d does (using default parameters, I tested and it results in the correct output):</p>
<p>Input: Tensor X with shape (N,C,L)</p>
<p>mean_X = torch.mean(X, dim=(0,2), keepdims=True)<br>
var_X = torch.mean((X-mean_X)*(x-mean_X),dim=(0,2), keepdims=True)<br>
BN = (x-mean_x)/np.sqrt(var_x+1e-5)</p>
<p>Output: BN</p>
<p>So the mean and variance are computed over all input axes (accept for C which allows for different channels). In the paper, however, they state that mean and variance are computed over the batch size, which I think corresponds to setting dim=0 above.<br>
Can someone confirm or explain.</p>
How to diagnose possible CPU memory leak?

username: huahuanZ
cooked: <p>The CPU memory just increases as my program running. Sorry that the codes is internally used so I can’t paste it. Here’s some information:</p>
<ol>
<li>My program runs in inference mode, I set <code>torch.no_grad()</code> context. So it can’t be the computation graph memory “leak”.</li>
<li>The neural networks are small nn.LSTM and nn.Linear models. When I set batch size to small value (like 4 or 8), the memory usage is stable. But when batch size increase to larger (32, 128, …), the memory usage just increases over iterations.</li>
<li>I use <code>psutil</code> to get the memory usage. I also try <code>tracemalloc</code> and torch profile, but the letter two tools can’t tell where the leak lies.</li>
<li>My program can run on CPU or GPU device. The possible memory leak only occurs when using CPU.</li>
</ol>
<p>The iteration loop is like</p>
<pre><code class="lang-auto">
    with torch.no_grad():
        for minibatch in testloader:
            my_function(minibatch)
            mt = psutil.Process(os.getpid()).memory_info().rss
            print(mt)
</code></pre>
<p>I know without the detailed codes it’s hard to find the cause. Any suggestion is welcome .</p>

username: ptrblck
cooked: <p>As a general debugging advice I would probably try to remove utilities (e.g. checkpointing, quantization, etc.) if used and check if the effect is still visible. Once you could narrow it down to a small subset of your use case you might be able to post a proxy workload which we could use to help in isolating the issue.</p>
Parametrization of bidirectional LSTM

username: sigma_x
cooked: <p>From this sketch I’m trying to understand the parametrization of bLSTM. Specifically, exactly how the outputs of forward and backward outputs are combined before the non-linearity. I didn’t find it in the source code. The connections are in the blue/orange box.</p>
<p>I’d say it’s something like this: <img src="https://discuss.pytorch.org/uploads/default/original/3X/3/7/370619d0a0cc0854a7e8f79264746c23a1e6d1b3.png" alt="lagrida_latex_editor" data-base62-sha1="7QLkzGkYXjHDGS6qA1kkYYhTxqb" width="255" height="23">, with both W and b being parameters and bias for the corresponding direction, but I’m not too sure. Could someone verify it perhaps?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/4/34b90de7962ead49de23d901fac44f1da4789571.png" data-download-href="https://discuss.pytorch.org/uploads/default/34b90de7962ead49de23d901fac44f1da4789571" title="Screenshot (421)"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/4/34b90de7962ead49de23d901fac44f1da4789571_2_690x325.png" alt="Screenshot (421)" data-base62-sha1="7wpiWmjBCPxvrKCZpXmFMOJ6AsV" width="690" height="325" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/4/34b90de7962ead49de23d901fac44f1da4789571_2_690x325.png, https://discuss.pytorch.org/uploads/default/original/3X/3/4/34b90de7962ead49de23d901fac44f1da4789571.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/3/4/34b90de7962ead49de23d901fac44f1da4789571.png 2x" data-dominant-color="E3E6E0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot (421)</span><span class="informations">793×374 78.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: vdw
cooked: <p><code>nn.LSTM</code> with <code>bidirectional=True</code> does not on it’s own combine the results of the forward and backward pass, this is up to you to decide how you want to do it.</p>
<p>You can see this in the shape of the output of <code>nn.LSTM</code>. If you check <a href="https://pytorch.org/docs/stable/generated/torch.nn.LSTM.html" rel="noopener nofollow ugc">the docs,</a> both the output and the hidden state(s) all include <code>D</code> as the number of directions (1 or 2) in the shape.</p>
Why is torch.mm for sparse CSR tensor slower in 1.10.2 than 1.9.1?

username: Stanibor
cooked: <p>Hi, I have noticed that using <code>torch.mm</code> operation using tensor with <code>sparse_csr</code> layout is significantly slower in pytorch 1.10.2 than in pytorch 1.9.1.</p>
<p>Here is a small code sample I’ve used to test it:</p>
<pre><code class="lang-auto">import timeit

print(torch.__version__)

torch.manual_seed(1233)
crows = torch.randint(0, 50, (1000000,)).cumsum(0).int()
crows[0] = 0
cols = torch.randint(0, 100000, (crows[-1],)).sort()[0].int()
mock_vals = torch.rand_like(cols, dtype=torch.float)

csr_mat = torch._sparse_csr_tensor(crow_indices = crows, col_indices=cols, values=mock_vals)
dense_mat = torch.rand(csr_mat.shape[-1], 20)

print(timeit.timeit("torch.mm(csr_mat, dense_mat)", globals=globals(), number=100))
</code></pre>
<p>In pytorch version 1.9.1, this code yields:</p>
<pre><code class="lang-auto">1.9.1+cpu
1.9244596790522337
</code></pre>
<p>In pytorch version 1.10.2 (same machine), almost same code yields (except this time I use <code>torch.sparse_csr_tensor</code> instead of <code>torch._sparse_csr_tensor</code>):</p>
<pre><code class="lang-auto">1.10.2+cpu
5.861249879002571
</code></pre>
<p>As you can see there is a significant difference in times between those versions. I would like to upgrade to pytorch 1.10.2 but I would also like to use sparse_csr matrix format as it provides  speed up in my project. However I’d expect that matrix multiplication would be faster for newer pytorch version. Have I overlooked something?</p>

username: ptrblck
cooked: <p>I guess the implementation might depend on the used CPU. In my setup I get:</p>
<ul>
<li>~1.85s in <code>1.9.1</code>
</li>
<li>~2.02s in <code>1.11.0.dev20220108</code>
</li>
</ul>
<p>which shows a regression, but isn’t as large as your observation.</p>

username: Stanibor
cooked: <p>I will check if 1.11.0 fixes it, but in my project I need to depend on stable version, so I think I have to investigate the difference.</p>
<p>On my setup I have <code>2x Intel(R) Xeon(R) E5-2696 v4</code> with <code>AVX2</code> support.<br>
Does sparse <code>torch.mm</code> method have different implementation between 1.9.1 and 1.10.2 or perhaps it is some difference in default variables / build configuration?</p>

username: ptrblck
cooked: <p>You could use <code>print(torch.__config__.show())</code> to check if the same libraries are installed (e.g. MKL etc.).</p>
Convolution Layers no Bias with batchnorm

username: nicozhou
cooked: <p>hi, i just wondering is it better to use bias=False when using batchnorm? Will our network break if not using disabling bias ?</p>

username: Yuv
cooked: <p>It’s very hard to answer your question without more details about the problem you refer to.<br>
Generally removing the bias may harm the performance but you can check it by trying.</p>

username: nicozhou
cooked: <p>Yeah, so I was really confused, as some may say that :</p>
<p>Note that, since we normalize Wu+b, the bias b can be ignored since its effect will be canceled by the subsequent mean subtraction (the role of the bias is subsumed by β in Alg. 1).</p>
<p>This is what I got from batchnorm paper.</p>

username: Yuv
cooked: <p>When using batch normalization you have a learnable parameter β which have the same role as bias when not using batch normalization.<br>
Adding bias term to Wx will result in a new term when averaging in the batch normalization algorithm but that term would vanish because the subsequent mean subtraction, and that why they ignore the biases and this is the purpose of the β learnable parameter.</p>

username: nicozhou
cooked: <p>So, it is safe to conclude that it is better to disable bias if we use batchnorm?</p>

username: Eta_C
cooked: <blockquote>
<p>the bias b can be ignored since its effect will be canceled by the subsequent mean subtraction</p>
</blockquote>
<pre data-code-wrap="python"><code class="lang-python"># Convolution
y = w * x + b 

# Batch Normalization (without momentum updating)
z = (y - E(y_i)) / STD(y_i)

# note that we have
y - E(y_i) = w * x + b - E(w * x_i + b) = w * x - E(w * x_i)
STD(y_i) = Sqrt(Sigma(y - E(y_i)) + epsilon)
</code></pre>
<blockquote>
<p>So, it is safe to conclude that it is better to disable bias if we use batchnorm?</p>
</blockquote>
<p>If you use BatchNorm <strong>after convolution</strong>, the answer is YES.</p>
Why does autograd track things like `torch.cat`?

username: Wesley_Neill
cooked: <p>I’m coding a timeseries autoencoder that reconstructs the elements of the input reverse order, one at a time. I’m using the <code>torch.cat</code> operation to concatenate each element back into sequence form.</p>
<pre><code class="lang-auto">        # reconstruct remaining elements
        for i in range(1, self.seq_len):
            hs_i, cs_i = self.cell(x_i, (hs_i, cs_i))
            x_i = self.dense(hs_i)
            x = torch.cat([x, x_i])
        
        return x
        
</code></pre>
<p>I noticed that the output has <code>grad_fn=&lt;CatBackward&gt;</code>.</p>
<p>I was wondering why the autograd system would track things like collecting elements into a sequence? Will this mess with my backprop when it’s time to train?</p>

username: ptrblck
cooked: <p><code>torch.cat</code> will be tracked by Autograd and the backward pass will pass through this operation into the incoming tensors. This is useful in case you want to e.g. concatenate outputs of two different models, calculate the loss, and compute the gradients for both models.<br>
If you don’t want to track this operation, you could <code>.detach()</code> the input tensors.</p>

username: Wesley_Neill
cooked: <p>Okay, that makes sense! So in my case I’ve got an LSTM cell in a for-loop and I’m concatenating the results of each pass into my final output sequence… back prop won’t go through that tensor unless the concatenation is tracked… is that a correct summary of what you are saying?</p>

username: ptrblck
cooked: <p>Yes and by default the <code>torch.cat</code> operation is tracked, so based on your description you would append the outputs of the model with the complete computation graph and would then backpropagate through all these outputs and the model.</p>

username: mac
cooked: <p>I understand that torch.cat is tracked by Autograd and that the backward pass will “pass through” these operations into the incoming tensors. Is there a way using the “loss.grad_fn.next_functions” commands to reach the leave nodes to verify this?</p>
Will pytorch support real ZeRO-1 natively?

username: Bean
cooked: <p>Hi, I found that the <a href="https://pytorch.org/tutorials/recipes/zero_redundancy_optimizer.html#/" rel="noopener nofollow ugc">torch.optim.ZeroRedundancyOptimizer</a> does not implement the real ZeRO-1 of DeepSpeed.</p>
<p>the training process of torch.optim.ZeroRedundancyOptimizer is as below (correct me if I’m wrong):</p>
<ol>
<li>Calculate grad of all layers on all devices, and do the AllReduce for all grads</li>
<li>Shard the total parameters by layer into different devices, and do the optimizing of Adam for corresponding sharded layers on certain devices.</li>
<li>Broadcast the results of Adam optimizing, which are parameters, to the other devices.</li>
</ol>
<p>Thus, the ZeroRedundancyOptimizer got extra communication of broadcast than the baseline.</p>
<p>But as I know, the steps of origin ZeRO-1 from deepspeed are:</p>
<ol>
<li>Calculate grad of all layers on all devices, and do the ReduceScatter for all grads, which result in a sharded grad tensor for each parameter tensor on every device.</li>
<li>Optimize the corresponding parameters with sharded grad tensors</li>
<li>Run an AllGather communication for optimized parameters</li>
</ol>
<p>Compared to baseline, it does not have extra communication.</p>
<p>So why does PyTorch implement the former design instead of the latter? Is there a road map for migration to the latter implementation?</p>

username: cbalioglu
cooked: <p>cc <a class="mention" href="/u/mrshenli">@mrshenli</a> who is the SME.</p>

username: Yanli_Zhao
cooked: <p><a class="mention" href="/u/bean">@Bean</a> what you mentioned is like DeepSpeed Zero2 algorithm. We are releasing a new API called torch.distributed.fsdp.FullyShardedDataParallel in PyTorch 1.11, which is similar to Zero3. We are also working in progress to make it easily configed as an algorithm similar to Zero 2 as well, basically shard optimizer states and grad only, and it quires all gather in the forward pass and reduce scatter in the backward pass.</p>

username: Bean
cooked: <p>Thanks for your reply, looking forward to torch 1.11 FSDP API!</p>
How to randomly choose a subset of classes every batch and sample from those chosen classes?

username: EChu
cooked: <p>Hi all,</p>
<p>I want to get samples from a random subset of classes and sample a batch of examples for this iteration. A running example would be, for training Cifar100, for each iteration in a single epoch, I want to get a minibatch of 256 images but consisting only of randomly chosen 50 of 100 classes w/o replacement, so each class would have around 5 examples. And for the next minibatch, I want another different 50 classes.</p>
<p>I am thinking of using a custom sampler to do this and get indices of examples of the same classes and then sample from them, but it would be costly.</p>
<p>Any thoughts would be appreciated!</p>
Log_softmax(): argument 'input' (position 1) must be Tensor, not Linear

username: Morphinepill
cooked: <p>I’m new to pytorch, how is my inputs Linear? when I print their type they are &lt;class ‘torch.Tensor’&gt;</p>
<pre><code class="lang-auto">class Network(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(150528, 250)
        self.fc2 = nn.Linear(250, 64)
        self.fc3 = nn.Linear(64, 10)

        
    def forward(self, x):
        x = x.view(x.shape[0], -1)
        
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))

        
        x = F.log_softmax(self.fc3, dim=1)
        return x
        
model = Network()
model
</code></pre>
<pre><code class="lang-auto">epochs = 10
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model.to(device)
steps = 0
running_loss = 0
print_every = 5

for epoch in range(epochs):
    for inputs, labels in trainloader:
        steps += 1
        # Move input and label tensors to the default device
        inputs, labels = inputs.to(device), labels.to(device)
        
        optimizer.zero_grad()
        print(type(inputs))
        logps = model.forward(inputs)
        loss = criterion(logps, labels)
        output, x = model(inputs)
        loss = criterion(output, labels)
        loss.backward()
        optimizer.step()

        running_loss += loss.item()
</code></pre>

username: suraj.pt
cooked: <p>Your input to <code>log_softmax</code> is <code>fc3</code>which is a <code>nn.Linear(64, 10)</code>. I presume you want to pass the output of <code>fc3</code> instead.</p>
<p>Replace <code>x = F.log_softmax(self.fc3, dim=1)</code> with <code>x = F.log_softmax(self.fc3(x), dim=1)</code></p>
<p>Cheers!</p>
nn.DataParallel multigpu: "...found two devices cuda:0 and cuda:1!"

username: avalon1511
cooked: <p>I have a model <code>net</code> that I am trying to train on multiple GPUs. I used <code>DataParallel</code> to enforce this, however during training when I run a forward pass through this model I am getting the following error</p>
<pre><code class="lang-auto">RuntimeError: Expected all tensors to be on the same device, 
but found at least two devices, cuda:1 and cuda:0!
</code></pre>
<p>the piece of code that is causing this error is</p>
<pre><code class="lang-auto"> codes = codes + self.latent_avg.repeat(codes.shape[0], 1, 1)
</code></pre>
<p>Codes is obtained as</p>
<pre><code class="lang-auto">codes = net.encoder(input)
out = net.decoder(codes)
</code></pre>
<p>where <code>net</code> is an encoder-decoder network</p>
<p>Not sure how to fix this error, any help is appreciated!</p>

username: suraj.pt
cooked: <p>Can you check if <code>latent_avg</code> is on a different device than <code>codes</code>? To run DP, <code>net</code> needs to be on cuda:0 =&gt; <code>codes</code> would be on cuda:0 so presumably self.latent_avg is on cuda:1 for you to get this error.</p>
<pre><code class="lang-auto">if self.latent_avg.device.index != 0:
  self.latent_avg = self.latent_avg.cuda(0)

codes = codes + self.latent_avg.repeat(codes.shape[0], 1, 1)
</code></pre>
RuntimeError: One of the differentiated Tensors appears to not have been used in the graph. Set allow_unused=True if this is the desired behavior. srun: error: cn31: task 0: Exited with exit code 1

username: Mari
cooked: <p>How can i solve this error?<br>
RuntimeError: One of the differentiated Tensors appears to not have been used in the graph. Set allow_unused=True if this is the desired behavior.<br>
srun: error: cn31: task 0: Exited with exit code 1</p>
<pre><code class="lang-auto">for batch_idx, (x_in,y_in) in enumerate(dataloader):
                    net2_u.zero_grad()
                    net2_v.zero_grad()
                    net2_p.zero_grad()
                    loss_eqn = criterion(x_in,y_in)
                    loss_bc = Loss_BC(xb,yb,ub,vb,x,y)
                    loss = loss_eqn + Lambda_BC* loss_bc
                    # loss.backward(create_graph=True)
                    loss.backward()
                    optimizer_u.step() 
                    optimizer_v.step() 
                    optimizer_p.step()
                    loss_eqn_a =loss_eqn.detach().cpu().numpy()
                    loss_eqn_n += loss_eqn_a
                    loss_bc_a= loss_bc.detach().cpu().numpy()
                    loss_bc_n += loss_bc_a  
                    n += 1         
                    
                    
                    
                    params, gradsH = get_params_grad(net2_u) #get model parameters and corresponding gradients
                
                    params[2].requires_grad = True
                    gradsH[2].requires_grad = True
                    tmp_num_data = x_in.size(0)
                    
                    v = [torch.randn(p.size()).to(device) for p in params]  # generate random vector
                    v = normalization(v)  # normalize the vector
                    THv = [torch.zeros(p.size()) for p in params[2]]
                    Hv = torch.autograd.grad(gradsH[2],params[2],grad_outputs=torch.ones_like(params[2]), retain_graph = True ,only_inputs=True)[0]
                  
</code></pre>
<p>If i set allow_unused=True it prints none.</p>
Is it possible to prune on a pretrained model using Pytorch Pruning API?

username: ehza
cooked: <pre><code class="lang-python">
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

class LeNet(nn.Module):
    def __init__(self):
        super(LeNet, self).__init__()
        # 1 input image channel, 6 output channels, 3x3 square conv kernel
        self.conv1 = nn.Conv2d(1, 6, 3)
        self.conv2 = nn.Conv2d(6, 16, 3)
        self.fc1 = nn.Linear(16 * 5 * 5, 120)  # 5x5 image dimension
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        x = F.max_pool2d(F.relu(self.conv1(x)), (2, 2))
        x = F.max_pool2d(F.relu(self.conv2(x)), 2)
        x = x.view(-1, int(x.nelement() / x.shape[0]))
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

model = LeNet()

parameters_to_prune = (
    (model.conv1, 'weight'),
    (model.conv2, 'weight'),
    (model.fc1, 'weight'),
    (model.fc2, 'weight'),
    (model.fc3, 'weight'),
)

prune.global_unstructured(
    parameters_to_prune,
    pruning_method=prune.L1Unstructured,
    amount=0.2,
)
</code></pre>
<p><a href="https://pytorch.org/tutorials/intermediate/pruning_tutorial.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/intermediate/pruning_tutorial.html</a></p>
<p>In this tutorial, the author discussed various types of pruning (structured, unstructured, local, global), but all those techniques were employed on a vanilla LeNet model. Is it possible to prune on a <strong>pretrained</strong> model using those builtin API. [<a href="https://pytorch.org/docs/stable/search.html?q=prune&amp;check_keywords=yes&amp;area=default" class="inline-onebox" rel="noopener nofollow ugc">Search — PyTorch 1.9.0 documentation</a>](Pytorch Pruning API)</p>
<p>If yes, then how?</p>
<p>For example,<br>
<code>models.resnet50(pretrained = True)</code></p>
<p><em>Do I need to assign<code> pretrained = False</code>?</em><br>
or<br>
Do I need to write those pretrained models from scratch to apply pruning techniques? Or something else?</p>

username: goksinan
cooked: <p>You can iterate over the modules of the pretrained model and prune each module one by one. The same tutorial has this example script:</p>
<pre><code class="lang-python">new_model = LeNet()  # Replace with your model
for name, module in new_model.named_modules():
    # prune 20% of connections in all 2D-conv layers
    if isinstance(module, torch.nn.Conv2d):
        prune.l1_unstructured(module, name='weight', amount=0.2)
    # prune 40% of connections in all linear layers
    elif isinstance(module, torch.nn.Linear):
        prune.l1_unstructured(module, name='weight', amount=0.4)

print(dict(new_model.named_buffers()).keys())  # to verify that all masks exist
</code></pre>
<p>However, I am not sure if you can gain anything (such as less memory, faster inference) with pruning, especially if you use the unstructured one. I had high expectations in the beginning, but was disappointed with what I got.</p>
Generating histogram feature of 2D tensor from 3D Tensor feature set

username: ammar_naich
cooked: <p>I have a 3D tensor of dimensions (3,4 7) where each element in 2-dim(4) has 7 attributes.<br>
What I want is to take the 4th attribute of all 4 elements and to calculate the histogram having 3<br>
hist values and store those values only. And ending up with a 2D tensor of shape (3,4).<br>
I have a small toy example for the task that I am working on. My solution ends up with a Tensor which has shape (1,3). Any hint or guidance will be appreciated</p>
<pre><code class="lang-auto">import torch
torch.manual_seed(1)
feature = torch.randint(1, 50, (3, 4,7))
feature.type(torch.FloatTensor)
attrbute_val = feature[:,:,3:4]
print(attrbute_val.shape)
print(attrbute_val)
histogram_feature = torch.histc(torch.tensor(attrbute_val,dtype=torch.float32), bins=3, min=1, max=50)
print("histogram_feature",histogram_feature)
</code></pre>

username: ammar_naich
cooked: <p>This is want i have come up with a naive solution</p>
<pre><code class="lang-auto">attrbute_val = feature[:,:,3:4]
print(attrbute_val.shape)
print(attrbute_val[:,:,0])

final_feature = np.zeros((3,3))
shape = attrbute_val[:,:,0].shape
for row in range(shape[0]):
  print("element wise features",attrbute_val[:,:,0][row])
  hist,_= np.histogram(attrbute_val[:,:,0][row], bins=3)
  print(row,",hist values",hist)
  final_feature[row,:] = hist

print("final feature shape", final_feature.shape)
print("final feature",final_feature)
</code></pre>
<p>I wish PyTorch has any way to apply customer function on dimensions</p>
Multiple operations in parallel on single GPU

username: Tushar_Gautam
cooked: <p>Hi</p>
<p>I’m working on a problem involving sensitivity analysis and hoping to use pytorch and it’s in-build operations instead of coding everything from scratch in CUDA.</p>
<p>I’ve a small example code (using NN example as most people would be familiar with this here) as follows, where computations involving <code>dZ</code> and <code>dA</code> are independent of that of <code>Z</code> and <code>A</code>.</p>
<pre><code class="lang-python">def sensitive(d_inp, inp, param):
    Z = torch.matmul(inp, param.T)
    dZ = torch.matmul(d_inp, param.T)

    A = torch.tanh(Z)
    dA = torch.unsqueeze(1 - torch.tanh(Z)**2, axis=1) * dZ

    return A, dA
</code></pre>
<p>I want to parallelise  the code such that <code>Z</code> and <code>dZ</code> are computed in parallel, followed by the parallel evaluation of <code>A</code> and <code>dA</code>.</p>
<p>I was looking for solution to this but couldn’t find anything. Hope someone can help me out here.</p>
<p>Thanks</p>

username: huahuanZ
cooked: <p>they might be asynchronously executed invisibly.<br>
Have a look at this note <a href="https://pytorch.org/docs/stable/notes/cuda.html#asynchronous-execution" class="inline-onebox" rel="noopener nofollow ugc">CUDA semantics — PyTorch 1.10 documentation</a></p>

username: Tushar_Gautam
cooked: <p>I just separated the two operations (value and gradient) into two separate functions and timed their execution independently. I don’t think it’s been executed asynchronously as their independent runs adds up to my earlier test.</p>

username: Tushar_Gautam
cooked: <p>Is there a way by which I can be certain that two operations are running in parallel?</p>
<p>I read a recent publication (arXiv pre-print) where they’ve done something similar and casually mentioned that they used operator overloading in PyTorch to facilitate the computations to run in parallel (they must be using inbuilt classes). I don’t understand how something like this can be done.</p>

username: huahuanZ
cooked: <p>Sorry, that’s out of my knowledge <img src="https://discuss.pytorch.org/images/emoji/apple/smiling_face_with_tear.png?v=12" title=":smiling_face_with_tear:" class="emoji" alt=":smiling_face_with_tear:" loading="lazy" width="20" height="20">.</p>
Conditional bach generation

username: Alex_K
cooked: <p>Hi, everyone! I’m a bit stuck with a quite simple problem <img src="https://discuss.pytorch.org/images/emoji/apple/weary.png?v=12" title=":weary:" class="emoji" alt=":weary:" loading="lazy" width="20" height="20"> Say we have a dataset of vectors, where vectors have either 1 or 0 as the last element. I need each batch to have vectors of one type. Therefore  in the dataloader we have to:</p>
<ul>
<li>pick 0 or 1 randomly;</li>
<li>pick a subset of the dataset where vectors end with the number generated in the previous step;</li>
<li>pick a random batch from the subset above;<br>
How would you implement it in PyTorch?<br>
Thanks in advance!</li>
</ul>

username: nivek
cooked: <p>You can implement a custom <code>Dataset</code> with the logic that you described in <code>__getitem__</code> (or an <code>IterableDataset</code> with the logic in <code>__iter__</code>). There is an <a href="https://pytorch.org/tutorials/beginner/basics/data_tutorial.html#creating-a-custom-dataset-for-your-files" rel="noopener nofollow ugc">example of writing a custom <code>Dataset</code></a> in the documentation.</p>

username: Alex_K
cooked: <p>Thanks! Does <code>__getitem__</code> returns the whole batch? Or a single item?</p>

username: eqy
cooked: <p><code>__getitem__</code> should return just a single item (consider that the <code>Dataset</code> is not expected to have any knowledge of what the downstream batch size specified by e.g., a <code>DataLoader</code> would be).</p>

username: Alex_K
cooked: <p>Yes, that is what I was assuming, however, I couldn’t find any source on how to rewrite <code>Dataloader</code> to return batch conditionally</p>

username: nivek
cooked: <p>In theory, you can make <code>__getitem__</code> conditionally return a batch, and then set <code>batch_size = 1</code> for <code>DataLoader</code>, but this is outside the general use case of <code>Dataset</code>/<code>DataLoader</code>.</p>
Deep copy of model weights

username: sofiane
cooked: <p>I’d like to make a deep copy of weights of a model, I find out that the first deep copy may be copying a reference dictionary, but that’s not the purpose of deepcopy. train(30) trains a model for 30 episodes, but this affects the state_dict that I deepcopied before.</p>
<pre><code class="lang-auto">model_sd = deepcopy(network.state_dict())
train(30)
another_model_sd = deepcopy(network.state_dict())
model_sd['l.0.weight'] == another_model_sd['l.0.weight']
</code></pre>
<p>I tried the same thing with deepcopy of network</p>
<pre><code class="lang-auto">copy1 = deepcopy(network)
train(30)
copy2 = deepcopy(network)
copy1.l[0].weight == copy2.l[0].weight
</code></pre>
<p>output for both methods :</p>
<blockquote>
<p>tensor([[True, True, True, True],<br>
[True, True, True, True],<br>
[True, True, True, True],<br>
[True, True, True, True],<br>
…</p>
</blockquote>

username: ptrblck
cooked: <p>I cannot reproduce the issue using:</p>
<pre><code class="lang-python">model = nn.Linear(1, 1)
optimizer = torch.optim.SGD(model.parameters(), lr=1)
sd_ref = copy.deepcopy(model.state_dict())

for _ in range(10):
    optimizer.zero_grad()
    out = model(torch.randn(1, 1))
    out.mean().backward()
    optimizer.step()
    
sd = copy.deepcopy(model.state_dict())

for key1, key2 in zip(sd_ref, sd):
    print((sd_ref[key1] - sd[key2]).abs().max())
# &gt; tensor(1.3497)
#   tensor(10.)
    
print(sd['weight'] == sd_ref['weight'])
# &gt; tensor([False])
print(sd['bias'] == sd_ref['bias'])
# &gt; tensor([False])
</code></pre>
<p>Are you sure the “global” network was changed and not a local object in <code>train</code>?</p>

username: sofiane
cooked: <p>Sorry that was a bug, the training is never done due to a condition I set that is never respected.</p>
How should I change the learning rate for smooth convergence of loss?

username: sprakashdash
cooked: <p>Here is the image of the loss value for actor in TD3. I am using an RNN kind of layer with MLP. The loss seems to do some kind of jiggling after 25k steps and the expected rewards are also not increasing much. How should I decrease the learning rate so that the loss should smoothly converge?<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/0/f/0f5b55a549cad440ca75cd9fa3f94df293e55412.png" alt="actor_loss" data-base62-sha1="2bQQpQMImP4NUEbPZ9miOYfqL0S" width="641" height="359"></p>
Two DDP models on the same GPU

username: mlearn
cooked: <p>Hi all!</p>
<p>I was wondering if I can initialize two DDP models on the same device within the same process group or if this might cause any bugs i.e. regarding the memory allocation.</p>

username: cbalioglu
cooked: <p>I would avoid that since it is an unsupported setting. Any reason for not using two separate process groups?</p>

username: mlearn
cooked: <p>Because I am using the NCCL backends and here <a href="https://pytorch.org/docs/stable/distributed.html#torch.distributed.new_group" rel="noopener nofollow ugc">Pytorch Doc</a> it says:</p>
<p>“Using multiple process groups with the <code>NCCL</code> backend concurrently is not safe and the user should perform explicit synchronization in their application to ensure only one process group is used at a time. This means collectives from one process group should have completed execution on the device (not just enqueued since CUDA execution is async) before collectives from another process group are enqueued. See <a href="https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/usage/communicators.html#using-multiple-nccl-communicators-concurrently" rel="noopener nofollow ugc">Using multiple NCCL communicators concurrently</a> for more details.”</p>
<p>and I am a bit unsure how to introduce explicit synchronisation.</p>

username: mlearn
cooked: <p>I am also trying to implement a GAN, so both process groups should communicate with each other. That’s why I would required subprocess groups, I guess.</p>
Transform 2D embeddings into 3D embeddings

username: AfonsoSalgadoSousa
cooked: <p>Hi. I want to fuse node embeddings from a GNN shaped {[N, 128], where N can vary, with hidden state from an LSTM shaped [2, 32, 128]. To do so, I wanted to add both feature tensors, but they need to be the same shape.<br>
I have something like:</p>
<pre><code class="lang-auto">N = 75
t = torch.rand(N, 128)
t.unsqueeze_(0)
t = t.expand(2,N, 128)
</code></pre>
<p>I do not seem to figure out how to make dimension 1 equal to 32. I am looking for some combination of transforms and learnable layers that can help me make this shape conversion. Thanks in advance for any help you can give.</p>
Feed torch.unique back to inputs

username: jan_biel
cooked: <p>Lets say i have a tensor and apply unique to it</p>
<pre><code class="lang-auto">t1 = torch.tensor([2,2,3,3,5,1,1,1])
u = t1.unique(dim=0, sorted=True, return_counts=True,  return_inverse=True)

-&gt;
(tensor([1, 2, 3, 5]), tensor([1, 1, 2, 2, 3, 0, 0, 0]), tensor([3, 2, 2, 1]))

how can i map the counts back to the original tensor like to
t1 = [2,2,3,3,5,1,1,1]
t2 = [2,2,2,2,1,3,3,3]
I know how to do this with a loop, is there a faster way to do it?
</code></pre>

username: ptrblck
cooked: <p>This should work:</p>
<pre><code class="lang-python">print(u[2][u[1]])
# &gt; tensor([2, 2, 2, 2, 1, 3, 3, 3])
</code></pre>
Asynchronous batch inference

username: zenosama
cooked: <p>Hello,</p>
<p>I want to learn how to enable asynchronous batch inference. As opposed to the common way that samples in a batch are computed (forward) at the same time synchronously within a process, I want to know how to compute (forward) each sample asynchronously in a batch using different processes because my model and data are too special to handle in a process synchronously (e.g., sample lengths in a batch vary too much)</p>
<p>My current idea is using torch.multiprocessing as one solution. However, I don’t know if it is the appropriate way. And is there any other solutions?</p>
<p>Thanks</p>

username: cbalioglu
cooked: <p>Have you checked TorchServe?</p>

username: zenosama
cooked: <aside class="quote no-group" data-username="cbalioglu" data-post="2" data-topic="144731">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/cbalioglu/48/33187_2.png" class="avatar"> cbalioglu:</div>
<blockquote>
<p>TorchServe</p>
</blockquote>
</aside>
<p>Hi Can, thanks for your reply. Yes, but my case is more complicated. For example, my model is an enc-dec model. I hope samples in a batch are encoded synchronously but to decode asynchronously. I didn’t find TorchServe can do kind of this and don’t think a framework can support it without any customization work.</p>

username: marksaroufim
cooked: <p>Hi <a class="mention" href="/u/zenosama">@zenosama</a> we should be able to handle this with our workflows feature - let me know if this helps <a href="https://github.com/pytorch/serve/tree/master/examples/Workflows" class="inline-onebox" rel="noopener nofollow ugc">serve/examples/Workflows at master · pytorch/serve · GitHub</a></p>

username: zenosama
cooked: <p>Thanks Mark for your help. Will definitely look into the examples to see how to work in my case.</p>
<p>Thank you</p>
Training using this Data loader and data set is too slow

username: Oussama_Bouldjedri
cooked: <p>I am working on a data set that I stored in pickle extention the data set is set as this :<br>
train data:<br>
classe1:<br>
instence1.pickle<br>
instance2.pickle<br>
…<br>
classe2:<br>
…</p>
<p>I think I am having a big delay in training because of this :</p>
<pre><code>class CustomDataset_train(Dataset):
   def __init__(self):


     self.data_path = dataset_dir_train
     file_list = glob.glob(self.data_path + "*")
     self.data = []
     for class_path in file_list:
          class_name = class_path.split("\\")[-1]
    
          for data_path in glob.glob(class_path + "/*.pickle"):
        
               self.data.append([data_path, class_name])
      

     self.class_map = {'class1 ': 0,
                    'class2':1,
                   ......
                  
                  
                  }


   def __len__(self):

       return len(self.data)
   def __getitem__(self, idx):

     data_path, class_name = self.data[idx]

    data = pickle.load(file=open(data_path, "rb"))


    class_id = self.class_map[class_name]


    fused_tensor_data=torch.from_numpy(data)





    return (fused_tensor_data, class_id)
</code></pre>
<p>the data is stored as a numpy array with the shape of 250,164,3 in every pickle file . is there a solution to make it faster ? something like DataFolder</p>
[HELP] Torch.onnx.export can not export onnx model

username: novice96
cooked: <p>Hi , I’m trying to export my trained model that in .pt format to ONNX but after I execute the script the program output bunch of logging stuff like in the pic below and the converted onnx format seems not appear in my local disk  I don’t know what I got wrong here?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/7/d7e847f6d3ebd870c66a08aa173007e7f252e72c.png" data-download-href="https://discuss.pytorch.org/uploads/default/d7e847f6d3ebd870c66a08aa173007e7f252e72c" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/7/d7e847f6d3ebd870c66a08aa173007e7f252e72c_2_690x429.png" alt="image" data-base62-sha1="uO0jBcxEEhK90Unm26er5e529UU" width="690" height="429" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/7/d7e847f6d3ebd870c66a08aa173007e7f252e72c_2_690x429.png, https://discuss.pytorch.org/uploads/default/optimized/3X/d/7/d7e847f6d3ebd870c66a08aa173007e7f252e72c_2_1035x643.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/d/7/d7e847f6d3ebd870c66a08aa173007e7f252e72c_2_1380x858.png 2x" data-dominant-color="242424"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1435×893 138 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: novice96
cooked: <p>Does anyone know the answer?</p>

username: novice96
cooked: <p>any admins knows the answer??</p>

username: novice96
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> sorry I’m getting stuck, do you know how?</p>

username: marksaroufim
cooked: <p>Hi <a class="mention" href="/u/novice96">@novice96</a>, can you please post the full logs in a github gist?</p>
Are weights frozen in nn.Identity?

username: kevin_zhang
cooked: <p>Quick question - I want to use nn.Identity as a placeholder.</p>
<p>Like:<br>
layer_one = nn.Linear(input_size, hidden_size) if use_linear else nn.Identity</p>
<p>However, I am worried that during backprop the Identity weight values will be updated. Will it?</p>
<p>Thanks,<br>
Kevin</p>

username: eqy
cooked: <p>The identity layer shouldn’t have any weights that could be updated.</p>
<pre><code class="lang-auto">import torch.nn
&gt;&gt;&gt; a = torch.nn.Linear(100, 100)
&gt;&gt;&gt; a.weight
tensor([[-0.0092, -0.0083, -0.0101,  ..., -0.0416,  0.0169, -0.0232],
        [ 0.0707,  0.0684, -0.0826,  ..., -0.0583, -0.0801, -0.0349],
        [ 0.0531,  0.0917, -0.0934,  ...,  0.0632, -0.0696, -0.0597],
        ...,
        [-0.0661,  0.0780,  0.0926,  ...,  0.0099, -0.0024, -0.0690],
        [ 0.0313,  0.0154, -0.0628,  ...,  0.0512,  0.0821, -0.0196],
        [ 0.0760,  0.0127, -0.0037,  ..., -0.0742, -0.0545, -0.0989]],
       requires_grad=True)
&gt;&gt;&gt; b = torch.nn.Identity()
&gt;&gt;&gt; b.weight
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "/pytorch/torch/nn/modules/module.py", line 1130, in __getattr__
    raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'Identity' object has no attribute 'weight'
</code></pre>

username: Shashank_Agnihotri
cooked: <p>What is I declare a nn.Parameter() and attach it to Identity() as ‘weight’ using register_parameter(), will the weight for Identity() then be updated using back propagation?</p>

username: ptrblck
cooked: <p>No, since the newly added <code>weight</code> attribute won’t be used in <code>nn.Identity</code>'s <code>forward</code> method.</p>

username: Shashank_Agnihotri
cooked: <p>How do I make sure that it is?<br>
I know it is a bit weird, so I’m trying to find an architecture, and on a edge there is a MixedOp containing Identity(), Zero(), Conv2d() etc.<br>
Conv2d() has its own weights which are used, and given to the optimizer, however for identity() and zero() I’m registering_parameter ‘weight’ and the optimizer is getting these parameters, and the mixedOp is basically the sum of outputs from all the edge operations.</p>

username: ptrblck
cooked: <p>If you want to create a new <code>nn.Parameter</code> and use it, you could write a custom <code>nn.Module</code> and initialize it in the <code>__init__</code> method and use it in the <code>forward</code>. <code>nn.Identity</code> does not use any parameters or buffers and will just return the input itself without any modification. Adding attributes to this module won’t change its <code>forward</code> implementation, so these attributes will never be used.</p>

username: Shashank_Agnihotri
cooked: <p>Thank you, this is helpful.</p>
Why quantized_conv2d layer is much slower than fp32

username: jhw4824
cooked: <p>Hello.</p>
<p>When I applied Quantization to a convolution based model, I found the inference latency is increased.<br>
For specific example, a latency of  a single conv2d layer(input_channel:4, output_channel:16) is about 1.249ms which is measured by PyTorch Profiler. After quantization, the latency of the same layer is about 4.716ms<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/3/138bf1a5448989caf9cf3bfc56e5b8e08e8af3ea.png" data-download-href="https://discuss.pytorch.org/uploads/default/138bf1a5448989caf9cf3bfc56e5b8e08e8af3ea" title="캡처2"><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/3/138bf1a5448989caf9cf3bfc56e5b8e08e8af3ea.png" alt="캡처2" data-base62-sha1="2MUTPdFpSuvMvuxyUVxhl19066K" width="690" height="184" data-dominant-color="BBC0BA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">캡처2</span><span class="informations">736×197 10.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I think the reason might be the output_channel is bigger than input_channel. is this right?<br>
and I want to know the specific purpose of the quantized conv layer’s followings(e.g. aten::contiguous, aten::empty_like etc) If I want to know about these operators, where should I look?</p>
<p>Thanks in advance for your help.</p>

username: andrewor
cooked: <p>Hi Je,</p>
<p>What qengine and hardware are you running this on? Unfortunately individual quantized ops are not heavily optimized at the moment. To speed up the overall model however, you can try to fuse adjacent layers together. For the other ops, you can find more information on the pytorch API docs (e.g. <a href="https://pytorch.org/docs/stable/generated/torch.empty_like.html" rel="noopener nofollow ugc">empty_like</a> and <a href="https://pytorch.org/docs/stable/generated/torch.Tensor.contiguous.html" rel="noopener nofollow ugc">contiguous</a>), and the “aten” you see just refers to the native C++ implementation of these ops.</p>
<p>Best,<br>
-Andrew</p>
Fuse_modules more sequence support

username: Arkeytect
cooked: <p>I noticed that  torch.ao.quantization.fuse_modules only supports a few sequences. Currently I have a model architecture of [‘conv1d’,  ‘ReLu’, ‘BatchNorm1d’,], and this is not supported at the moment. I have two questions, when will this sequence be supported? And what is the workaround for this situation? If I dont run the module fusion, I get the following error: “RuntimeError: Could not run ‘aten::native_batch_norm’ with arguments from the ‘QuantizedCPU’ backend.”</p>
<p>Thanks!</p>

username: andrewor
cooked: <p>Hi Hua Chen,</p>
<p>Actually fusing the modules you have listed should be supported with the fuse_modules API. You can find an example of how this is done in this <a href="https://pytorch.org/tutorials/advanced/static_quantization_tutorial.html" rel="noopener nofollow ugc">tutorial</a> (search for ConvBNReLU). As for the error you’re getting, it usually means you’re passing a quantized tensor to a non-quantized kernel, so you may be missing a DeQuantStub somewhere in your model. Please see this <a href="https://github.com/pytorch/pytorch/blob/352eeb2ef92f2d98d15792d15e6f8067945d6b85/docs/source/quantization.rst#passing-a-quantized-tensor-into-a-non-quantized-kernel" rel="noopener nofollow ugc">doc</a> for more detail.</p>
<p>Best,<br>
-Andrew</p>

username: Arkeytect
cooked: <p>Hi Andrew:<br>
Thanks for your reply! Here is the error message I get from fuse_modules:</p>
<blockquote>
<p>AssertionError: did not find fuser method for: (&lt;class ‘torch.nn.modules.conv.Conv1d’&gt;, &lt;class ‘torch.nn.modules.activation.ReLU’&gt;, &lt;class ‘torch.nn.modules.batchnorm.BatchNorm1d’&gt;)</p>
</blockquote>
<p>In the tutorial, the order of fusion modules are ‘conv’ + ‘bn’ + ‘relu’ where mine is ‘conv’+‘relu’+bn". Would that be a problem?</p>
<p>Best,<br>
Hua</p>

username: andrewor
cooked: <p>Hi Hua Chen,</p>
<p>My mistake. I misread the order of the ops. Unfortunately <code>conv + relu + bn</code> is currently not supported as you suggested, and I am not aware of any immediate plans to add support for it. What you can do is only fuse <code>conv + relu</code>, which is supported (see <a href="https://pytorch.org/tutorials/recipes/fuse.html" rel="noopener nofollow ugc">this link</a> for a full list of supported patterns).</p>
<p>Best,<br>
-Andrew</p>