akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Select variable number of values from a batched sequence and make it a padded sequence

M_N: <p>Currently, I am doing this using list comprehension and pad_sequence but it seems slow.<br>
Are there any possible ways to do this faster using some pytorch operations?</p>
<pre><code class="lang-auto">import torch
import torch.nn.utils.rnn as rnn

a = torch.tensor([[1,4,7], [2,5,8]])
mask = a&gt;4

selected_no_pad = [a[i][mask[i]] for i in range(a.size(0))]
# [tensor([7]), tensor([5, 8])]

selected = rnn.pad_sequence(selected_no_pad, batch_first=True, padding_value=-1)
# tensor([[ 7, -1], [ 5,  8]])
</code></pre>
<p>Thanks in advance.</p>

M_N: <p>This almost solved my problem.</p><aside class="onebox stackexchange" data-onebox-src="https://stackoverflow.com/questions/69503156/pytorch-batch-masked-select-implementation">
  <header class="source">

      <a href="https://stackoverflow.com/questions/69503156/pytorch-batch-masked-select-implementation" target="_blank" rel="noopener nofollow ugc">stackoverflow.com</a>
  </header>

  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/2940718/daniel" target="_blank" rel="noopener nofollow ugc">
    <img alt="Daniel" src="https://www.gravatar.com/avatar/ba9b84cda095a4ada68f5050b74dd68f?s=256&amp;d=identicon&amp;r=PG&amp;f=y&amp;so-version=2" class="thumbnail onebox-avatar" width="256" height="256">
  </a>

<h4>
  <a href="https://stackoverflow.com/questions/69503156/pytorch-batch-masked-select-implementation" target="_blank" rel="noopener nofollow ugc">PyTorch Batch masked select implementation</a>
</h4>

<div class="tags">
  <strong>deep-learning, pytorch, pytorch-dataloader</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stackoverflow.com/users/2940718/daniel" target="_blank" rel="noopener nofollow ugc">
    Daniel
  </a>
  on <a href="https://stackoverflow.com/questions/69503156/pytorch-batch-masked-select-implementation" target="_blank" rel="noopener nofollow ugc">01:53AM - 09 Oct 21 UTC</a>
</div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
Transfer Conv2D layers from Keras to Pytorch Give a Wrong Result

malioboro: <p>I create this simple model in Keras:</p>
<pre><code class="lang-auto">model = keras.Sequential([
    Conv2D(3, (3, 3), activation='relu'),
    Flatten(),
    Dense(10, activation='softmax')
])
keras_weights = model.get_weights()
</code></pre>
<p>And I also create a similar model in PyTorch:</p>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(3, 3, kernel_size=3)
        self.fc = nn.Linear(2700, 10)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        x = x.view(-1, 2700)
        x = self.fc(x) 
        return F.softmax(x)
pt_model = Net()
</code></pre>
<p>I know the Keras parameter’s shape is in a different order than PyTorch, <a href="https://discuss.pytorch.org/t/transferring-weights-from-keras-to-pytorch/9889/27">as discussed here</a>. So this is the way I transfer Keras parameters to PyTorch:</p>
<pre><code class="lang-auto">for i, (n, p) in enumerate(pt_model.named_parameters()):
    with torch.no_grad():
        if 'conv' in n and 'weight' in n:
            p.copy_(torch.from_numpy(np.transpose(keras_weights[i], (3,2,0,1)))) # for conv layer's weight
        elif 'weight' in n:
            p.copy_(torch.from_numpy(np.transpose(keras_weights[i], (1,0)))) # for dense layer's weight
        else:
            p.copy_(torch.from_numpy(keras_weights[i]))
</code></pre>
<p>There is no “size mismatch error” so far. However, when I compared the result, it showed different results:</p>
<pre><code class="lang-auto">x = np.ones((1,32,32,3))/2 # for keras
y = torch.ones((1,3,32,32))/2 # for pytorch

pt_model.eval()
print(pt_model(y))
print(model(x))

# output:
# [[0.1495, 0.0770, 0.0740, 0.0601, 0.1135, 0.1814, 0.1109, 0.0918, 0.0777, 0.0640]] # pytorch
# [[0.13737796 0.0906269  0.12652941 0.08460842 0.11374795 0.10845644 0.07447342 0.0781411   0.10338538 0.08265299]] # keras
</code></pre>
<p>But when I tried to remove the conv layer (the model is only a single dense layer), the transfer worked, and the result showed the same value (slightly different in precision). Can someone help me to find out what’s the problem with the transfer process?</p>
<p>Here I provide the code in Colab <a href="https://colab.research.google.com/drive/1iMarcjfU35i-XYaTA5FmISLWnaGlnr79?usp=sharing" class="inline-onebox" rel="noopener nofollow ugc">Google Colab</a></p>
Pytorch broadcasting operation takes too much gpu memory

Eddie1997: <p>Hi, I’m trying to calculate a specific loss, which needs to do the broadcasting operation, however I found it takes too much GPU memory, can anyone help me with it? I’ll show the example code below:</p>
<p>x = torch.randn(32, 7862, 1, 3).cuda()<br>
y = torch.randn(32, 1, 7862, 3).cuda()<br>
loss = torch.linalg.norm(x - y, dim = -1)</p>
<p>print(‘before sub’)<br>
print('mem expected in MB: ', (x.nelement() + y.nelement()) * 4 / 1024 ** 2)<br>
print('mem allocated in MB: ', torch.cuda.memory_allocated() / 1024 ** 2)<br>
print('max mem allocated in MB: ', torch.cuda.max_memory_allocated() / 1024 ** 2)</p>
<p>print(‘after sub’)<br>
print('mem expected in MB: ', (x.nelement() + y.nelement() + loss.nelement()) * 4 / 1024 ** 2)<br>
print('mem allocated in MB: ', torch.cuda.memory_allocated() / 1024 ** 2)<br>
print('max mem allocated in MB: ', torch.cuda.max_memory_allocated() / 1024 ** 2)</p>
<p>Many thanks!</p>

Eddie1997: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> hello sir, could you please help me with this issue? Many thanks!</p>

ptrblck: <p>Could you explain why you think this operation takes “too much” memory?<br>
In your example you are broadcasting the tensors to <code>tmp = x - y</code> with a shape of <code>[32, 7862, 7862, 3]</code>, which will use ~22GB of memory in <code>float32</code>.</p>

Eddie1997: <p>Sorry I describe it wrongly, my question is, is there any way to reduce such large GPU memory? Many Thanks!</p>

ptrblck: <p>The final output shape of <code>loss</code> will be <code>[32, 7862, 7862]</code>, so around ~7GB. You could use a for loop to iterate e.g. <code>dim2</code> in <code>y</code> and calculate the <code>norm</code> repeatedly. However, the loop could cause a massive slowdown.<br>
Let me check if there are more elegant approaches.</p>

Eddie1997: <p>Hello sir, thank you for your advice, I will try the iteration of dim2 later to see if it could work, and I will keep an eye on this post to see if there are elegant approaches.</p>
<p>Thank you again for your help.</p>
Forward method in custom layer with 3D tensor multiplying weights

William_Caster: <p>Hello there. I’m new to pytorch and I’m having problems implementing a custom forward pass. I’m trying to implement a custom linear for a MLP with the following forward method:</p>
<pre><code class="lang-auto">def forward(self, x, hebb):

      torch.clamp(hebb, min = -1.0, max = 1.0)

      if self.training:

          unit_noise = torch.rand(self.size_out) * self.noise_scale # noise to be added to units' outputs.

          w_times_x = torch.add(torch.mm(x, (self.weights + hebb).t()), unit_noise)

      else:

          w_times_x = torch.mm(x, (self.weights + hebb).t())

      yout = torch.add(w_times_x, self.bias)

      hebb = torch.einsum('bi,bj-&gt;bij', x, yout) # H(i, j) = y_i * y_j

      return yout, hebb
</code></pre>
<p>The problem that I’m having is that <strong>hebb</strong> is a 3D tensor where the first dimension is the batch size, so I can’t add it to <strong>self.weights</strong> since it is a 2D tensor. What I need is that each “sample index” in the batch has access to the respective index is this hebb 3D tensor, like so when going through the training loop:</p>
<pre><code class="lang-auto">hebb = torch.zeros((batch_size, n, m), dtype = torch.float32, requires_grad = False)

for batch_inputs, batch_outputs in dataloader:
        model(batch_inputs, hebb)
        # inside the foward pass I would need something like:
        # w_times_x = torch.add(torch.mm(x[sample_index], (self.weights + hebb[sample_index]).t()), unit_noise)
</code></pre>
<p>I can’t really provide the samples one-by-one so I’m trying to make this work with batches. Thx for the help in advance.</p>
Is pytorch compatible with AMD Ryzen CPUs

Phungie: <p>I am planning to buy a new CPU and I was wondering if pytorch is compatible with AMD Ryzen CPUs. I have read in some threads that pytorch does not work well with AMD cpus but they are all from 3 years ago. I want to know if those issues have been solved or they are still a problem.</p>

ptrblck: <p>Could you post links to these issues and did you see a referenced GitHub issue, which was still open or already closed?</p>

Phungie: <p>There was this reddit thread and one other on quora that I couldn’t find. These threads just made me doubt if there was any issues now.</p>
<p>Reddit thread: <a href="https://www.reddit.com/r/MachineLearning/comments/iap6yo/discussion_pytorch_favors_intel_against_amds/" class="inline-onebox" rel="noopener nofollow ugc">Reddit - Dive into anything</a></p>

ptrblck: <p>The linked thread mentions that AMD CPUs cannot use Intel’s MKL which is true and will influence the performance. I would not consider this a bug as it seems the lack of support of MKL for AMD CPUs is by design (I’m not at Intel so cannot speak for them).<br>
Besides that I don’t see any other issues mentioned there.</p>

Phungie: <p>Wouldn’t this lack of support for MKL be problematic?</p>

ptrblck: <p>Not for the functionality of CPU operations, but for their performance as the fallback might be slower than MKL. However, you would need to profile it in the end to get a proper comparison as I’m just speculating.<br>
It thus depends on your point of view if you think a potentially slower execution path on the CPU is problematic or not.</p>
<p>But to your original point: yes, PyTorch should be compatible with AMD CPUs functionality-wise.</p>
DataParallel or DistributedDataParallel workaround

SmoothKen: <p>Hi,</p>
<p>For a complex model, DistributedDataParallel does not work as the support is not yet there as of 06/23. DataParallel throws me something like</p>
<pre><code class="lang-auto">RuntimeError: Input type (CUDAComplexFloatType) and weight type (torch.cuda.FloatTensor) should be the same
</code></pre>
<p>despite my first layer being a complex Conv2d. Assuming a single server, is there some sort of workaround so that I can train using multiple GPUs?</p>
Beginner LM-Transformer tutorial: how to see text output?

drscotthawley: <p>Hi, I’ve very much enjoyed the beginner tutorial on tranformers, <a href="https://pytorch.org/tutorials/beginner/translation_transformer.html" class="inline-onebox" rel="noopener nofollow ugc">Language Translation with nn.Transformer and torchtext — PyTorch Tutorials 2.1.1+cu121 documentation</a>, …up the point where it suddenly ends with only looking at scores, but never shows us any sample output text.</p>
<p>How do we get text?  i.e. how do we go from a sequence of model-generated floating-point embedding values back to tokens, back to text?  Is there a <code>vocab</code> function for that? (I’m just getting started with <code>torchtext</code>.)</p>
<p>For example, I’d love to be able to supply a prompt and see what it predicts for the next “n” tokens.<br>
However, even just being able to see some sample generated text vs. target text during training or evaluation would be very helpful.</p>
<p>I failed to find any similar discussion questions on this topic (Sorry, they probably exist. Feel free to point me to one.)</p>
<hr>
<p><strong>UPDATE:</strong></p>
<p>I apparently missed the part where the tutorial says,</p>
<blockquote>
<p>In this tutorial, we train a <code>nn.TransformerEncoder</code> model on a language modeling task. Please note that this tutorial does not cover the training of <a href="https://pytorch.org/docs/stable/generated/torch.nn.TransformerDecoder.html#torch.nn.TransformerDecoder" rel="noopener nofollow ugc">nn.TransformerDecoder</a>, as depicted in the right half of the diagram above."</p>
</blockquote>
<p>But given that the title and the opening text tell us we’ll be predicting <em>words</em>:</p>
<blockquote>
<p>This is a tutorial on training a model to predict the next word in a sequence using the <a href="https://pytorch.org/docs/stable/generated/torch.nn.Transformer.html" rel="noopener nofollow ugc">nn.Transformer</a> module.</p>
</blockquote>
<p>…I found the premature ending of the tutorial to be disappointing.</p>
<p>Is there a place where the decoder side is trained as well, so that we can see the words?  (And could we add a link to that to the tutorial)?</p>
Profiling with DataParallel to compute memory usage

nik1996: <p>Hi,</p>
<p>I am trying to do a forward pass with a model wrapped in torch.nn.DataParallel and compute the GPU memory usage using the torch.profiler.profiler.profile method (code snippet provided below). It seems to work fine when I set the number of GPUs to 1 (i.e., the ‘profile’ method output seems to match the same for the non-DataParallel single-gpu version of the model). However, the output doesn’t look credible when I increase the number of GPUs to more than 1. Is it even possible to get the memory usage during a multi-gpu forward pass?</p>
<p>Code snippet:</p>
<pre><code class="lang-auto">with profiler.profile(activities=[ProfilerActivity.CPU, ProfilerActivity.CUDA],
                      profile_memory=True,
                      record_shapes=True) as prof:
    out = model(input)
</code></pre>
Manual iteration over dataloader memory leak, how to store the iterator correctly?

Oliver: <p>Hi everyone,</p>
<p>I’ve been experiencing this issue:</p><aside class="quote" data-post="1" data-topic="16774">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/gullal/48/12135_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/memory-leak-while-iterating-over-dataloader/16774">Memory leak while iterating over dataloader</a> 
  </div>
  <blockquote>
    In an ideal case I prefer enumerating over dataloader with a for loop. 
But for a particular implementation, I need multiple batches within one iteration in a sequence. 
To do this I use, next(iter(dataloader)) to get as many batches as I need within one iteration. 
The issue is, this causes linear increase in RAM over iterations and epochs. Somehow, the memory is not cleared after every iteration. 
Any way I can avoid using iter and get multiple batches within one iteration?
  </blockquote>
</aside>

<p>From what I understood I should do is store the iterator which I do.<br>
I.e. <code>iterds = iter(dataloader)</code></p>
<p>then I get the item with <code>next(iterds)</code></p>
<p>Is this also using the <code>cycle()</code> method that is described by the OP of the old issue?</p>
<p>How would I do it correctly then?</p>
<p>Many thanks in advance!</p>

Oliver: <p>Did anyone have an answer to this?</p>

ptrblck: <p>It’s unclear what your exact use case is, so could you post a minimal and executable code snippet reproducing the issue?</p>
Backpropagation through combine_state_for_ensemble

LLlearner: <p>In order to train an ensemble, I am using combine_state_for_ensemble. The code snippet looks as follows:</p>
<pre><code class="lang-auto">import torch
from torch import vmap
import functorch
import torchopt
import torch.nn.functional as F
import matplotlib.pyplot as plt
from functorch import combine_state_for_ensemble
x = torch.unsqueeze(torch.linspace(-1, 1, 100), dim=1)
y = x.pow(2) + 0.2*torch.rand(x.size())


class Net(torch.nn.Module):

    def __init__(self, n_feature, n_hidden, n_output):
        super(Net, self).__init__()
        self.hidden = torch.nn.Linear(n_feature, n_hidden) 
        self.predict = torch.nn.Linear(n_hidden, n_output) 

    def forward(self, x):
        x = F.relu(self.hidden(x))     
        x = self.predict(x) 
        return x

ensembles = [Net(n_feature=1, n_hidden=10, n_output=1) for _ in range(3)]
model, params, buffers = combine_state_for_ensemble(ensembles)
import ipdb; ipdb.set_trace(context=21)
optimizer = torchopt.FuncOptimizer(torchopt.adam(lr=0.2)) 
loss_func = torch.nn.MSELoss() 

for t in range(200):
    pred_mapping = vmap(model, in_dims=(0, 0, None))
    pred = pred_mapping(params, buffers, x) 
    loss = loss_func(pred, y)
    if t % 5 == 0:
        print(f"The loss in iteration {t} is {loss}")
        import ipdb; ipdb.set_trace(context=21)
        print(f"First params set is: {params[0]}")
    params = optimizer.step(loss, params)
</code></pre>
<p>When running the code, I get the error “RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn”, and while debugging I noticed that the parameters returned by combine_state_for_ensemble have grad None. How to deal with the backpropagation in this case?</p>

LLlearner: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> could you plz help in here? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

soulitzer: <p>Generally the way you would debug in these cases is to check your tensors as you forward to see whether they “require grad” or not. If you realize as you go forward that your tensors are no longer requiring grad, you probably did something that autograd did not understand.</p>
Transfer learning "from scratch"?

cpt.nemo: <p>Hey folks,</p>
<p>I am new to transfer learning, and I am working on a segmentation task using PT/ MONAI. I downloaded some pre-trained weights and now, I’d like to use them to freeze some layers on those weights and update the weihts of the remaining layers. My goal is to compare, if a fully frozen network (except final conv layer), a network with only first, or first and second layer frozen, result in better/ worse performance.</p>
<p>I am unsure, if the way I do it is correct. Mostly, I am unsure about the fact, if I have actually loaded the weights and the model is pretrained. And I am unsure, if the remain ing, unfrozen weights are actually learning.</p>
<p>My segmentation is for shoulder, while the pretrained model is for the spleen.</p>
<pre><code class="lang-auto"># set the flag, it will change the initial model object 
self.PRETRAINED = True

def model(self, pretrained):
    # Init model 
    model = UNet(
        spatial_dims=3,
        in_channels=1,
        out_channels=2,
        channels=(16, 32, 64, 128, 256), 
        strides=(2, 2, 2, 2),
        num_res_units=2,
        norm=Norm.BATCH,
    ).to(self.device)
    
    # Use pretrained weights
    if pretrained:          
        file = r'C:\Users\BKeoh\Desktop\spleen_ct_segmentation_v0.5.3\spleen_ct_segmentation\models\model.pt'
        model.load_state_dict(torch.load(file))
  
        # Pattern to match the parameter names
        pattern = 'model.0.*'
  
        # Iterate through model parameters
        for name, param in model.named_parameters():
            # Check if the parameter name matches the pattern using fnmatch
            if fnmatch.fnmatch(name, pattern):
                param.requires_grad = False
                    
        print(f'Using pretrained model with {len(pattern)} frozen layers')
    
    else:            
        print('Using randomly init. model')
  
    return model
</code></pre>
<p>Here, I am ssuming to use the <code>pattern = 'model.0.*'</code> to freeze the weights of the first layer of the Unet encoder.</p>
<p>Thanks in advance for any hints!<br>
Cheers.</p>

ptrblck: <p>Your code looks correct assuming you have checked the pattern matching.<br>
The model loading code also looks correct as you would see an error if the <code>state_dict</code> doesn’t match the model architecture.</p>
Creating a Triangular weight matrix with a hidden Layer instead of a fully connected one

Anubhab_Ghosh: <p>Hi Everyone,</p>
<p>I’m quite new to PyTorch and I am excited about using it for some projects.</p>
<p>I have a question regarding a the use of a linear hidden layer. I want to create a hidden layer and put it in nn.Sequential( ) where the weight matrix associated with the hidden layer is desired to be Lower Triangular or Upper Triangular in nature, and then apply some activations and then optimize using Autograd. The problem is nn.Linear( ) creates a fully connected layer with a complete weight matrix which is not what I want.</p>
<p>Any help in this regard would be appreciated.</p>

ptrblck: <p>You could initialize the weight matrices in your desired shape, e.g. by using <code>torch.triu</code>, and then zeroing out the gradients at the same locations.<br>
Here is a small code snippet for this use case, which could be a good starter:</p>
<pre><code class="lang-python"># Init
def triu_init(m):
    if isinstance(m, nn.Linear):
        with torch.no_grad():
            m.weight.copy_(torch.triu(m.weight))

# Zero out gradients
def get_zero_grad_hook(mask):
    def hook(grad):
        return grad * mask
    return hook

# Setup
model = nn.Sequential(
    nn.Linear(10, 10),
    nn.ReLU(),
    nn.Linear(10, 10),
    nn.ReLU(),
    nn.Linear(10, 10)
)
model.apply(triu_init)

# Check params
for name, param in model.named_parameters():
    print(name, param)

# Dummy intpus
x = torch.randn(10, 10)
target = torch.randn(10, 10)
optimizer = torch.optim.SGD(model.parameters(), lr=1e-3)
criterion = nn.MSELoss()

# Create masks for all layers (we can just use one)
mask = torch.triu(torch.ones_like(model[0].weight))
# Register with hook
model[0].weight.register_hook(get_zero_grad_hook(mask))
model[2].weight.register_hook(get_zero_grad_hook(mask))
model[4].weight.register_hook(get_zero_grad_hook(mask))

# Train for some epochs
for epoch in range(10):
    optimizer.zero_grad()
    output = model(x)
    loss = criterion(output, target)
    loss.backward()        
    optimizer.step()
    print('Epoch {}, loss {}'.format(epoch, loss.item()))

# Verify
for name, param in model.named_parameters():
    print(name, param)
</code></pre>

Anubhab_Ghosh: <p>This solution really helped. Thanks so much <a class="mention" href="/u/ptrblck">@ptrblck</a>.</p>

Thomas_Ahle: <p>I encapsulated Patrick’s answer in a class, in case anybody else finds it useful:</p>
<pre><code class="lang-auto">class TriLinear(nn.Linear):
    def __init__(self, in_features, out_features):
        super().__init__(in_features, out_features)
        with torch.no_grad():
            self.weight.copy_(torch.triu(self.weight))
        self.weight.register_hook(lambda grad: grad * torch.triu(torch.ones_like(grad)))
</code></pre>
EuroPython Pytorch Sprint 22nd-23rd July 2023 (Prague)

pgryko: <p>Europython is running next week (17th-23rd) July in Prague. As part of the conference we are running a sprint weekend and one of the sprints is on pytorch (<a href="https://ep2023.europython.eu/sprints#pytorch---tensors-and-dynamic-neural-networks-in-python" class="inline-onebox" rel="noopener nofollow ugc">EuroPython 2023 Sprints | July 17th-23rd 2023 | Prague, Czech Republic &amp; Remote</a>)</p>
<p>If your in Prague - you welcome to join us! We are also looking for volunteers with some pytorch development experience to mentor remotely during the hackathon via europython’s discord channel.</p>
Pytorch Cuda Help, won't install

Friday64: <p>I need help installing torch-cuda,<br>
pip3 install torch-cuda is not working<br>
thanks</p>
<p>pip3 install torch-cuda<br>
ERROR: Could not find a version that satisfies the requirement torch-cuda (from versions: none)<br>
ERROR: No matching distribution found for torch-cuda</p>

ptrblck: <p><code>torch-cuda</code> is not a valid package so please refer to the install instructions given <a href="https://pytorch.org/get-started/locally/">here</a>.</p>
Using 5d tensor in Pytorch-Android

ShnekaM: <p>I have a model that takes (Batch Size, Channel, Number of images, Height, and Width) as input and produces the output.  I am trying to get that model working in Android.</p>
<p>For this, I am taking two images as input, each of the dimensions (C, H, W). Now I must concatenate the two images and include a batch size of 1. Is there a way to do this in Android using Pytorch? Both cat and resize don’t seem to work in torch. Is there another way to do it?</p>
<p>Example:<br>
Image1 Shape : (3, 190, 640)    (given)<br>
Image2 Shape : (3, 190, 640)    (given)<br>
Concatenated Image Shape : (2, 3, 190, 640)     (need to do)<br>
Reshaped Image Shape : (3, 2, 190, 640) (need to do)<br>
Added batch size : (1, 3, 2, 190, 640) (need to do)</p>
<p>Is there a way to achieve this using pytorch-android version?</p>
Varied batch size for compiled model

UmarSpa: <p>I have an experiment setting where I have a different batch size at each iteration during the training. I am using torch.compile() for my model. Whenever the model sees new batch size, it re-compiles and thus the whole process becomes extremely slow.</p>
<p>Is it possible to have a compiled model, which doesn’t re-compile every time it sees new batch size?</p>

marksaroufim: <p>Yes you can set dynamic=True for now. Another option is to warm up and precompile your model with multiple batch sizes. Eventually we’ll probably have support for nested tensors with torch.compile to make this easier.</p>
Memory leak while iterating over dataloader

Gullal: <p>In an ideal case I prefer enumerating over dataloader with a for loop.<br>
But for a particular implementation, I need multiple batches within one iteration in a sequence.<br>
To do this I use, next(iter(dataloader)) to get as many batches as I need within one iteration.</p>
<p>The issue is, this causes linear increase in RAM over iterations and epochs. Somehow, the memory is not cleared after every iteration.</p>
<p>Any way I can avoid using iter and get multiple batches within one iteration?</p>

SimonW: <p>store the iterator, don’t create it everytime</p>

Gullal: <p>Sorry for the confusion. I am storing the iterator and using next() in every iteration to get the batch.</p>
<p>Both lead to memory leak.</p>

Gullal: <p>Found the problem which was causing memory leak. The code I was using also uses itertools cycle() to iterate over another dataloader which was causing the memory leak.</p>
<p>Using iter(dataloader) does not cause any memory leak.</p>

sh0416: <aside class="quote no-group quote-modified" data-username="Gullal" data-post="4" data-topic="16774">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/gullal/48/12135_2.png" class="avatar"> Gullal:</div>
<blockquote></blockquote>
</aside>
<p>Why cycle cause memory leak???</p>
<p>I also experience same issue…</p>

Oliver: <p>Hi Gullal,</p>
<p>I have a similar problem, what do you mean with itertools was using <code>cycle()</code> ? Like the normal <code>iter()</code> function? So how did you solve it?</p>
<p>I am using <code>iter(dataloader)</code> and it causes a memory leak</p>
How to perform "map reduce" style computations efficiently?

hendy: <p>By “map reduce”, let’s say I have three parameters A={set of tensors}, B={set of tensors}, and C={set of tensors}. There may be tens of thousands or more combinations of these.</p>
<p>Now I want to perform some custom computation function (i.e. “map”) between specific combinations of these parameters, and some input data, say { (x1, y1, z1), (x2, y2, z3), … }.  The results of these computations, will be aggregated/mapped/associated back (i.e. “reduce”) based on the parameters A, B, C.</p>
<p>The straightforward way is to use a nested for loop. However I’d like to be able to perform this efficiently, using GPU acceleration where possible, since each particular computation is independent from others.</p>
<p>Thank you in advance.</p>

lzm2256: <p>It has been 5 years… and it seems there is no way to do this.</p>
<aside class="quote" data-post="19" data-topic="70814">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hypnagogic/48/12231_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/fast-way-to-use-map-in-pytorch/70814/19">Fast way to use `map` in PyTorch?</a> 
  </div>
  <blockquote>
    Just a quick thought from a cursory skim of this problem. I’s wondering whether a purely multiprocessing flow may help (on cpus) here: 

chunk the lists (A, B)
use a multiprocessing.Pool.map(func, zip(A, B))


something like (quick pseudocode) 
def chunks(a:List, b:List, n:int) -&gt; Generator;

nproc = os.cpu_count() - 1
with Pool(proceses=nproc) as pool:
  func = functools.partial(process_matrices, *params)
  pool.map(func, zip(a_chunk, b_chunk))
  </blockquote>
</aside>
Efficient ensemble training on multiple GPU

Stefaanhess: <p>Hey,</p>
<p>I am trying to train an ensemble of models on a dataset. At the moment I am using a wrapper, which wraps all my models of the ensemble into a Module:</p>
<pre><code>class Ensemble(nn.Module):

    def __init__(self, models):
        super().__init__()
        self.models = models

    def forward(self, x):
        y = []
        for model in self.models:
            y.append(model(x))
        return y
</code></pre>
<p>This works well, but I think it is not very efficient, because the for-loop does not parallelize well and I still need a lot of memory for the backward-pass, since all results are collected in a single loss function. What would be the best way, to make the training of this ensemble more efficient?<br>
I was thinking of separating the models during training and putting them on different GPUs, but I do not enough GPUs to put every model on a single GPU…<br>
Is it somehow possible to loop over the dataset and over the models in order to efficiently distribute the training over the GPUs? Is there maybe a totally different approach?</p>
<p>Thanks for helping!</p>

LLlearner: <p>Did you find a way to make it efficient?</p>
Torch.var and torch.mean output shape different

lzm2256: <p>I think the two closely related functions <code>torch.var</code> and <code>torch.mean</code> should have the same output shape, but this is not true. Is there any reason for this?</p>
<pre><code class="lang-auto">a = torch.randn(4,3)
a.var(1, True).shape # (4)
a.mean(1, True).shape # (4,1)
</code></pre>
<p>Apparently var is ignoring the <code>keepdims</code> argument.</p>

ptrblck: <aside class="quote no-group" data-username="lzm2256" data-post="1" data-topic="184173">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/l/8baadc/48.png" class="avatar"> lzm2256:</div>
<blockquote>
<p>Apparently var is ignoring the <code>keepdims</code> argument.</p>
</blockquote>
</aside>
<p>That’s not the case and you should explicitly specify the <code>keepdim</code> argument which works fine:</p>
<pre data-code-wrap="python"><code class="lang-python">a.var(1, keepdim=True).shape
# torch.Size([4, 1])
a.mean(1, keepdim=True).shape
# torch.Size([4, 1])
</code></pre>
<p>In your current code the <code>True</code> argument in <code>var</code> would be passed to <code>correction</code> instead of <code>keepdim</code> as seen in the <a href="https://pytorch.org/docs/stable/generated/torch.Tensor.var.html#torch.Tensor.var">docs</a>.</p>

lzm2256: <p>Oh I see. My mistake. Thanks!</p>
Typo in setting requires_grad for model parameters

satyajitvg: <p>Hi , I was trying to freeze some layers in a resnet and had a typo in my code</p>
<pre><code class="lang-auto">weights_v2 = ResNet50_Weights.IMAGENET1K_V2
resnet = resnet50(weights=weights_v2)
for param in resnet.parameters():
    param.requres_grad = True # typo here (should be requires_grad)
</code></pre>
<p>This code did not error out and I spent some time trying to figure out what was the issue, before I noticed the typo.<br>
I realize that it’s possible to add attributes to tensors dynamically and hence there are no errors thrown.<br>
Just sharing this in case someone runs into something similar.</p>
How to use cuda in ubantu 20.04?

krishna511: <p>Hello there,<br>
I have recently change my os to ubantu 20.04. Installed Anaconda, and then in an environment using spyder, I have successfully installed pytorch in same, now after execulting following command:</p>
<pre><code class="lang-auto">torch.cuda.is_available()

</code></pre>
<p>The result is</p>
<p><strong>False</strong></p>
<p>I have graphic card in my laptop.<br>
Can anybody please help with this.</p>
<p>Regards</p>

ptrblck: <p>The PyTorch binaries ship with their own CUDA dependencies and only a properly installed NVIDIA driver is needed.<br>
Either your driver installation is broken or you’ve installed the CPU-only binaries.</p>

krishna511: <p>Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a><br>
However my <code>cuda</code> is stable and I am installing using command available on official website. Sir, my cuda is 12.0 , and the available link for pytorch downloading is for 11.8. I think this is the problem. Is pytorch supporting currently for cuda 12?</p>

ptrblck: <p>No, this shouldn’t be a problem as your locally installed CUDA toolkit won’t be used unless you build PyTorch from source or a custom CUDA extension.<br>
To run any script with the Pytorch binary you would only need a properly installed NVIDIA driver.<br>
The PyTorch nightly binaries ship with CUDA 11.8 and 12.1 and you can select them in the install matrix.<br>
What is <code>python -m torch.utils.collect_env</code> returning?</p>
Where is the implementation of `at::cuda::cumsum_out`?

Bob_Liao: <p>I found a calling of <code>at::cuda::cumsum_out</code> in <code>Multinomial.cu</code> cuda source:</p>
<p><a href="https://github.com/pytorch/pytorch/blob/1d02106e03bcb70054b467956603772b27699a65/aten/src/ATen/native/cuda/MultinomialKernel.cu#L417" rel="noopener nofollow ugc">cumsum_out</a></p>
<p>But I can’t find it’s definition all over the torch repo.</p>

Bob_Liao: <p>OK, I got it: it’s definition is in <code>RegisterCUDA.cpp</code>, just a wrapper to <code>at::native::cumsum</code>.</p>
Different results for GAN model using save and load and directly using the model in current running session after training

sangal: <p>I am training a conditional GAN model for generating images. condition is an image<br>
The problem is following:<br>
Say I train my model for 15 epochs, after that I save it using:</p>
<pre><code class="lang-auto">path_model = 'path.pt'
torch.save(Generator.state_dict(),path_model) 
</code></pre>
<p>Now I generate images using this saved model on test dataset.<br>
I load the model using:</p>
<pre><code class="lang-auto">load_model_name = 'path.pt'
model = Generator_c()   #genertor model of GAN
model.load_state_dict(torch.load(load_model_name))
model.eval()
</code></pre>
<p>But the result obtained using this saved model <strong>(model)</strong> and the model I can access in my current session (<strong>Generator</strong>) after 15 epochs on the same inputs are very different.<br>
1&gt; Result from Saved and load model (<strong>model</strong>) on left and<br>
2&gt; Result from Generator in the current session (<strong>Generator</strong>) on right<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/0/209cf58fe6b64f0f4a338f49d9698a321c20c315.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/209cf58fe6b64f0f4a338f49d9698a321c20c315" title="Screen Shot 2023-07-13 at 5.29.36 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/0/209cf58fe6b64f0f4a338f49d9698a321c20c315_2_517x249.jpeg" alt="Screen Shot 2023-07-13 at 5.29.36 PM" data-base62-sha1="4EvySOBJaOyjGfvcPYgukQUwszH" width="517" height="249" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/0/209cf58fe6b64f0f4a338f49d9698a321c20c315_2_517x249.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/2/0/209cf58fe6b64f0f4a338f49d9698a321c20c315_2_775x373.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/2/0/209cf58fe6b64f0f4a338f49d9698a321c20c315_2_1034x498.jpeg 2x" data-dominant-color="797979"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2023-07-13 at 5.29.36 PM</span><span class="informations">1496×724 63.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I have compared the two by explicitly printing:</p>
<pre><code class="lang-auto">model1.state_dict()
model2.state_dict() 
</code></pre>
<p>both giving the same output. Can anyone help me in understanding what might be the problem here ?</p>
<p>Thanks</p>

ptrblck: <p>Since you’ve already compared the parameters of the loaded model, you should check the data loading pipeline next and make sure the data is processed in the same way.<br>
As a quick check you could also use a static input (e.g. <code>torch.ones</code>) and compare the outputs of the model to make sure the actual model execution is not the problem.</p>