akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Batch of images, change dimension

PhysicsIsFun: <p>Greetings!</p>
<p>I am trying to preprocess RGB images to create batches of the correct shape (as required by my network).<br>
Example: CIFAR10</p>
<pre><code class="lang-auto">transfos = transforms.Compose([torchvision.transforms.Grayscale(),transforms.ToTensor(),transforms.Normalize((0.4749), (0.2382))])
train = datasets.CIFAR10(".", train = True, download = True, transform = transfos)
test = datasets.CIFAR10(".", train = False, download = True, transform = transfos)

batches_train = torch.utils.data.DataLoader(train, batch_size=bs, shuffle=True)
batches_test = torch.utils.data.DataLoader(test, batch_size=bs, shuffle=True)
</code></pre>
<p>The problem now is that each batch has dimension (batchsize, 1, H, W).<br>
For all batches, I would like to have the dimension (batchsize, H*W). Is there a simple way to use transforms to do this?</p>
<p>Thank you!</p>
<p>PiF</p>

eqy: <p>Does something like:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch
&gt;&gt;&gt; data = torch.randn(32, 1, 32, 32)
&gt;&gt;&gt; data.shape
torch.Size([32, 1, 32, 32])
&gt;&gt;&gt; data = data.reshape(32, -1)
&gt;&gt;&gt; data.shape
torch.Size([32, 1024])
</code></pre>
<p>work?</p>

PhysicsIsFun: <p>That’s how I do it right now, but it strikes me as very inefficient as I have to do this each time my Dataloader hands me a new batch. I was hoping there is a preprocessing solution so that my Dataloader hands me the batches in the correct shape (no more reshaping during training).</p>

eqy: <p>You can also do the reshape in your dataloader if you think that is a significant source of inefficiency. However, this is unlikely because a reshape doesn’t actually change the data layout or require any copying of data. It just changes the strides for indexing.</p>

PhysicsIsFun: <p>Alright, then it might not be computationally inefficient, but it would help readability if I could tell the dataloader from the get go what to do. How do I do it in the Dataloader? The Dataloader is not a simple tensor.</p>

eqy: <p>You can add another transform function to your list of your transforms to do the reshape e.g.,</p>
<pre><code class="lang-auto">def reshape_transform():
    def fn(data):
        batch_size = data.size(0)
        return data.reshape(batch_size, -1)
    return fn

transfos = transforms.Compose([torchvision.transforms.Grayscale(),transforms.ToTensor(),transforms.Normalize((0.4749), (0.2382)), reshape_transform()])
</code></pre>

PhysicsIsFun: <p>This should do it!<br>
I will try it asap. Thank you!</p>
Is there a way to access the local tensors in a backward pass in register_backward_hook

lcskrishna: <p>Currently the register_backward_hook/register_backward_full_hook has only access to the grad_inputs and the grad_outputs. I would like to know if there is any other way to access the local inputs to backward pass (Outputs of a forward pass) via backward hook or some other means?</p>

Yu_Beomgon: <p>Hi, did you find the answers?<br>
if you know how to do, could you please share?</p>
Expected 4-dimensional input for 4-dimensional weight [8, 1, 10, 2], but got 5-dimensional input of size [32, 1, 1, 128, 81] instead

krishna511: <p>Hi there<br>
I am using below function for feature extraction. Using mfcc and melspectrograms  in my model. Both outputs should be of similar dimension. There is not problem with mfcc alone but I am not getting why model is taking 5 dimensions inmelspectrogram. Can anyone figure It out plz<br>
Regards</p>
<pre><code class="lang-auto">class FeatureExtractor(object):
    def __init__(self, rate):
        self.rate = rate

    def get_features(self, features_to_use, X):
        X_features = None
        accepted_features_to_use = ('mfcc', ''melspectrogram')
        if features_to_use not in accepted_features_to_use:
            raise NotImplementedError("{} not in {}!".format(features_to_use, accepted_features_to_use))
        
        if features_to_use in ('mfcc'):
            X_features = self.get_mfcc(X,26)
      
        if features_to_use in ('melspectrogram'):
            X_features = self.get_melspectrogram(X)
      
        return X_features

  


    def get_mfcc(self, X, n_mfcc=13):
        def _get_mfcc(x):
            mfcc_data = librosa.feature.mfcc(x, sr=self.rate, n_mfcc=n_mfcc)
            
            return mfcc_data

        X_features = np.apply_along_axis(_get_mfcc, 1, X)
        return X_features

   

    def get_melspectrogram(self, X):
        def _get_melspectrogram(x):
            mel = librosa.feature.melspectrogram(y=x, sr=self.rate, n_fft=800, hop_length=400)[np.newaxis, :]
            delta = librosa.feature.delta(mel)
            delta_delta = librosa.feature.delta(delta)
            out = np.concatenate((mel, delta, delta_delta))
            return mel

        X_features = np.apply_along_axis(_get_melspectrogram, 1, X)
        return X_features

  
</code></pre>
<p>Here the shapes of X is <code>(5984, 32000)</code></p>

krishna511: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> sir please have a look in this code,<br>
Regards</p>

ptrblck: <p>I’m unsure how this shape is created, but would guess that the transformations are creating it.<br>
Could you add <code>print</code> statements before and after each transformation and check the shapes?</p>

krishna511: <p>In case of mfcc</p>
<pre><code class="lang-auto">mfcc_data.shape=(26,63)
print(X_features.shape)=(8,26,63)
</code></pre>
<p>in melspectrogram</p>
<pre><code class="lang-auto">print(mel.shape)=(1,128,81)
print(X_features.shape)=(8,1,128,81)
</code></pre>
<p>The shape of both the features must be same.</p>

krishna511: <p>Sir actually that [np.newaxis, :] is creating that extra axis<br>
removing it giving both the shapes same.  I am really sorry for this,<br>
But I am confused what is the purpose of this<br>
[np.newaxis, :] ??</p>

ptrblck: <p>The purpose of <code>array[np.newaxis]</code> is to add a new dimension in the specified dim, so you are right in removing it in case it’s not needed.</p>
torch::NoGradGuard appears thread suspending

YangangCao: <p>Hi,  When I use libtorch to forward audios with GPU and use torch::NoGradGuard, The first audio file is normal, but the second will appear thread suspending, the GPU utilization become 0. After a moment, this will be fine and never appear suspending. When I use multi thread, they are all  suspending when forward the second audio file in every thread, I must wait quite a long time…</p>
<p>When I use libtorch to forward audios with GPU and not use torch::NoGradGuard, Thread suspending will much better, however, still a few thread suspending I can feel.</p>
<p>I change torch::NoGradGuard into torch::autogrid::AutoGradMode guard(false), get same situation.</p>
<p>I have no idea about that, can anybody help?</p>

albanD: <p>Hi,</p>
<p>This is hard to say. Are you sure that you don’t have other work that is happening there that happens on the side?</p>
<p>If not, you can use gdb and suspend the execution while it seems to “hang” and see where it is.</p>
Equivalent of numpy.ma.array to mask values in pytorch?

achaiah: <p>Does anyone know how to quickly apply a mask to a torch tensor? Say I have a tensor <strong>t_in</strong> with some values in it and a mask <strong>t_mask</strong> with values 0 and 1. In numpy I could do the following:</p>
<pre><code class="lang-auto">t_result = np.ma.array(t_in, mask=t_mask)
</code></pre>
<p>That would mask all the values of <strong>t_in</strong> where the value of the equivalent index of <strong>t_mask</strong> is 0.  Then I can do operations such as <code>t_result.sum()</code> and it will only give me the sum of the non-masked values. I’m primarily looking to do just that, but cannot get my head around how it can be done efficiently with torch tensors.</p>
<p>Thanks</p>

achaiah: <p>Hmm ok so after digging more looks like something like this could work:</p>
<pre><code class="lang-auto">t_m_bool = t_mask.type(torch.ByteTensor)
t_result = t_in.masked_select(t_m_bool)
</code></pre>
<p>Would love it if someone else could confirm that this is the best approach.</p>
<p>Thanks</p>

albanD: <p>Hi,</p>
<p>If you just want to set the values to 0 when the mask is 0 then the mask and your tensor should be of the same type and just multiply the two.<br>
If you want to extract as a new tensor all the entries of your tensor where the mask value is 1, the the mask should be a ByteTensor and you can use masked_select or []: tensor[mask] or tensor.masked_select(mask).</p>

achaiah: <p>Ok, thank you for confirming.</p>

Maghoumi: <p>I recently found the need for doing masked operations on PyTorch tensors. Here’s how I implemented them (taken from <a href="https://www.codefull.net/2020/03/masked-tensor-operations-in-pytorch/" rel="nofollow noopener">my blog post</a>):</p>
<pre><code class="lang-auto">def masked_mean(tensor, mask, dim):
    """Finding the mean along dim"""
    masked = torch.mul(tensor, mask)  # Apply the mask using an element-wise multiply
    return masked.sum(dim=dim) / mask.sum(dim=dim)  # Find the average!

def masked_max(tensor, mask, dim):
    """Finding the max along dim"""
    masked = torch.mul(tensor, mask)
    neg_inf = torch.zeros_like(tensor)
    neg_inf[~mask] = -math.inf  # Place the smallest values possible in masked positions
    return (masked + neg_inf).max(dim=dim)
</code></pre>

wiiiktor: <p>Will backpropagation still work after using such a masking operation?</p>

albanD: <p>Yes it will.<br>
The values that are used will get gradients corresponding to how they’re used and the values that are zeroed out will get 0 gradient.</p>
Vgg with or without batch norm seems exrremely different

Kevinkevin189: <p>When I’m coding on a binary segmentation task, background as zero and foreground as one.I choose vgg16 with or without batch normalization as my network backbone.Both are pre trained on Imagenet. And I use 1 image to overfit this network to test its performance.The result is weird and I can’t figure it out.<br>
The with version, is able to produce a segmentation map,but the without one, produces all zero’s prediction.<br>
Seems the without version is not tranferrable ?<br>
Why it happens? I use weighted cross entropy as loss function and resizedcrop as data augmentation method,And I tried to remove weight and augmentation, it’s not better.I’m confused.<br>
Network Detail:<br>
My network are simple,encoder is vgg16,and I use Bilinear upsample ,Conv2d(512,2) and Sigmoid to transfer the feature to segmentation map directly. this method always works well while overfitting performance test.<br>
the with version result is coarse but at least it produces something positive. the without one produces only black.</p>

Kevinkevin189: <p>one the other hand, loss is constant in without version, but descend then  keep constant in with version</p>

ptrblck: <p>Could you try to play around with some hyperparameters (learning rate, weight init etc.)?<br>
Your model without BatchNorm layers might be more sensitive to the input distribution.<br>
Are you normalizing the image?</p>

CasellaJr: <p>Did you solve? I have the same problem using a simpler VGG, on MNIST, so background black and numbers white</p>
CUDA out of memory(specific question)

June90: <p>I have GTX 2080 ti 11G and I’m training EfficientDet on Window 10.<br>
I don’t fully understand why I have only 320.00 KiB free? Is anyone can explain this issue?<br>
I think to be working properly, Tried to allocate 16.00 MiB should be less than 320.00 KiB free.<br>
I should know why…</p>
<p><strong>RuntimeError: CUDA out of memory. Tried to allocate 16.00 MiB (GPU 0; 11.00 GiB total capacity; 8.87 GiB already allocated; 320.00 KiB free; 9.03 GiB reserved in total by PyTorch)</strong></p>

June90: <p><a class="mention" href="/u/stas">@stas</a> Could help me with these issues?</p>

stas: <p>Sometimes this has to do with memory fragmentation. Sometimes you are just out of memory. Here clearly it’s the latter since 16MB &gt; 320KB.</p>
<p>Typically users are puzzled about the memory fragmentation scenario where the OOM happens trying to allocate say 16MB, while reporting 200MB free, which makes no sense immediately, until you understand that in those 200MB there is not a single contiguous chunk of free memory that is larger than 16MB.</p>
<p>You can find the memory fragmentation discussed in multiple threads here on this forum, e.g. here is one of them where I was trying to figure it out:</p><aside class="quote quote-modified" data-post="1" data-topic="34073">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/stas/48/8409_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/gpu-ram-fragmentation-diagnostics/34073">GPU RAM fragmentation diagnostics</a> 
  </div>
  <blockquote>
    Following up on <a href="https://discuss.pytorch.org/t/unable-to-allocate-cuda-memory-when-there-is-enough-of-cached-memory/33296/7" class="inline-onebox">Unable to allocate cuda memory, when there is enough of cached memory</a>, while there is no way to defrag nvidia GPU RAM, is there a way to get the memory allocation map? I’m asking in the simple context of just having one process using the GPU exclusively. 
Using free memory info from nvml can be very misleading due to fragmentation, so it would be useful to be able to have some sort of estimation of fragmentation, e.g. 1GB free, 0.3G of which is in chunks less than 50MB. 
And it w…
  </blockquote>
</aside>
Please help me reduce this slow loop into primitives

hinken: <p>I’m trying to find a more low level pyTorch method to run the code below, as it’s really slow now with the two Python-based loops. It’s essentially a sort of “expansion” of A[] by a sliding window from B…</p>
<pre><code class="lang-auto">for i in range(100):
        for j in range(100):
            corr[i, j, :, : ] = A[i, j] * B[ i:i+25, j:j+25 ]
</code></pre>
<p>(the numbers 100 or 25 are just placeholders)</p>
<p>I’ve been stuck on this for a while and every time I find some primitive that seems it could help, it has some limitation that makes it NOT fit the problem <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>Please help me Obi-Wan, you’re my only hope…</p>

ptrblck: <p>You could use <code>unfold</code> to achieve this:</p>
<pre><code class="lang-auto">A = torch.ones(10, 10) * 2
B = torch.arange(14*14).view(14, 14)
corr = torch.zeros(10, 10, 5, 5)

# Slow method
for i in range(10):
    for j in range(10):
        corr[i, j, :, :] = A[i, j] * B[i:i+5, j:j+5]

# Hopefully faster
corr2 = torch.zeros(10, 10, 5, 5)
D = B.unfold(0, 5, 1).unfold(1, 5, 1)
corr2 = D * A.unsqueeze(2).unsqueeze(3)

(corr == corr2).all()
print(corr[0, 0])
print(corr2[0, 0])
</code></pre>

hinken: <p>Thanks for the unfold trick, I tested this and it gave a speedup of 39x <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"> I was thinking earlier about some way of expanding B so it would match the topology of the corr[] result, but couldn’t find the method.</p>
<p>I guess it does require a bunch of passes over the data, because of the multiple chained unfolds, so perhaps it could be made faster if there was indeed a CUDA primitive that did it all in one pass, but I’ll be able to search for that later when this become a bottleneck again <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>BTW, in a special case, A and B are the same array, and in that case it looks eerily similar to some kind of 2D auto-correlation (of a limited offset size), something you’d expect there to be a primitive for somewhere maybe.</p>

asha97: <pre><code class="lang-auto">for i in range(1024):
    for j in range(1024):
        val= matrix[i,j]
</code></pre>
<p>This is taking alot of time.</p>
What is the best way to pick up some rows in a tensor?

mucahit: <p>In some case, we have to pick up from a tensor some rows that consist a specific value in corresponding another tensor or matrix. Here, I have implemented a function that does this process, but I wonder is there any other faster way?</p>
<pre><code class="lang-auto">def pick_up_corresponding_vectors(embedding_tensor,hp,wanted):
    batch_size, sequence_length, hidden_size = embedding_tensor.shape
    batch_is,token_is = (hp==wanted).nonzero(as_tuple=True)
    indices = torch.zeros((batch_size,torch.bincount(batch_is).max().item()))
    for batch_i, token_i in zip(batch_is,token_is):
        m =(batch_is==batch_i).nonzero().flatten().tolist()
        k = len(m)
        indices[batch_i,:k]= token_is[m]
    batch_size, max_seq = indices.shape 
    indices_ = indices.repeat(1,hidden_size).view(batch_size,-1,max_seq).transpose(2,1).type(torch.int64)
    return torch.gather(embedding_tensor,1,indices_)
</code></pre>
<p><a href="https://github.com/maltintas45/subtoken_level_2_token_level/blob/main/pick_up_some_rows_in_tensor.ipynb" rel="noopener nofollow ugc">Here</a> a notebook to check it easily.  Also you can find more explanation about required the process.</p>
<p>Thanks in advance.</p>
Sample time binary classification/multi-target regression

vpeterson: <p>Hi !<br>
first of all, thank you to anyone who posts and helps in this forum! I am new in deep learning and pytorch and this forum has been one of the best guides for my learning period.</p>
<p>I am trying to develop a net that is able to predict to each sample time the presence or absence of the beginning of my positive class (we can think about this problem like in segmentation but, only in one dimension). Samples with label 1 correspond to those targets which have an onset beginning, while samples with label 0, the target is a flat line (all zeros).<br>
I have as inputs some spectrograms. I use sampler for balancing the classes during the DataLoader. I have built a CNN+RNN net, where the output_size =  number_of_time_samples (i.e. the predictions are of size Batch size x time samples). I am using as activation function ReLu. I have tried different cost functions, like DICE (thinking about this problem as a segmentation problem) and MSE (regression model). In both cases, the net always tries to find a peak, although in those cases where it shouldn’t.<br>
So, is there any way to train the net for a multi-target regression problem?</p>
<p>Thank you!</p>
Embedding from segmentation model

curious.researcher: <p>I’m trying to get mean embedding from finetuned <code>torchvision.models.segmentation.fcn_resnet50</code> model.<br>
One approach that I’ve tested is to swap classifier with <code>nn.Identity</code> layer (returns tensor of shape [batch_size, feature_num, h, w]) and use <code>torch.mean</code> on inference:</p>
<pre><code class="lang-auto">model_embedder.classifier = Identity()
outputs = torch.mean(model_embedder(inputs), dim=[2, 3])
</code></pre>
<p>But I’ve noticed that <a href="https://github.com/pytorch/vision/blob/master/torchvision/models/segmentation/_utils.py" rel="noopener nofollow ugc"><code>forward()</code></a> method of torchvision segmentation models is like:</p>
<pre><code class="lang-auto">...
x = features["out"]
x = self.classifier(x)
x = F.interpolate(x, size=input_shape, mode='bilinear', align_corners=False)
result["out"] = x
...
</code></pre>
<p>And use of <code>F.interpolate()</code> on features may lead to unexpected results (memory leakage and segmentation fault specifically <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> )</p>
<p>I’ve also tried to add <code>AvgPooling</code> layer to model directly:</p>
<pre><code class="lang-auto">class EmbeddingAvgPooling(nn.Module):
    def __init__(self):
        super(EmbeddingAvgPooling, self).__init__()
        
    def forward(self, x):
        return torch.mean(x, dim=[2, 3])
  
model_embedder.classifier = EmbeddingAvgPooling()
</code></pre>
<p>But of course it’s not connected with <code>F.interpolate()</code> application problem in any way.<br>
(In this setup the error is not segmentation fault but rather <code>size mismatch</code> from <code>F.interpolate()</code>, which is expectable).</p>
<p>What can I do to fix this problem? Do I need to overwrite <code>forward()</code> method of the model (how?) or maybe other variants are possible?</p>
<p>P.S. I understand that use of the segmentation model as embedder might be not the greatest idea, but currently trying to test specifically this approach.</p>

curious.researcher: <p>For me the solution was to just use <code>model.backbone(x)</code> instead of ‘model(x)’ on inference:</p>
<pre><code class="lang-auto">outputs = torch.mean(model_embedder.backbone(inputs), dim=[2, 3])
</code></pre>
Is it possible for use context of forward in backward path?

beomgon: <p>Hi,<br>
thanks in advance.<br>
I want to customize back propagation to my own.<br>
I found the below code, and its working well.</p>
<p>class FeatureExtractor(torch.autograd.Function):<br>
<span class="mention">@staticmethod</span><br>
def forward(ctx, input, weight, bias=None, stride=2, padding=3, dilation=1, groups=1):<br>
ctx.save_for_backward(input, weight, bias, confs)</p>
<pre><code>@staticmethod
def backward(ctx, grad_output):
     input, weight, bias, confs = ctx.saved_variables
</code></pre>
<p>Now I want to test with more big model like resnet18 using imagenet.<br>
using above, code is messy.</p>
<p>Can I use forward backward hook for this purpose.<br>
How can I save some tensor in the forward path, and send it to the backward path??<br>
(is there similar to context in above?)</p>
<p>I want to use some value of forward path in back propagation during convolution.<br>
Is it possible??</p>
How to test sparsity of a NN in PyTorch?

Joost_van_der_Burgt: <p>I would like to evaluate a trained NN by looking at the activation distribution of the network’s neurons over a test sample (or batch of samples).</p>
<p>Is there a way to track which / how many neurons were activated in a classification task? And are there other options in PyTorch to evaluate the sparsity of a NN (and compare different NN’s in that regard)?</p>
<p>Thanks!</p>
Could 1 machine with 8 GPUs run 2 DDP process?

Ardeal: <p>Hi,</p>
<p>There are 8 GPU on my computer(1 computer).<br>
I have run 1 process with 2 GPUs(index=0,1).<br>
Could I run 1 more  process on this computer with GPU 3 and 4?</p>

ptrblck: <p>Yes, you could execute multiple scripts and might want to mask the GPUs via <code>CUDA_VISIBLE_DEVICES</code> to make sure the current script sees only the desired devices.</p>

Ardeal: <p>Thank you!<br>
It does work. I have just now tried.</p>
Static quantization: different training and inference backends

Serhiy_Shekhovtsov: <p>What is the best way to handle different training and inference backends? In <a href="https://spell.ml/blog/pytorch-quantization-X8e7wBAAACIAHPhT" rel="noopener nofollow ugc">this blog</a> it says:</p>
<blockquote>
<p><strong>static quantization must be performed on a machine with the same architecture as your deployment target.</strong> If you are using FBGEMM, you must perform the calibration pass on an x86 CPU; if you are using QNNPACK, calibration needs to happen on an ARM CPU</p>
</blockquote>
<p>But I can’t find anything about this in <a href="https://pytorch.org/tutorials/advanced/static_quantization_tutorial.html" rel="noopener nofollow ugc">the official tutorial</a>. How accurate is this statement? Is it true for both options(<strong>post-training calibration</strong> and <strong>quantization-aware training</strong>) or only for calibration-based one?</p>

tom: <p>Hi, I meant to reply here, but forgot.</p>
<p>There are some subtle caveats, but I don’t think the author’s description is completely accurate here.</p>
<ul>
<li>So the quantization configuration is backend specific and the operator coverage may differ. This means that it is likely preferable to use the same quantization backend, i.e. when I do this, I use QNNPACK for both the training/conversion (on x86) and on the ARM target.</li>
<li>Except for bugs, I would expect that there is actually less potential for variation in the computation results for the quantized model, as the machine precision deviations originating from floating point non-commutativity should go away for the quantized (part of the) computation.</li>
<li>However, there is no firm requirement to apply absolutely the same. If there were, we’d be in lots of trouble with Quantization Aware Training…</li>
</ul>
<p>Best regards</p>
<p>Thomas</p>
<p>P.S.: I have an ongoing four-part series on quantizing an audio model. In the <a href="https://twitter.com/ThomasViehmann/status/1412824060770324483" rel="noopener nofollow ugc">part 2 posted today</a> I am trying to cover my world-view of what is going on with quantization and in the next part we’ll actually do the quantization.</p>

Serhiy_Shekhovtsov: <p>Thanks <a class="mention" href="/u/tom">@tom</a> this helps a lot!</p>
<p>One more thing I’d like to confirm: Once you train, calibrate, and quantize the model on <code>QNNPACK</code> is it still fine to <strong>evaluate</strong> the quantized model on x86 machine? Can we expect an accurate score, or should we use the target device(target CPU architecture) for the evaluation run?</p>
Is it possible to execute two modules in Parallel in Pytorch?

Shisho_Sama: <p>Hello everyone.<br>
Is it possible to run two modules in parallel in Pytorch?<br>
Currently all I have seen so far, has been in a sequential manner but can I for example do sth like this :</p>
<pre><code class="lang-auto">       input 
p1       |        p2
|********|***********
|                   |
|                   |
|********|***********
         |
         |
     output
</code></pre>
<p>we can implement this in Pytorch easily by just first running operations in path1(p1) and then path2(p2)<br>
and then combine their results.<br>
But is there a way that I can make p1 and p2 run in parallel and execute them faster this way without one being delayed before the other one runs first?<br>
Thanks in advance</p>

ptrblck: <p>If you are running the code on a GPU and call <code>p1</code> and <code>p2</code> after each other, these calls will be queued onto the device and executed asynchronously. Depending on the workload of <code>p1</code>, <code>p2</code> might start while <code>p1</code> is still executing.<br>
If you have multiple GPUs in your system (and don’t use data parallel), you could execute <code>p1</code> and <code>p2</code> on each device and concatenate the result back on a single device.</p>

ecolss: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>, may I ask a follow-up question.</p>
<blockquote>
<p>If you are running the code on a GPU and call <code>p1</code> and <code>p2</code> after each other, these calls will be queued onto the device and executed asynchronously. Depending on the workload of <code>p1</code> , <code>p2</code> might start while <code>p1</code> is still executing.</p>
</blockquote>
<p>So in the case of single-GPU, <code>p1</code> and <code>p2</code> could run in parallel depending on the workload, and this is great since we don’t have to code any particular logic to make them run in parallel.</p>
<blockquote>
<p>If you have multiple GPUs in your system (and don’t use data parallel), you could execute <code>p1</code> and <code>p2</code> on each device and concatenate the result back on a single device.</p>
</blockquote>
<p>But I’m not clear about the 2nd part of your answer, do you mean in the case of multiple-GPU, we need to explicitly code the logic (e.g. moving <code>p1</code> to device1, and <code>p2</code> to device2) to make them run in parallel?</p>

ptrblck: <p>Yes, you could manually move the data as well as model copies to multiple devices and execute them in parallel directly. While the first approach could also run workloads in parallel, it depends on the actual workload as mentioned before.<br>
I wouldn’t assume to see overlapping kernel execution when e.g. cublas matmul kernels are executed, as they tend to allocate all compute resources of the device so that other kernels would have to wait.</p>
Is it possible to quantize all weights of object detection models such as faster - rcnn?

yichong96: <p>The <a href="https://discuss.pytorch.org/t/tutorial-on-quantizing-object-detection-model/102604">thread</a> here states that we can only quantise the backbone of object detection models. Is there by any chance of quantising the whole detection model as well? For example, by defining custom self.quant and self.dequant layers before the forward pass of the RPN and Box predictor modules ?</p>
<p>I managed to quantise the backbone, but it doesn’t give me much decrease in model size because the weights from the RPN and Box predictor modules are far greater than the backbone (mobilenetv2)</p>
Best way to handle variable number of inputs

blueeagle: <p>Hi everyone,</p>
<p>I am currently implementing a PyTorch Model, that has to deal with a varying number of inputs, i.e. it can be initialized to handle for example 3 or 6 input tensors. I am currently trying to figure out what is the best way to implement this. I thought of using a dictionary in my forward method.</p>
<pre><code class="lang-auto">def forward(self, mydict):
     pass
</code></pre>
<p>Does anyone of you have another idea to handle this?</p>
<p>Thank you very much!</p>

pascal_notsawo: <p>Why not use <a href="https://book.pythontips.com/en/latest/args_and_kwargs.html" rel="noopener nofollow ugc">*args and **kwargs</a>?</p>
PyTorch wheel includes dynamically linked libcudart, but libtorch_python.so is statically linked

Element: <p>I require a PyTorch environment which is dynamically linked with libcudart on Linux. I installed the following official wheel:<br>
<a href="https://download.pytorch.org/whl/cu101/torch-1.8.1%2Bcu101-cp36-cp36m-linux_x86_64.whl" class="onebox" target="_blank" rel="noopener nofollow ugc">https://download.pytorch.org/whl/cu101/torch-1.8.1%2Bcu101-cp36-cp36m-linux_x86_64.whl</a></p>
<p>While it includes a dynamic cudart library libcudart-1b201d85.so.10.1, it appears that libtorch_python.so in particular is statically linked to cudart.  Is this intended?  This results in different versions of cudart functions being called.</p>
<p>Do I need to resort to building PyTorch to have full dynamic linking to cudart?  Or does anyone know of any available PyTorch wheels which are truly dynamically linked to cudart?</p>
<p>Thank you for any tips on this.  I’m trying to avoid building PyTorch as it is a bit of a beast, takes over 5 hours, and has crashed my machine several times from out of memory situations, despite having 16GB of memory.</p>
Problems during data loading procedure

YoonhoRoh: <p>Hello, I’m trying to load cifar datasets using class dataset(torch.utils.data.Dataset)<br>
Here is my code</p>
<pre><code class="lang-auto">import numpy as np
import os
from glob import glob
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
import matplotlib.pyplot as plt
from torchvision import datasets, transforms
no_cuda=False
use_cuda = not no_cuda and torch.cuda.is_available()
device = torch.device('cuda' if use_cuda else 'cpu')
os.environ['KMP_DUPLICATE_LIB_OK'] = 'True'


train_paths = glob('dataset/cifar/train/*.png')
test_paths = glob('dataset/cifar/test/*.png')
kwargs = {'num_workers':1, 'pin_memory' : True } if use_cuda else {}
def getlabel(path):
    return path.split('_')[-1].replace('.png', '')
label_names = [getlabel(path) for path in train_paths]
classes = np.unique(label_names)
def get_label(path):
    lbl_name =  path.split('_')[-1].replace('.png', '')
    label = np.argmax(classes == lbl_name) 
    return label

class Dataset(torch.utils.data.Dataset):
    def __init__(self, data_paths, transform=None):
        self.data_paths = data_paths
        self.transform = transform

    def __len__(self):
        return len(self.data_paths)

    def __getitem__(self, idx):
        path = self.data_paths[idx]
        image = Image.open(path)
        label = get_label(path)

        if self.transform:
            image = self.transform(image)
            return image, label

train_loader = torch.utils.data.DataLoader(Dataset(train_paths, transforms.Compose([transforms.RandomHorizontalFlip(), transforms.ToTensor(),
                                                                                    transforms.Normalize(mean = [0.406], std=[0.225])])), batch_size = 64,
                                           shuffle = True, **kwargs)
test_loader = torch.utils.data.DataLoader(Dataset(test_paths, transforms.Compose([transforms.ToTensor(),
                                                                                    transforms.Normalize(mean = [0.406], std=[0.225])])), batch_size = 64,
                                           shuffle = True, **kwargs)
image, label = next(iter(train_loader))
print(image.shape)
</code></pre>
<p>However, error message<br>
’ An attempt has been made to start a new process before the<br>
current process has finished its bootstrapping phase.</p>
<pre><code>    This probably means that you are not using fork to start your
    child processes and you have forgotten to use the proper idiom
    in the main module:

        if __name__ == '__main__':
            freeze_support()
            ...

    The "freeze_support()" line can be omitted if the program
    is not going to be frozen to produce an executable.'
</code></pre>
<p>comes out.<br>
Is there any way to solve this problem?<br>
I am looking forward to see an help. Thanks in advance.<br>
Kind regards,<br>
Yoon Ho</p>

ptrblck: <p>Have you tried to add the if-clause protection as given in the error message?<br>
If so, did you receive the same error?</p>
How does torchvision.datasets.ImageNet requires the path?

Pablo1: <p>Does it need ImageNet to be decompressed or in a .tar file?</p>
<pre><code class="lang-auto">imagenet_data = torchvision.datasets.ImageNet('imagenet_compressed.tar')
</code></pre>
<p>or</p>
<pre><code class="lang-auto">imagenet_data = torchvision.datasets.ImageNet('imagenet_decompressed_folder')
</code></pre>
<p><a href="https://pytorch.org/vision/stable/datasets.html#imagenet" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/vision/stable/datasets.html#imagenet</a></p>

eqy: <p>It expects a root directory rather than the compressed archive:<br>
<code>root (string) – Root directory of the ImageNet Dataset.</code></p>
Can't add two different dimension tensor value

arid: <p>Now I want to do something like this:</p>
<pre><code class="lang-auto">tensor= [torch.Size([32, 1, 3, 3]), torch.Size([10]), torch.Size([128, 9216]), torch.Size([64])]
for i, t in enumerate(tensor):
    if i &gt; = 1:
        n_t = t[i].add(t[i-1])
</code></pre>

LeviViana: <p>what are you expecting as output?</p>
Model outputs nan

Rajat_Sharma1: <p>I have a pytorch model which outputs <code>nan</code> after few epochs. There are no <code>nan</code> in the input as well as no <code>logs or divisions</code> in the loss that can make <code>nan</code>. On debugging I found that the last two layers of the model outputs the <code>nan</code> at some places. However, it automatically resolves after few epochs and then again <code>nan</code> after few epochs.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/c/9c626efd87ae00eb57db90096868f77aa8e5b2b5.png" data-download-href="https://discuss.pytorch.org/uploads/default/9c626efd87ae00eb57db90096868f77aa8e5b2b5" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/c/9c626efd87ae00eb57db90096868f77aa8e5b2b5_2_690x202.png" alt="image" data-base62-sha1="mjrnseaEFd1c5YPLrPz855zaIiF" width="690" height="202" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/c/9c626efd87ae00eb57db90096868f77aa8e5b2b5_2_690x202.png, https://discuss.pytorch.org/uploads/default/optimized/3X/9/c/9c626efd87ae00eb57db90096868f77aa8e5b2b5_2_1035x303.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/9/c/9c626efd87ae00eb57db90096868f77aa8e5b2b5_2_1380x404.png 2x" data-dominant-color="060606"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">2116×620 38.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>What could be the potential reason for this. My loss function consists of regression ad cross entropy-loss. For regression loss, I am calculating the the L1 loss only at places where target values are less than 1 where as cross-entropy loss over whole grid by converting target and output of models as occupancy grid (values less than 1  as 1 and greater than 1 as 0).</p>

Kapil_Rana: <p>Cross entropy loss different in Pytorch<br>
This criterion combines [<code>LogSoftmax</code>] and [<code>NLLLoss</code>] in one single class.</p>
<p>what is the last layer activation function, cross-entropy consider last should be scalar.</p>

Rajat_Sharma1: <p>I don’t use any activation function at the last layer. Anyway, the problem is solved by setting lower learning rate.</p>
Sampling batches

victorc25: <p>Hello!</p>
<p>I have a case for image to image translation, where I want to use two classes of images (landscapes and faces), but I want to randomize batches of images in a way that classes are not mixed.</p>
<p>For example, landscape images should be used 4 out of 5 times and faces only 1 out of 5.</p>
<p>I haven’t found much of references for a case like this, I read a comment about using ConcatDataset to combine the two classes in a single dataset and then use ConcatDataset.cumulative_sizes to get the boundaries.</p>
<p>I’m planning to use a custom sampler and randomly select the index range for either class, but is there any better way to do this? Or any recommendation or pointers?</p>

ptrblck: <p>In case the class distribution (4 landscape samples + 1 face sample) is not a hard requirement, you could use a <code>WeightedRandomSampler</code> and specify the sample weights in such a way that the desired distribution would be used in the “average batch”.<br>
However, if the distribution is fixed, your approach seems like the best one, i.e. create a custom sampler and yield the precomputed indices to the <code>Dataset</code>.</p>

victorc25: <p>Thanks for your feedback! I did write the sampler and it works as expected, but it needs optimization now since I’m using loops of lists to get the indexes and it’s very slow, but I’ll take care of that later <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"> .</p>
<p><code>WeightedRandomSampler</code> would work if I wanted to do oversampling and have mixed images in the same batch, but I want to keep the classes separate in this case. Sadly I didn’t find a way to make it do what I needed.</p>
Tensor assigment when doubly indexed

jungerm2: <p>I’m having trouble understanding why this piece of code does not work as expected:</p>
<pre><code class="lang-auto">a = torch.rand(1000)
idx1 = torch.arange(500, 600)
idx2 = torch.rand(100) &lt; 0.5

a[idx1][idx2] = 1.0
print(a[idx1][idx2])
</code></pre>
<p>I expect the result to be all ones, but the tensor is unchanged. If instead, I index only once like so:</p>
<pre><code class="lang-auto">a[idx1] = 1.0
print(a[idx1])
</code></pre>
<p>Then, as expected I get all ones. Why does the double index not work and how can I achieve this behavior?</p>

curious.researcher: <p>What you are loking for is:</p>
<pre><code class="lang-auto">a[idx1[idx2]] = 1.0
</code></pre>
<p>(Mask <code>idx1</code> indexes tensor using boolean <code>idx2</code> tensor before slicing tensor <code>a</code>)</p>

jungerm2: <p>Maybe my example wasn’t broad enough. Your solution doesn’t work if the idx shapes don’t match. Say idx1 is a boolean array of shape 1000, and idx2 is of shape idx1.sum(). What would you do then?</p>
<p>Something like <code>torch.where(idx1)[0][idx2]</code> could work, but that seems very inefficient.</p>
Can't find Inplace operation: RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation

wxystudio: <p>I know this error is because inplace operation, but I don’t know where is the operation in my code.<br>
I define my model forward function as follow:</p>
<pre><code class="lang-auto">def forward(self, input):
    x = input[0]
    y = input[1]
    x = self.feat(x)
    x_proj = self.proj(x)
    x = x_proj
    x = self.toimg(x, y)
    x = self.concat(x, y)
    x = self.resnet(x)
    return x, x_proj
</code></pre>
<p>And I want to use x_proj to compute a extra loss:</p>
<pre><code class="lang-auto">def feature_regularizer(x_proj):
    device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
    min = torch.zeros(x_proj.shape).to(device).to(torch.float32)
    max = torch.ones(x_proj.shape).to(device).to(torch.float32)
    max[:, 0, :] = max[:, 0, :] * 1920
    max[:, 1, :] = max[:, 1, :] * 1200
    max[:, 2, :] = max[:, 2, :] * 200
    min = min - x_proj
    max = x_proj - max
    loss_min = F.relu(min)
    loss_max = F.relu(max)
    loss = torch.sum(loss_min) + torch.sum(loss_max)
    return loss
</code></pre>
<p>Finally I add this extra loss to the main loss:</p>
<pre><code class="lang-auto">pred, x_proj = classifier(input)
loss = scheduler(pred, target)
reg = feature_regularizer(x_proj) * 1e-8
loss = loss + reg
</code></pre>
<p><strong>I’m very certain the inplace operation is in def feature_regularizer(x_proj), because when I delete this loss, I can run the code.</strong><br>
But I can’t find the inplace operation in def feature_regularizer(x_proj), so I’m confused, pls help me</p>

albanD: <p>Hi,</p>
<p>As mentioned in the error message, you should be able to enable anomaly mode to get a better idea where the error comes from.<br>
In the function you mention, doing <code>max[:, 0, :] = XXX</code> is an inplace operation. So most likely the issue?<br>
You can try to replace this with:<br>
<code>max = max * torch.tensor([[[1920,], [1200,], [200,]],])</code> (weird brackets to make it a tensor of size <code>[1, 3, 1]</code> to make sure broadcasting does what you want.</p>

wxystudio: <aside class="quote no-group" data-username="albanD" data-post="2" data-topic="99225">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar"> albanD:</div>
<blockquote>
<p>max = max * torch.tensor([[[1920,], [1200,], [200,]],])</p>
</blockquote>
</aside>
<p>sorry the solution is wrong, I still get the inplace error.<br>
I think this no-inplace constraint targets the  tensor which is forward along the model, only this tensor needs back-propagation, so you can’t change it. But in my code the “max” tensor is just a tensor.</p>

albanD: <p>Ho right it does not require gradients.<br>
In that case you should enable anomaly mode to get more details on where the error comes from by following the instructions in the error message.</p>
Does captum insights support Neuron Attribution?

zoombinis: <p>I’ve followed the tutorial for Integrated Gradient visualization with Captum Insights, but what I’d really like to see is a UI that lets me visualize which neurons are responding to a given input image.</p>
<p>Is this a feature captum insights supports or do I need to roll my own visualization for something like this?</p>
<p>Super helpful in deciding whether to freeze certain layers or neurons so it’d be helpful knowing if this is a feature Captum insights will someday support.</p>
What is the difference between loss and loss.item()

aleemsidra: <p>Loss of a single batch is calculated as:</p>
<pre><code class="lang-auto">loss = criterion(predictions, target)
</code></pre>
<p>What does the function <code>loss.item()</code> specifically do? The values I got after debugging both are the same.</p>

curious.researcher: <p>Relevant links:</p>
<p><a href="https://pytorch.org/docs/stable/generated/torch.Tensor.item.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.Tensor.item.html</a></p>
<blockquote>
<p><code>Tensor.</code> <code>item</code> () → number</p>
<p>Returns the value of this tensor as a standard Python number. This only works for tensors with one element.</p>
</blockquote>
<aside class="quote" data-post="2" data-topic="69629">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/does-item-automatically-move-the-data-to-the-cpu/69629/2">Does .item() automatically move the data to the cpu?</a> 
  </div>
  <blockquote>
    Hi, 
Yes .item() moves the data to CPU. It converts the value into a plain python number. And plain python number can only live on the CPU.
  </blockquote>
</aside>

<p>So, basically <code>loss</code> is one-element PyTorch tensor in your case, and <code>.item()</code> converts its value to a standard Python number.</p>
What is the use of nn.Identity?

vainaijr: <p>example for its use?</p>

tom: <p>I am not sure I subscribe to the use-cases, but there is some discussion in the <a href="https://github.com/pytorch/pytorch/issues/9160" rel="nofollow noopener">feature request</a> for it.</p>
<p>Best regards</p>
<p>Thomas</p>

Sunshine352: <p><em>CLASS</em> <code>torch.nn.</code> <code>Identity</code> ( *<em>args</em> ,  **<em>kwargs</em> )<a href="https://pytorch.org/docs/stable/_modules/torch/nn/modules/linear.html#Identity" rel="noopener nofollow ugc">[SOURCE]</a></p>
<p>A placeholder identity operator that is argument-insensitive.</p>
<p>Parameters</p>
<ul>
<li><strong>args</strong>  – any argument (unused)</li>
<li><strong>kwargs</strong>  – any keyword argument (unused)</li>
</ul>
<p>Examples:</p>
<blockquote>
<blockquote>
<blockquote>
<p>m = nn.Identity(54, unused_argument1=0.1, unused_argument2=False) &gt;&gt;&gt; input = torch.randn(128, 20) &gt;&gt;&gt; output = m(input) &gt;&gt;&gt; print(output.size()) torch.Size([128, 20])</p>
</blockquote>
</blockquote>
</blockquote>

Ursula_Iguaran: <p>I’ve used it to make my code more compact:</p>
<blockquote>
<p>self.layer = nn.Sequential(…,<br>
nn.Dropout(0.5) if dropout else nn.Identity(),<br>
…)</p>
</blockquote>

Jonathan_Rayner: <p>One way I’ve used it: suppose you register a hook to track something about the output of every layer in a network. But if you also want track this statistic for the input to the network, but not the input to any other layer, you have some inconvenient <code>if</code> statements to write.</p>
<p>Instead, just create a dummy layer at the start of the network (or wherever is useful):</p>
<pre><code class="lang-auto">def foo(module, input, ouput):
    # do something with 'output' only
    return


class Network(nn.Module):

    def __init__(self, ...):
        self.Input = nn.Identity()
        ...

        for name, layer in self.named_children():
            layer.register_forward_hook(foo)

    def forward(self, x):
        y = self.Input(x)
   
        # do stuff with 'y'
        ...
</code></pre>