akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Pytorch gradients not being calculated with torch.nn.parameter variable

username: Gul_Zain
cooked: <p>Hello,<br>
I am using code from “<a href="https://github.com/BatsResearch/csp" class="inline-onebox" rel="noopener nofollow ugc">GitHub - BatsResearch/csp: Learning to compose soft prompts for compositional zero-shot learning.</a>” repository. I want to update the tensor soft_embedding mentioned here: <a href="https://github.com/BatsResearch/csp/blob/main/models/csp.py#L99" class="inline-onebox" rel="noopener nofollow ugc">csp/csp.py at main · BatsResearch/csp · GitHub</a>. This tensor is wrapped in torch.nn.parameter and is the only tensor passed to the optimizer (shown in this line: <a href="https://github.com/BatsResearch/csp/blob/main/models/csp.py#L125" class="inline-onebox" rel="noopener nofollow ugc">csp/csp.py at main · BatsResearch/csp · GitHub</a>). I apply cross entropy loss on the logits and try to update the tensor soft_embedding through that. However, during training, i get None value whenever i print model.soft_embeddings.grad. My training sequence is shown below:</p>
<pre><code class="lang-auto">if image_extractor:
        image_extractor.train()
    model.train() # Let's switch to training

    train_loss = 0.0
    # for testing purposes
    count=0
    prev_soft_embed=copy.deepcopy(model.soft_embeddings)
    for idx, data in tqdm(enumerate(trainloader), total=len(trainloader), desc = 'Training'):
        for d in range(len(data)-1):
            data[d]=data[d].to(device)
        data[0] = model.encode_image(data[0])
        loss, _ = model(data)
        optimizer.zero_grad()
        loss.backward(retain_graph=True)
        print("grad printing")
        print(model.soft_embeddings.grad) # output here is none
        print("------------------------")
        optimizer.step()
        if torch.all(prev_soft_embed.eq(model.soft_embeddings)):
            print("no updates made") #i always get this print statement
        else:
            print("updates made")

        train_loss += loss.item()
    train_loss = train_loss/len(trainloader)
    writer.add_scalar('Loss/train_total', train_loss, epoch)
    print('Epoch: {}| Loss: {}'.format(epoch, round(train_loss, 2)))
</code></pre>
<p>I am not sure why this is the case. Requires_grad is also set to True. Looking forward to suggestions.</p>

username: Gul_Zain
cooked: <p>It is a rookie mistake but leaving the answer down here. A for loop I believe was breaking the computation graph. Following was the for loop.</p>
<pre><code class="lang-auto">for pair_index in range(len(pairs)):
        tokens[class_indices]=self.construct_token_tensors(pair_index)
</code></pre>
<p>changed it to following</p>
<pre><code class="lang-auto">pair_indices=list(range(len(pairs)))
tokens=self.construct_token_tensors(pair_indices)
</code></pre>
<p>Whoever has this issue. First look and see if the computation graph is being broken. Avoid for loops and using numpy arrays and then converting them to tensors.</p>
Is there a way to apply a real fast Fourier transform across all axis of the input?

username: Persona_Aleatoria
cooked: <p>Hi,</p>
<p>I have seen that the function torch.fft.rfft only applies the real fast Fourier transform across the last dimension of your input, and applies the usual fast Fourier transform across of the rest of the dimensions.</p>
<p>Is there a way to apply rfft across all the dimensions of your input?</p>
Vectorized slice function

username: mmoelle1
cooked: <p>Is there a vectorized variant of the <code>slice</code> function or an alternative to achieve the following task?</p>
<p>Assume that I have a two-dimensional tensor <code>A</code> from which I want to access a slice, e.g.</p>
<pre><code class="lang-auto">A.index( { Slice(i-1, i+1, 1), Slice(j-1, j+1, 1) } )
</code></pre>
<p>This works well and gives me a 3x3 sub-tensor with entries <code>A[i-1:i+1][j-1:j+1]</code> as long as <code>i</code> and <code>j</code> are scalars. Instead of looping over a set of <code>i</code>’s and <code>j</code>’s I would like to ‘stack’ all of them along the third dimension so that I can apply some operation (in my case this is a <code>matmul</code> with another three-dimensional tensor) to all of them.</p>
<p>The reason for this request is that one-by-one processing requires me to store the result with <code>index_put_</code> which is leading to problems when calculating gradients (see my other post <a href="https://discuss.pytorch.org/t/index-put-breaks-gradient-calculation/156195">here</a>).</p>
Batch_size parameter of DataLoader when using DDP

username: CoffeeBumbleBee
cooked: <p>My goal is the same as in <a href="https://discuss.pytorch.org/t/questions-about-loss-and-backward-process-in-dataparallel/156397">this question</a>.<br>
Basically I want to train a CNN with batch size 16 and image_size 256. Ssince one of my GPU’s can only handle batch size of 8 with image size of 256 my idea is to split this work between 2 GPU’s so that they split batch size of 16 into 2 batches of 8.<br>
I tried to do this with DataParallel model wrapping with no success as this model inluces additional overheads of memory which my GPU’s can’t handle.<br>
I switched to DistributedDataParallel and I am getting the same memory error <code>RuntimeError: CUDA out of memory.</code> as soon as I try to calculate predictions with <code>y_pred = unet(x)</code>.<br>
My way of splitting the dataset is same as in docs:</p>
<pre><code class="lang-auto">sampler = DistributedSampler(dataset_train, num_replicas=world_size, rank=rank, shuffle=False, drop_last=False)
dataloader = DataLoader(dataset_train, batch_size=params['batch_size'], pin_memory=pin_memory, num_workers=num_workers, drop_last=False, sampler=sampler)
</code></pre>
<p>My questiong is what should the parameter <code>batch_size</code> of <code>DataLoader</code> be?<br>
Should it be the total batch size I want to be splitted (in my case 16) or should it be the batch size for each GPU (in my case 8)?</p>
<p>I tried to find the answer in docs with no luck.</p>

username: Jianxin_Wang
cooked: <p>if you DDP within a single server that has two GPUs, then the batch size for each GPU should be 8. DDP runs each batch separately and no output gathers in the default device.</p>
What's a good way to visualize a large .ONNX model?

username: Amur_Saqib_Pal
cooked: <p>I’m trying to visualize a large deep learning model that I’ve exported in the .onnx format. In the past, I’ve used Netron to visualize ONNX graphs but this model is apparently too big for it - every time I try loading it, it throws the following error.</p>
<p><strong>Maximum call stack size exceeded</strong></p>
<p>So, I was wondering if there are any other tools that would support a very deep model and display it? Thanks in advance.</p>
Oscillations in loss function given fixed sample

username: robinho
cooked: <p>In the oldest gradient descent where batch=whole <strong>fixed</strong> sample, should the loss function at all be able to oscillate over epochs? Assuming the following algorithm:</p>
<pre><code class="lang-auto">for epoch in epochs
adam.step()
end
</code></pre>
<p>I think it should just decrease monotonically given a <strong>fixed</strong> sample. If the sample has randomness over epochs, then it’s possible. Has adam anything to do with that, since <strong>adam</strong> isn’t the oldest gradient descent?</p>

username: ParGG
cooked: <p>Generally speaking, even in a convex optimisation problem you might have the loss oscillating. For instance, this could happen if you are close to the global optima but your learning rate is such that you overshoot and end up in a higher loss region. To avoid that, you can use more complicated algorithms such as: <a href="https://en.wikipedia.org/wiki/Backtracking_line_search" class="inline-onebox" rel="noopener nofollow ugc">Backtracking line search - Wikipedia</a> or <a href="https://en.wikipedia.org/wiki/Wolfe_conditions" class="inline-onebox" rel="noopener nofollow ugc">Wolfe conditions - Wikipedia</a> that make sure that you won’t overshoot. Those would eventually give you some guarantees about the monotonic convergence.</p>

username: robinho
cooked: <p>thank you. Is there a way of degenerating adam back to the oldest GD by tunning adam’s parameters?</p>

username: ParGG
cooked: <p>I am not sure why you want to do that, but if you set weight decay and both betas to 0, then you would basically get an inefficient SDG…</p>
Use GNN to see how it labels a single instance of a graph

username: emiserry
cooked: <p>Hi,</p>
<p>I’m completely new to pytorch and pytorch geometric and came across this colab tutorial</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://colab.research.google.com/drive/14GPEIR7uRz50K9E_p9OUwjSOig0ZOB_E?usp=sharing#scrollTo=iM0yCSAeSIrO">
  <header class="source">
      <img src="https://colab.research.google.com/img/favicon.ico?vrz=colab-20220712-060048-RC00_460423519" class="site-icon" width="128" height="128">

      <a href="https://colab.research.google.com/drive/14GPEIR7uRz50K9E_p9OUwjSOig0ZOB_E?usp=sharing#scrollTo=iM0yCSAeSIrO" target="_blank" rel="noopener nofollow ugc">colab.research.google.com</a>
  </header>

  <article class="onebox-body">
    <img src="https://colab.research.google.com/img/colab_favicon_256px.png" class="thumbnail onebox-avatar" width="260" height="260">

<h3><a href="https://colab.research.google.com/drive/14GPEIR7uRz50K9E_p9OUwjSOig0ZOB_E?usp=sharing#scrollTo=iM0yCSAeSIrO" target="_blank" rel="noopener nofollow ugc">Google Colaboratory</a></h3>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>My question is how I would go about giving the trained model a single MUTAG graph as an input to see how it classifies it. As an output I’d simply like to get the classification/label so I can compare it to the actual label.</p>
How to train a model on multiple GPUs?

username: Huntsman
cooked: <p>I am using HuggingFace x PyTorch, and I have a model instance of the <code>RobertaForMaskedLM</code> class, and I wish to train the model on Masked Language Modeling on a dataset with roughly 2M data points with a max seq length of 512, and a total batch size of 2048. Now, I have 4 GPUs at my disposal but am unable to utilize all of them to evenly distribute the batches among them. What I wish to do is send batches each of size <code>2048/4 = 512</code> to the 4 GPUs, to achieve a cumulative batch size of 2048.<br>
I tried using <code>model = nn.DataParallel(model.cuda())</code>, but I get a CUDA out of memory error for every batch size I use (even as low as 8). I am creating the dataloaders as follows:</p>
<blockquote>
<p><code>train_dataloader = DataLoader(train_dataset,                                       shuffle=True,                                       batch_size = data_args.train_batch_size,                                       collate_fn = data_collator,                                       pin_memory=True,                                       num_workers=4)</code></p>
</blockquote>
<p>Where <code>train_batch_size=2048</code></p>

username: ptrblck
cooked: <p>I would recommend to use <code>DistributedDataParallel</code> as it would avoid the overheads introduced by <code>nn.DataParallel</code> and would give you a better speedup.</p>

username: Huntsman
cooked: <p>Could you please link me to a tutorial to do so?</p>

username: thecho7
cooked: <p><a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/intermediate/ddp_tutorial.html</a></p>
<p>This is an official tutorial.</p>

username: lab176344
cooked: <p>If DDP is overwhelming you can consider <a href="https://huggingface.co/docs/accelerate/accelerator" class="inline-onebox" rel="noopener nofollow ugc">Accelerator</a></p>
My feature map value is zero. How can I fix this problem?

username: skarlans
cooked: <p>Hello.</p>
<p>I’m using CNN for learning the shape features from the mechanical part images (Like levers, screws)</p>
<p>I made a lot of images that has lever, spring, cam by using matlab.</p>
<p>and I successfully trained the CNN model that has 97% accuracy for test dataset.</p>
<p>But last convolutional layer has the 4 output(4 Feature Map) and 3 Feature Maps are zero (like null space)<br>
Only 1 feature map has values that activated with images</p>
<p>Why do the feature maps that are only zero values happen?</p>
<p>How can I fix the problem so that has 4 Feature Maps that has some values??</p>

username: Floopybeep
cooked: <p>Usually some feature maps will be blank in a deep CNN, and that’s perfectly normal. That’s because each layer learns the shape of its filters through training. Lower level filters detect whole objects, and higher(deeper) level filters detect specific shapes (noses, handles, etc.). This means that many higher level filters won’t be able to detect anything, and they will produce feature maps that are usually blank.</p>
<p>In your case, however, your final feature map has 4 channels, 3 of which are blank. Perhaps you are using far too big of a network for your dataset size? In any case, if the model goes up to 97% accuracy for the test/eval dataset, then the number of blank feature maps probably won’t matter too much.</p>
Animated (3D) t-sne

username: Berny
cooked: <p>Hi. I’ m doing the t-sne of a test dataset using this code:</p>
<pre><code>        for batch_ts, (XTest, YTest) in enumerate(test_loader):
            XTest = XTest.to(device)
            recon_batch, mu, logvar, latent_space = model(XTest)
            tsne = TSNE(3, verbose=0)
            tsne_proj = tsne.fit_transform(latent_space.detach().numpy())
            cmap = cm.get_cmap('tab20')
            fig, ax = plt.subplots(figsize=(8, 8))
            num_categories = 7
            for lab in range(num_categories):
                indices = YTest == lab
                ax.scatter(tsne_proj[indices, 0],
                                 tsne_proj[indices, 1],
                                 tsne_proj[indices, 2],
                                 c=np.array(cmap(lab)).reshape(1, 4),
                                 label=lab,
                                 alpha=0.5)
            ax.legend(fontsize='large', markerscale=2)
            plt.show()
</code></pre>
<p>I get this error: need at least one array to concatenate. If I do 2D t-sne, the error disappears.<br>
How can I fix it? And is it possible to perform an animated plot of t-sne?</p>

username: ptrblck
cooked: <p>Could you check which method raises the error and check what the expected inputs would be?<br>
I guess a <code>tsne</code> operation might be failing as the inputs might be wrong.</p>
Accuacy of pruned MobileNet V2 after finetune is low

username: goldenrain
cooked: <p>Hi everyone, I am using <code>torch.nn.utils.prune.global_unstructured</code> to prune models. I noticed that if we prune a scratch-trained MobileNet V2 model, the accuracy usually does not drop significantly (the prune ratio is 0.4, the drop is within 2%). However, if we first finetune a pretrained model and then prune it, the accuracy will drop a lot.</p>
<p>To be more specific, here are two ways to get pruned MobileNet V2 models:</p>
<ol>
<li>
<p>First create a model, train it on CIFAR10, CIFAR100, SVHN for several epochs, achieving 92%, 72.5%, 96% accuracy, respectively. Then use the global pruning method to prune them with a 0.4 prune ratio. The accuracy of the pruned models is 91.9%, 71%, 96%, respectively.</p>
</li>
<li>
<p>First fetch a pretrianed MobileNet V2 model. Then finetune it on CIFAR10, CIFAR100, SVHN, the finetuning process includes two steps where in the first step only the last layer is updated and in the second step all layers get updated. The accuracy of these models are 92.5%, 72.8%, 95%, respectively. Finally, prune the finetuned models with a 0.4 prune ratio, resulting in 46.9%, 26.9%, 82.9% accuracy, respectively.</p>
</li>
</ol>
<p>I don’t understand how this can happen. Is there something I am missing? Or is this is a normal situation? Any help or suggestions would be appreciated.</p>

username: goldenrain
cooked: <p>More info about the results:</p>
<ol>
<li>I also tried ResNet18, the accuracy does not drop significantly when pruning a finetuned model.</li>
<li>I also tried different pruning ratio, ranging from 0.2 to 0.6, the results are similar.</li>
<li>The MobileNet V2 model I used is from <code>torchvision.models</code>
</li>
</ol>
The intermediate node inference results of the jit.trace model

username: Wen_Xiao
cooked: <h1>
<a name="background-1" class="anchor" href="#background-1"></a>background</h1>
<p>Use jit.trace to export the torchscript model, and then use tvm import for compilation and acceleration. However, sometimes the inference results of tvm and pytorch are different, and it is difficult to check which layer the difference comes from. Therefore, it is currently necessary to obtain the output of the intermediate node of the torchscript model to check which layer the difference comes from.</p>
<h1>
<a name="question-2" class="anchor" href="#question-2"></a>question</h1>
<p>I learned that register_forward_hook can implement such a function, but it does not work with RecursiveScriptModule. I am very new to pytorch, I would like to ask how much work is required to make RecursiveScriptModule support functions like register_forward_hook, and what are the approximate steps. Also since this is the debug version, I don’t care about the performance of inference.</p>
Construct a graph

username: Samah_Abu_saleem
cooked: <p>I have done object detection on my custom dataset and I got output as shown in an image and I saved the results in txt files. The txt file has label, bbox coordinator and the score. How to represent each bounding box as a node??</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/0/40930c190ac529d09b17dd0e9830f0e7c6b010f0.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/40930c190ac529d09b17dd0e9830f0e7c6b010f0" title="1_EYtn2YE7b6MTzMQyD2R3nA"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/0/40930c190ac529d09b17dd0e9830f0e7c6b010f0_2_690x306.jpeg" alt="1_EYtn2YE7b6MTzMQyD2R3nA" data-base62-sha1="9dfBxaKFHdC5RtbDGfQXQMJMrba" width="690" height="306" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/0/40930c190ac529d09b17dd0e9830f0e7c6b010f0_2_690x306.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/4/0/40930c190ac529d09b17dd0e9830f0e7c6b010f0.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/4/0/40930c190ac529d09b17dd0e9830f0e7c6b010f0.jpeg 2x" data-dominant-color="9F948E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">1_EYtn2YE7b6MTzMQyD2R3nA</span><span class="informations">900×400 106 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>any help would be much appreciated!</p>

username: ParGG
cooked: <p>Could you be more specific?</p>
<ul>
<li>Which information do you want to have stored in each node?</li>
<li>Once you have the nodes, which is the relationship between them? To construct a graph we have to draw edges. but these connections should have some meaning.</li>
</ul>

username: Samah_Abu_saleem
cooked: <p>I try to implement this from the paper “OD-GCN: OBJECT DETECTION BOOSTED BY KNOWLEDGE GCN”. The node will have the feature from the Yolo.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/5/e59a91cb519ef34288756e2415cafcf5e8499314.png" alt="image" data-base62-sha1="wLavgPlTBOgCD0jpUVF0VEtgUzG" width="620" height="94"></p>
Issue with DataParallel model

username: CoffeeBumbleBee
cooked: <p>I am trying to use 2 GPU’s for training.</p>
<p>I followed <a href="https://pytorch.org/tutorials/beginner/blitz/data_parallel_tutorial.html" rel="noopener nofollow ugc">this tutorial</a> and tested 1 GPU with batch size of 8 and image size of 512 and got the output:</p>
<pre><code class="lang-auto">In Model: input size torch.Size([8, 1, 512, 512]) output size torch.Size([8, 1, 512, 512])
</code></pre>
<p>This is maximum capacity for 1 of my GPU’s. My idea is to train on 2 GPU’s and increase the batch size to 16 so each of my GPU’s can get batches of 8.<br>
When I try to run this setup about 10 iterations run smooth with expected output</p>
<pre><code class="lang-auto">In Model: input size torch.Size([8, 1, 512, 512]) output size torch.Size([8, 1, 512, 512])
In Model: input size torch.Size([8, 1, 512, 512]) output size torch.Size([8, 1, 512, 512])
</code></pre>
<p>but then I get an error:</p>
<pre><code class="lang-auto">RuntimeError: CUDA out of memory. Tried to allocate 256.00 MiB (GPU 0; 7.80 GiB total capacity; 5.51 GiB already allocated; 191.31 MiB free;
</code></pre>
<p>This error appears when <code>loss.backward()</code> is called during the training process.<br>
This is puzzling me since by my knowledge there should be enough VRAM in one card to withstand batch size of 8 with image size of 512.<br>
Is there some extra memory needed when using 2 GPU’s that I am not aware of so this task is not so straightforward?</p>
<p>Just to mention I have tested running my training on 2 GPUs with batch size 8 and image size 512 and everything works fine: GPU’s split the batch and each gets 4 images with output:</p>
<pre><code class="lang-auto">In Model: input size torch.Size([4, 1, 512, 512]) output size torch.Size([4, 1, 512, 512])
In Model: input size torch.Size([4, 1, 512, 512]) output size torch.Size([4, 1, 512, 512])
</code></pre>
<p>Why wouldn’t this work when I increase batch size to 16?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="CoffeeBumbleBee" data-post="1" data-topic="156579">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/coffeebumblebee/48/52571_2.png" class="avatar"> CoffeeBumbleBee:</div>
<blockquote>
<p>This is puzzling me since by my knowledge there should be enough VRAM in one card to withstand batch size of 8 with image size of 512.<br>
Is there some extra memory needed when using 2 GPU’s that I am not aware of so this task is not so straightforward?</p>
</blockquote>
</aside>
<p>Yes, <code>nn.DataParallel</code> is usually creating a memory imbalance and will increase the memory usage on the default device as it’s used to store all inputs, all outputs, and to calculate the loss.<br>
We thus recommend to use <code>DistributedDataParallel</code> to avoid this overhead and to get a better performance compared to <code>DataParallel</code>.</p>
Selecting from tensor given the boundaries by two index tensors

username: sajastu
cooked: <p>I’m trying to select a number of data from a universal tensor (<code>encoder_outputs</code>) according to the start and end boundaries by two index tensors. Take the below as an example:</p>
<pre><code># print(start_pre_ids)

tensor([[[ 564,  258, 1239,  528,  579,    0,  183, 1088,  899,   76],
         [1309, 1379, 1766, 1414, 1694, 1648, 1731, 1610, 1856, 1435],
         [1933, 2106, 2030, 1967, 1885, 2242, 2034, 2071, 1902, 2102],
         [2636, 3704, 3267, 2589, 3293, 3497, 3450, 3122, 3347,    0],
         [3939, 4064, 3823, 3872, 3813, 4296, 4538, 4501, 4398,    0],
         [4830, 4890, 4837, 4621, 4808, 4911, 4655, 4928, 4787, 4981],
         [5473, 6525, 5710, 6403, 5414, 5436, 6358, 5012,    0,    0],
         [7025, 6866, 7171, 6733, 6712, 7215, 7279,    0,    0,    0],
         [7507, 7643, 7533, 7741, 7428, 7597, 7390, 7414, 7710,    0],
         [7869, 7790, 8143, 8021, 7821, 8013, 8168, 7886, 7900, 8079]]],
       device='cuda:0')



# print(end_pre_ids)

tensor([[[ 592,  310, 1271,  543,  629,   80,  221, 1115,  938,   98],
         [1379, 1414, 1806, 1430, 1729, 1669, 1738, 1639, 1923, 1503],
         [1937, 2152, 2047, 2030, 1919, 2273, 2038, 2161, 1933, 2174],
         [2659, 3725, 3334, 2643, 3343, 3529, 3483, 3167, 3419,    0],
         [3945, 4112, 3825, 3947, 3843, 4306, 4665, 4589, 4413,    0],
         [4852, 4911, 4890, 4628, 4842, 4932, 4787, 4981, 4804, 5012],
         [5556, 6576, 5748, 6420, 5422, 5492, 6406, 5033,    0,    0],
         [7063, 6927, 7252, 6793, 6786, 7259, 7300,    0,    0,    0],
         [7556, 7689, 7600, 7767, 7465, 7611, 7454, 7438, 7741,    0],
         [7917, 7903, 8174, 8085, 7846, 8021, 8182, 7909, 7917, 8137]]],
       device='cuda:0')
</code></pre>
<p>When I’m doing <code>encoder_outputs[:, start_pre_ids:end_pre_ids]</code>, I’m getting this error:</p>
<blockquote>
<p>*** TypeError: only integer tensors of a single element can be converted to an index</p>
</blockquote>
<p>Is there a way that I can select from <code>encoder_outputs</code> based off the <code>start_pre_ids</code> and <code>end_pre_ids</code>? The <code>encoder_outputs</code>’s shape is: <code>[1, 8192, 1024]</code></p>
How to convert pytorch weights to bytes and back?

username: sjsj0
cooked: <p>I want to convert the weights to bytes and transfer it using zeroMQ socket and after recieving at other end again want to convert it in original form so that the model can use it.</p>
<p>The weights are of the form &lt;class “collections.OrderedDict”&gt; and i am not able to convert it into bytes and further.<br>
Can someone help me with this??</p>
How do you use Pytorch model's function in Onnx to get output instead of model.forward() function

username: deshwal.mahesh
cooked: <p><strong>TL;DR: How can I use <code>model.whatever_function(input)</code> instead of <code>model.forward(input)</code> for the <code>onnxruntime</code></strong>?</p>
<p>I use <a href="https://github.com/openai/CLIP" rel="noopener nofollow ugc">CLIP embedding</a> to create embedding for my Image and texts as:</p>
<p><a href="https://github.com/openai/CLIP/pull/219" rel="noopener nofollow ugc">Code is from the official git merge</a></p>
<pre><code class="lang-auto">! pip install ftfy regex tqdm
! pip install git+https://github.com/openai/CLIP.git

import clip
import torch

device = 'cuda' if torch.cuda.is_available() else 'cpu'

model, preprocess = clip.load("RN50", device=device) # Load any model
model = model.eval() # Inference Only

img_size = model.visual.input_resolution
dummy_image = torch.randn(10, 3, img_size, img_size).to(device)
image_embedding = model.encode_image(dummy_image).to(device))

dummy_texts = clip.tokenize(["quick brown fox", "lorem ipsum"]).to(device)
model.encode_text(dummy_texts)
</code></pre>
<p>and it works fine giving me <code>[Batch, 1024]</code> tensors for both for the loaded model.</p>
<p>Now I have quantized my model in <code>Onnx</code> as:</p>
<pre><code class="lang-auto">model.forward(dummy_image,dummy_texts) # Original CLIP result (1)

torch.onnx.export(model, (dummy_image, dummy_texts), "model.onnx", export_params=True,
  input_names=["IMAGE", "TEXT"],
  output_names=["LOGITS_PER_IMAGE", "LOGITS_PER_TEXT"],
  opset_version=14,
  dynamic_axes={
      "IMAGE": {
          0: "image_batch_size",
      },
      "TEXT": {
          0: "text_batch_size",
      },
      "LOGITS_PER_IMAGE": {
          0: "image_batch_size",
          1: "text_batch_size",
      },
      "LOGITS_PER_TEXT": {
          0: "text_batch_size",
          1: "image_batch_size",
      },
  }
)
</code></pre>
<p>and the model is saved.</p>
<p>When I test the model as :</p>
<pre><code class="lang-auto"># Now run onnxruntime to verify
import onnxruntime as ort

ort_sess = ort.InferenceSession("model.onnx")
result=ort_sess.run(["LOGITS_PER_IMAGE", "LOGITS_PER_TEXT"], 
  {"IMAGE": dummy_image.numpy(), "TEXT": dummy_texts.numpy()})
</code></pre>
<p>It gives me a list of length 2, one for each image and text and the <code>result[0]</code> has shape of <code>[Batch,2]</code>.</p>

username: jerryzh168
cooked: <p>looks like this is not related to quantization, could you add a different tag, looks like we don’t have a tag for onnxruntime right now, maybe you can just remove the tag and there might be some that is able to help</p>
Why do we use threading.Thread and multiprocessing.Process in dataloader.py

username: acplus
cooked: <p>I am reading the code of <code>dataloader.py</code>.<br>
PyTorch adopts threading.Thread to manage _pin_memory,. <a href="https://github.com/pytorch/pytorch/blob/master/torch/utils/data/dataloader.py#L1067" class="inline-onebox" rel="noopener nofollow ugc">pytorch/dataloader.py at master · pytorch/pytorch · GitHub</a>; However, it leverages mutliprocessing.Process to manage dataloader worker <a href="https://github.com/pytorch/pytorch/blob/master/torch/utils/data/dataloader.py#L1048" class="inline-onebox" rel="noopener nofollow ugc">pytorch/dataloader.py at master · pytorch/pytorch · GitHub</a>.</p>
<p>Any considerations about this ?</p>

username: nivek
cooked: <p>Does <a href="https://github.com/pytorch/pytorch/blob/12c30a8250d1a00a0ac2e03d70ec909c76d922c3/torch/utils/data/dataloader.py#L926-L936" rel="noopener nofollow ugc">this section</a> answer your question?</p>
<pre><code class="lang-python"># `pin_memory_thread_done_event`:
    #   A `threading.Event` for a similar purpose to that of
    #   `workers_done_event`, but is for the `pin_memory_thread`. The reason
    #   that separate events are needed is that `pin_memory_thread` reads from
    #   the output queue of the workers. But the workers, upon seeing that
    #   `workers_done_event` is set, only wants to see the final `None`, and is
    #   not required to flush all data in the output queue (e.g., it may call
    #   `cancel_join_thread` on that queue if its `IterableDataset` iterator
    #   happens to exhaust coincidentally, which is out of the control of the
    #   main process). Thus, since we will exit `pin_memory_thread` before the
    #   workers (see below), two separete events are used.
</code></pre>
Filter Output Using Mask

username: se7en.dreams
cooked: <p>I have an output of shape <code>14 x 10 x 128</code>, where <code>14</code> is the <code>batch_size</code>, <code>10</code> is the <code>sequence_length</code>, and <code>128</code> is the object vector representing the objects associated with each sequence element.</p>
<p>Now, not all the sequence elements are relevant. For example, if we look at the first batch element (<code>10 x 128</code>), the sequence in this is made up of only <code>3</code> elements, i.e only <code>3 x 128</code> are useful, whereas the rest <code>7 x 128</code> are just padded elements. I have the mask that has this information is of shape <code>14 x 10</code>. How do I filter the output so that I get only the “relevant” output using the mask?</p>
<p>For example (shapes simplified to keep it succinct):</p>
<pre><code class="lang-python">outputs =  torch.tensor([[[0.2, 0.3], [0.1, 0.4], [0.5, 0.6]], [[0.7, 0.8], [0.9, 0.11], [0.14, 0.15]]])
masks = torch.tensor([[1,1,0], [1,0,0]]) # first batch element, only two elements in seq are relevant, and only one in second

# do I first expand mask?
masks = masks[:, :, None].expand_as(outputs)

# then get the required outputs?
outputs = outputs * masks
</code></pre>

username: ParGG
cooked: <p>You can either do how you proposed (also without <code>expanding_as(outputs)</code> since broadcasting takes care of it) or in the following ways. You can pick your favourite <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"></p>
<pre><code class="lang-python"># shape (14,10,128)
outputs =  torch.tensor(
                         [
                          [
                           [0.2, 0.3],
                           [0.1, 0.4],
                           [0.5, 0.6]
                          ],
                          [
                           [0.7, 0.8],
                           [0.9, 0.11],
                           [0.14, 0.15]
                          ]
                         ]
)
# shape (14,10)
masks = torch.tensor(
                     [
                      [1,1,0],
                      [1,0,0]
                     ]
) # first batch element, only two elements in seq are relevant, and only one in second

# Einsum
masked_outptus = torch.einsum("ijk,ij-&gt;ijk", outputs, masks)
# or unsqueezed + mul
masked_outptus = torch.mul(outputs, masks.unsqueeze(-1)) # outputs * masks.unsqueeze(-1)
</code></pre>
Gradients all 0

username: rjsdebug
cooked: <p>I am trying to implement something that tells me the gradients of each pixel with respect to class probability. I am not sure why, but the gradients for each pixel are always 0.</p>
<p>Here is the main part of the code that is likely causing the issue (but I am not sure what it is):</p>
<pre><code class="lang-auto">    model.eval() # no dropout, in eval mode
    grad_X=X.type(torch.FloatTensor)
    grad_X = torch.reshape(grad_X, (60000,1,28,28))
    grad_X = grad_X[:10]
    print("images:")
    print(grad_X[0])
    """ if get weird errors, make sure this is ok"""
    grad_Y = train_dataset.targets
    grad_Y = grad_Y[:10] # number of training examples to debug with
    print("labels")
    print(grad_Y)
#  print("dones splicing")

    grad_X.requires_grad = True
    # am i doing backprop on the correct thing?
    node_sel = grad_Y.shape
    ones = torch.ones(node_sel) # nx1, one for each training example
#  print("feeding forward")
    logits  = model(grad_X) # feed forward, keeping track of the calculation graph
#  print("done with logits")
   # print("probs")
    probs = torch.nn.Softmax(dim=1)(logits) # n x 10
    #print(probs)
    sel_nodes = probs[torch.arange(len(grad_Y)), grad_Y.type(torch.LongTensor)]
    print("sel nodes shape")
    print(sel_nodes.shape)
    print(sel_nodes)
# print("running gradients")
    sel_nodes.backward(ones) # is this correct?
    print("nodes to do backprop on")
    print(sel_nodes)
    grad = grad_X.grad.data.numpy()
   # grad = grad.squeeze()
    print("gradient")
    print(grad.shape)
    print(grad[3]) # RUN THIS, see if it is really just 0s, it is for all i 
</code></pre>
How to get transpsoed data buffer in pytorch c++

username: Alvin_Chen
cooked: <p>I want get the data buffer atfer transposed a tensor, however, tensor would not really transpose the tensor(the data in buffer is still the original data sequence).</p>
<p>I use contiguous method to get the transposed data, for 2d tensor, it works well, but for higher dimension(e.g. 3d) tensor, this method not work. So how can i get the transposed data?</p>
<pre><code class="lang-auto">// e.g. b is a 3d tensor, how can i get the transposed data?
auto a = b.transpose(1, 2); 

</code></pre>

username: ptrblck
cooked: <p><code>transpose</code> should work on the 3D tensor as seen here:</p>
<pre><code class="lang-python">x = torch.randn(2, 2, 3)
x.transpose(1, 2)
</code></pre>
<p>Could you describe the issue a bit more, please?</p>

username: Alvin_Chen
cooked: <p>Thanks for your feedback. Transpose would work，but I want to get the transposed data in C++. e.g. I have a tensor as below.</p>
<pre><code class="lang-auto"># python
x = torch.tensor([[[1, 2, 3], [4, 5, 6]]])
 
# in pytorch c++ side(I want to add a new op in c++), I want to 
# get the data after transpose. the data of x in memory should 
# be as: {1, 2, 3, 4, 5, 6}

auto x_t = x.transpose(1, 2);

# however after transpose the data of x_t is still {1, 2, 3, 4, 5, 6}
# anyway, it is the expected result for tensor mechanism.
x_t = x_t.contiguous();

# after contiguous, I would get transposed data as {1, 4, 2, 5, 3, 6} for 2D tensor,
# but it not works for 3D tensor, how should I do?
</code></pre>

username: ptrblck
cooked: <p>I’m still unsure what’s failing, as calling <code>contiguous()</code> on a 3D tensor should also work as seen here:</p>
<pre><code class="lang-python">x = torch.arange(2*2*3).view(2, 2, 3)
print(x)
# tensor([[[ 0,  1,  2],
#          [ 3,  4,  5]],

#         [[ 6,  7,  8],
#          [ 9, 10, 11]]])
print(x.shape, x.stride())
# torch.Size([2, 2, 3]) (6, 3, 1)
print(x.is_contiguous())
# True

y = x.transpose(1, 2)
print(y)
# tensor([[[ 0,  3],
#          [ 1,  4],
#          [ 2,  5]],

#         [[ 6,  9],
#          [ 7, 10],
#          [ 8, 11]]])
print(y.shape, y.stride())
# torch.Size([2, 3, 2]) (6, 1, 3)
print(y.is_contiguous())
# False
y = y.contiguous()
print(y.shape, y.stride())
# torch.Size([2, 3, 2]) (6, 2, 1)
print(y.is_contiguous())
# True
</code></pre>
<p>The transpose will manipulate the meta data of the tensor as seen by the shape and stride and the <code>contiguous()</code> call will copy it into a memory-contiguous layout again.</p>

username: Alvin_Chen
cooked: <p>Hi, ptrblck, thanks for your attention.<br>
Yes，you are right. The issue is caused by no sync after contiguous as my data is on cuda. Thanks again.</p>
TorchServe CORS settings: CORS Missing Allow Origin

username: sixthgalaxy
cooked: <p>Can you please have a look at the  below screenshot, and please let me know if I am using the correct CORS settings. I am getting “CORS missing allow origion” error.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/e/0ece64163f5fa7b4a74884ecc550d8fd7abcc367.png" data-download-href="https://discuss.pytorch.org/uploads/default/0ece64163f5fa7b4a74884ecc550d8fd7abcc367" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/e/0ece64163f5fa7b4a74884ecc550d8fd7abcc367.png" alt="image" data-base62-sha1="26YSgdSryVst3EmfsCZgbYDB5fF" width="690" height="256" data-dominant-color="070D0D"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1094×407 13.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: aanav
cooked: <p>Any luck fixing this issue? I’m having the same problem.</p>
Backpropagate the loss of a certain layer

username: S_M
cooked: <p>Hello all<br>
How can I backpropagate loss over a specific layer of network (not the all the layers or the layers before the specific layer). I want to update the weights of a certain layer.<br>
Thanks</p>

username: ptrblck
cooked: <p>You could pass the desired parameters as the <code>inputs</code> argument to the <code>.backward()</code> call, which would then accumulate the gradients only in these parameters and will ignore the rest.</p>

username: S_M
cooked: <p>I wonder if you would share with me any example code.</p>

username: ptrblck
cooked: <p>This should work:</p>
<pre><code class="lang-python">model = models.resnet18()

x = torch.randn(2, 3, 224, 224)
out = model(x)
loss = out.mean()

loss.backward(inputs=list(model.conv1.parameters()))

for name, param in model.named_parameters():
    print(name, param.grad)
</code></pre>
TypeError: max_pool2d(): argument 'kernel_size' (position 2) must be tuple of ints, not tuple

username: Vipin_Gautam
cooked: <p>def pool2d(tensor, type= ‘max’):<br>
sz = tensor.size()<br>
if type == ‘max’:<br>
x = torch.nn.functional.max_pool2d(tensor, kernel_size=(sz[2]/8, sz[3]) )<br>
if type == ‘mean’:<br>
x = torch.nn.functional.mean_pool2d(tensor, kernel_size=(sz[2]/8, sz[3]) )<br>
x = x[0].cpu().data.numpy()<br>
x = np.transpose(x,(2,1,0))[0]<br>
return x</p>
<p>TypeError: max_pool2d(): argument ‘kernel_size’ (position 2) must be tuple of ints, not tuple</p>
<p>I am facing above error and I am not sure what is the fix for this, please provide some help.</p>

username: ptrblck
cooked: <p>The first value of the <code>tuple</code> is a <code>float</code> so use:</p>
<pre><code class="lang-python">x = torch.nn.functional.max_pool2d(tensor, kernel_size=(sz[2]//8, sz[3]))
</code></pre>
<p>and it should work.</p>

username: Vipin_Gautam
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> That was spot on, thank you so much <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
Calculating gradient second moment

username: Or_Nahir
cooked: <p>Hi,</p>
<p>I’m trying to implement a different kind of second moment estimation for Adam like optimizers. The basic idea is to use the fact that each pixel contributes a realization of the gradient (gW_i). Nominally we sum them all to get the gradient of W, gW. On these gradients we calculate first and second moments. I want to estimate the second moment by taking E(gW_i ** 2).</p>
<p>I have a few implementation problems:</p>
<ol>
<li>
<p>E(gW_i ** 2) can be calculated if in the backwards pass if I had g_input and layer inputs (x) using: gW = conv(x, g_input) and E(gW ** 2) = conv(x**2, g_input ** 2). (See <a href="https://pavisj.medium.com/convolutions-and-backpropagations-46026a8f5d2c" class="inline-onebox" rel="noopener nofollow ugc">How does Backpropagation work in a CNN? | Medium</a>). I couldn’t find a hook that gives me both x and g_input.</p>
</li>
<li>
<p>How can I expose the second moment to the optimizer ? It usually get just the gradient.</p>
</li>
<li>
<p>The solution should work on a generic CNN without too much internal changes to the code.</p>
</li>
</ol>
<p>Thanks!</p>
Running feature extraction seperate from classification

username: Umar_Muhammad_Bello
cooked: <p>I want to know if there is a method of running feature extraction of resnet101 separately and then running classification separately</p>

username: ptrblck
cooked: <p>I assume you are referring to the conv blocks as feature extractors and the linear layers as the classifier.<br>
If so, you could create a custom <code>nn.Module</code> and rewrite the <code>forward</code> method.<br>
E.g. you could wrap the <a href="https://github.com/pytorch/vision/blob/9effc4cdc940952500c72b3ea5a0d7348f33b004/torchvision/models/resnet.py#L268-L276">feature extraction part</a> and the <a href="https://github.com/pytorch/vision/blob/9effc4cdc940952500c72b3ea5a0d7348f33b004/torchvision/models/resnet.py#L278-L280">classifier</a> into separate modules and could use them in a standalone manner. (The best approach depends on your actual use case and how you would like to use it)</p>
Jupyter does not point to the right python version

username: nikiguo93
cooked: <p>hi, I create a virtual environment with python3.8. then I activate it and open the jupyter. when i run the code on jupyter, it seems that the jupyter use the python3.9 (~/.local/lib/python3.9/site-packages/tensorflow_hub/<strong>init</strong>.py in <br>
86<br>
87<br>
—&gt; 88 from tensorflow_hub.estimator import LatestModuleExporter<br>
89 from tensorflow_hub.estimator import register_module_for_export)  than python3.8. but the kernel of jupyter is the environment(python3.8) I activated. and when I use ‘!python --version’, it returned python3.8.<br>
can anybody help me with that?</p>

username: ptrblck
cooked: <p>You might need to select the right Python environment again in the notebook.<br>
In case you are running into TensorFlow issues (as indicated in your code), the TF discussion board would be a better place to ask these questions as you would meet the TF experts there.</p>