akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Data augmentation help

Samuel_Bachorik: <p>Hi, i want to ask if I want to rotate images for neural network if this is normal way, if there is no problem with that  black corners. I am not sure if there is another way to do this.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/4/14c76348521a2a6513d1f78342bb20eab0a458d0.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/14c76348521a2a6513d1f78342bb20eab0a458d0" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/4/14c76348521a2a6513d1f78342bb20eab0a458d0.jpeg" alt="image" data-base62-sha1="2XOJCBbY2zok8qjqiHsQTQkzDIA" width="671" height="500" data-dominant-color="6E6E6D"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">846×630 43 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/4/742ff9c8468bcec441f751f792de730610734996.png" data-download-href="https://discuss.pytorch.org/uploads/default/742ff9c8468bcec441f751f792de730610734996" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/4/742ff9c8468bcec441f751f792de730610734996_2_671x500.png" alt="image" data-base62-sha1="gzQaN5IxImrsQTtZ4xJwl7ukbfE" width="671" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/4/742ff9c8468bcec441f751f792de730610734996_2_671x500.png, https://discuss.pytorch.org/uploads/default/original/3X/7/4/742ff9c8468bcec441f751f792de730610734996.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/7/4/742ff9c8468bcec441f751f792de730610734996.png 2x" data-dominant-color="B33FA0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">849×632 69.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

Andrei_Cristea: <p>torchvision.transforms.RandomRotation() lets you specify a <strong>fill</strong> argument, which you can use to vary the color used in the corners so it’s not always black.</p>
<p><a href="https://pytorch.org/vision/main/generated/torchvision.transforms.RandomRotation.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/vision/main/generated/torchvision.transforms.RandomRotation.html</a></p>
Problem with torchvision functional rotate

Cuadrado: <p>Greetings!</p>
<p>I have been trying to perform some data augmentation on my tensors, and I wanted to rotate them. This is a basic example of the code I am working with so far:</p>
<pre><code class="lang-auto">import torch
import torch
from torchvision.transforms.functional import rotate

img = torch.randn(4, 2, 11, 256, 256)

rotated_img = rotate(img, +90)
</code></pre>
<p>However, when trying to run it, I get the following error:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "basicrotate.py", line 7, in &lt;module&gt;
    rotated_img = rotate(img, +90)
  File "/home/javier/directory_env/spikingjelly_env/lib/python3.8/site-packages/torchvision/transforms/functional.py", line 1078, in rotate
    return F_t.rotate(img, matrix=matrix, interpolation=interpolation.value, expand=expand, fill=fill)
  File "/home/javier/directory_env/spikingjelly_env/lib/python3.8/site-packages/torchvision/transforms/functional_tensor.py", line 686, in rotate
    return _apply_grid_transform(img, grid, interpolation, fill=fill)
  File "/home/javier/directory_env/spikingjelly_env/lib/python3.8/site-packages/torchvision/transforms/functional_tensor.py", line 586, in _apply_grid_transform
    img = grid_sample(img, grid, mode=mode, padding_mode="zeros", align_corners=False)
  File "/home/javier/directory_env/spikingjelly_env/lib/python3.8/site-packages/torch/nn/functional.py", line 4201, in grid_sample
    return torch.grid_sampler(input, grid, mode_enum, padding_mode_enum, align_corners)
RuntimeError: grid_sampler(): expected 4D or 5D input and grid with same number of dimensions, but got input with sizes [4, 2, 11, 256, 256] and grid with sizes [4, 256, 256, 2]
</code></pre>
<p>Could any of you help me fix this issue? According to the docs, the input to rotate can be any tensor of shape [… H, W], arbitrarily large before the last two dimensions, so I really don’t know what I’m missing here… Than you very much for your help!</p>
<p>PS.- for additional information:<br>
Torch version: 1.11.0+cu102<br>
Torchvision version: 0.12.0+cu102</p>

Andrei_Cristea: <p>Javier, this doesn’t address your direct question (why doesn’t it work per what the docs say) but as a quick workaround, you could try (using the dimensions from your example):</p>
<p><code>rotated_img = rotate(img.view(4, 2 * 11, 256, 256), +90).view(4, 2, 11, 256, 256)</code></p>

Cuadrado: <p>Thanks! That is precisely what I am doing so far, but I wanted to know if there was a native, more efficient way to make it work <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=12" title=":sweat_smile:" class="emoji" alt=":sweat_smile:" loading="lazy" width="20" height="20"></p>
Concatenating 2d planes with the input image

aloha: <p>Hi,</p>
<p>I’d like to concatenate each input image with 2d matrices but I’m stuck with errors.</p>
<p>I get  (&lt;class ‘RuntimeError’&gt;, RuntimeError(‘Sizes of tensors must match except in dimension 0. Got 1 and 8 (The offending index is 0)’))<br>
since the input_img has size of {Tensor: (8, 3, 256, 256)} and 2d matrices have size of  {Tensor: (1, 1, 256, 256)}</p>
<p>input image size = 3x256x256<br>
batch size = 8</p>
<p>Please help me out with this problem…! Any advice will be appreciated!</p>
<p>Thank you.</p>

Matias_Vasquez: <p>Can you explain what this 2D matrix is?</p>
<p>I can understand that there are 8 images (batch size), each of them have 3 channels (RGB) and are of size 256 x256.</p>
<p>However you only have <strong>1</strong> of these 2D matrices (batch = 1) with the same H and W.</p>
<p>So, do you want to concatenate the same 2D matrix to every image.</p>
<p>Also, do you want to <strong>ADD</strong> it the values of the RGB? → BxCxHxW = 8x3x256x256</p>
<p>Or do you want to concatenate it as a new channel? → BxCxHxW = 8x4x256x256</p>
<p>Or do you want it as a depth D dimension? → BxCxDxHxW = 8x3x1x256x256</p>

Matias_Vasquez: <p>What you are doing is correct. I don´t see why this would be wrong.</p>

aloha: <p>Thanks for your reply!</p>
HuBERT to extract audio features

Gorgen: <p>Dear eveyone: sorry to disturb us. When I follow the HuBert tutorials to extract audio features.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/a/1ae0cf40998191d9c9125965211467b12ac9fc70.png" data-download-href="https://discuss.pytorch.org/uploads/default/1ae0cf40998191d9c9125965211467b12ac9fc70" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/a/1ae0cf40998191d9c9125965211467b12ac9fc70_2_597x499.png" alt="image" data-base62-sha1="3PM47KZpOfgrphdGupGVWDXHzH2" width="597" height="499" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/a/1ae0cf40998191d9c9125965211467b12ac9fc70_2_597x499.png, https://discuss.pytorch.org/uploads/default/optimized/3X/1/a/1ae0cf40998191d9c9125965211467b12ac9fc70_2_895x748.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/1/a/1ae0cf40998191d9c9125965211467b12ac9fc70.png 2x" data-dominant-color="F5F4F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1001×838 62.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/b/5b0b9fc7f114e14748cb868c85065efb0d13f43f.png" data-download-href="https://discuss.pytorch.org/uploads/default/5b0b9fc7f114e14748cb868c85065efb0d13f43f" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/b/5b0b9fc7f114e14748cb868c85065efb0d13f43f.png" alt="image" data-base62-sha1="cZqlYpBrZ8QKhTjgv4kDxhl4dSD" width="690" height="492" data-dominant-color="F6F6F6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1004×717 33.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Finally, features, _ = model.extract_features(waveform).<br>
features are saved in the form of a list. There are a total of 12 elements in the list, each element is in the form of tensor, and the shape is [2, 40, 768]. So when you finally use the feature, do you stack the 12 tensor directly and shape it into [12, 2, 40, 768]. Is there a theoretical problem with doing this?Could you please give me some suggestions?<br>
Thanks, best wishes.</p>
Dataset with multiple spatial temporal graphs

alexa: <p>Hello, I would like to create custom Dataset and my use case is the following:</p>
<ul>
<li>I have 75 patients and for each of them I have connectivity graph with signal  that changes over time<br>
( so the graph is static, although I have different signals at time t0 and t1)</li>
</ul>
<p>I suppose they are spatial-temporal graphs and I can use some models from Pytorch Geometric Temporal.</p>
<p>Speaking about the data - how should I load them? For each patient should I save only 1 graph with multiple node features (representing temporal signal) or multiple graphs should be used (separate for each timestep) ?</p>
<p>Disclaimer: I am a newbie in Graph Neural Nets. I do not expect the code but the help with understanding my use case. Any links to similar problem solution’s will be appreciated.</p>
Unable to run on V100

Hannibal046: <p>Hi, I build pytorch from source by <code>TORCH_CUDA_ARCH_LIST="8.0" python setup.py install </code>. But when running on GPU cluster, I encounter following error, Can anyone help me ? What should I do to make my pytorch compatible both in V100 and A100 ?</p>
<pre><code class="lang-auto">Found GPU5 Tesla V100-SXM2-32GB which is of cuda capability 7.0.
    PyTorch no longer supports this GPU because it is too old.
    The minimum cuda capability supported by this library is 7.5.
</code></pre>
<pre><code class="lang-auto">RuntimeError: CUDA error: no kernel image is available for execution on the device
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
</code></pre>

ptrblck: <p>In your current build process you are only building PyTorch for compute capability 8.0 (Ampere).<br>
If you want to add Volta, use <code>TORCH_CUDA_ARCH_LIST="7.0 7.5 8.0"</code>.</p>

Hannibal046: <p>Thanks so much! It solved all my problem</p>
How the perform matrix multiply between a 2dim tensor and a 3dim tensor?

Anikily: <p>e.g.<br>
a.shape = (1024, 768) and b.shape = (768, 8, 512)<br>
how can i get the results which shape is 1024, 8, 512?<br>
I have tried the operation @, torch.bmm, torch.mm, torch.matmul, but it seems that all above dons’t work in this stuation.<br>
Thanks!</p>

Matias_Vasquez: <p>You can use <code>torch.einsum</code>. It works like magic.</p>
<pre><code class="lang-python">import torch

a = torch.rand((1024, 768))
b = torch.rand((768, 8, 512))

c = torch.einsum('ij, jkl -&gt; ikl', [a, b])

print(c.shape)
</code></pre>
How to parametrize dimension expansion [:, None, None], [None, None,:],...?

jayz: <p>Hello,</p>
<p>I have script where I would like to reshape a tensor based on a dimension parameter dim.<br>
Say the tensor has tensor.shape = [T].</p>
<p>When dim=0, this should be executed:</p>
<blockquote>
<p>tensor[:,None,None] # resulting shape is [T,1,1]</p>
</blockquote>
<p>when dim=1</p>
<blockquote>
<p>tensor[None,:,None] # resulting shape is [1,T,1]</p>
</blockquote>
<p>when dim=2</p>
<blockquote>
<p>tensor[None,None,:] # resulting shape is [1,1,T]</p>
</blockquote>
<p>Is there a corresponding command to do this depending on dim?<br>
I only know unsqueeze, which only works for single dimension adding.</p>
<p>Thanks!<br>
Best, JZ</p>

Andrei_Cristea: <p>Does the below snippet help, where x is your original tensor of length 100 (T in your case)?</p>
<pre><code class="lang-auto">T = 100
dim = 2
x = torch.randn(T)

newdims = [1, 1, 1]
newdims[dim] = T

x_resized = x.view(tuple(newdims))

x_resized.shape
Out: torch.Size([1, 1, 100])
</code></pre>

jayz: <p>nice, thats exactly what I need. I did not work with view so far, so I did not have it in mind. Still learning the vocabulary <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"> Thanks!<br>
Best, JZ</p>
Dataloader trend

dadaddapyer: <p>Hi,<br>
I have the following question.</p>
<ol>
<li>What is the current most popular to store a large dataset (more than 30GB) using PyTorch?</li>
<li>Why do I see people store a large dataset into multiple hdf5 files instead of just one? Will it increase efficiency?</li>
<li>And how to load multiple  hdf5 files efficiently in data loading? Basically, how to write the <strong>init</strong>() and get_item(). I see the post here <a href="https://discuss.pytorch.org/t/dataloader-when-num-worker-0-there-is-bug/25643?fbclid=IwAR2jFrRkKXv4PL9urrZeiHT_a3eEn7eZDWjUaQ-zcLP6BRtMO7e0nMgwlKU" class="inline-onebox">DataLoader, when num_worker &gt;0, there is bug</a>, but it only talked about a single hdf5 file.<br>
Best regrads.</li>
</ol>
Module 'torchvision.datasets' has no attribute 'OxfordIIITPet'

Entropy: <p>According to the torchvision documentation, the Oxford IIIT Pets dataset should be available but when I try to load it using <code>trainset = torchvision.datasets.OxfordIIITPet(root="./", download=True, target_types='segmentation')</code>, I just get the error in the title.</p>

ptrblck: <p>The <code>OxfordIIITPet</code> dataset was added in January 2022, so you might need to update your <code>torchvision</code> version.</p>
How to extract a certain subset from a torchvision dataset?

TinfoilHat0: <p>I’m using a torchvision dataset (FMNIST), and I’d like to extract a subset from the training portion, and use it as my validation dataset. For example, I’d like to extract half of class 1 instances from the training data, and use them only in my validation data.</p>
<p>How can I do it?</p>

nivek: <p>The dataset’s <code>__getitem__</code> returns a tuple of <code>(image, target)</code> where the target is index of the target class. You should be able to iterate through the dataset and identify the indices that you want to keep/discard based on your criteria.</p>
<p>One way to do this is:</p>
<ol>
<li>Iterate through the dataset, find the indices for you want for training and the indices you want for validation</li>
<li>Set <code>train_ds = torch.utils.data.Subset(dataset, train_indices)</code>, and <code>val_ds = torch.utils.data.Subset(dataset, val_indices)</code>
</li>
</ol>

TinfoilHat0: <p>Thanks. I didn’t know of data.Subset.</p>
Differentiate weight gradients with respect to labels

Simas: <p>Hey all, I’m wondering if it’s possible to compute the derivative of updated model parameters with respect to the labels, more concretely:</p>
<pre><code class="lang-auto">dw_{t+1}/dy_t = d/dy_t(w_t - \eta*dL/dw_t) = -\eta*d/dy_t(dL/dw_t)
</code></pre>
<p>where w_{t+1} are the updated weights after computing loss at time step t with some learning rate \eta. This should be possible as the loss at time t is a function of y_t.</p>
<p>When I try to do this with autograd.grad() the shape of the gradients is the same as the shape of my labels, when it should also incorporate the shape of the weights.</p>
<p>What am I doing wrong?</p>
<p>Here’s some example code:</p>
<pre><code class="lang-auto">import torch
from torch import nn, optim
from torch.autograd import grad

class Model(torch.nn.Module):
    def __init__(self):
        super(Model,self).__init__()
        self.layer1 = nn.Sequential(
            nn.Linear(32,16),
            nn.ReLU()
        )
        self.layer2 = nn.Sequential(
            nn.Linear(16,8),
            nn.ReLU()
        )
        self.out = nn.Linear(8,3)
        
    def forward(self, x):
        x = self.layer1(x)
        x = self.layer2(x)
        x = self.out(x)
        return x
    
x = torch.randn(32, 32)
y = torch.nn.functional.one_hot(torch.randint(3, (32,)))
y = torch.tensor(y, dtype = float, requires_grad = True)

model = Model()
loss = nn.CrossEntropyLoss()
opt = torch.optim.Adam(model.parameters(), lr = .001)

opt.zero_grad()
logits = model(x)
error = loss(logits, y)
error.backward(retain_graph = True, create_graph = True)

dw_dy = grad(model.layer2[0].weight.grad, y, torch.ones_like(model.layer2[0].weight.grad))[0]

assert dw_dy.shape == y.shape
</code></pre>
Customize Torchtext Dataset WMT14

keyu_Chen: <p>Recently I want to pre-train an attention model using my own idea.<br>
Since WMT14 is no longer available in torchtext.dataset, I try to build a customized torchtext dataset using WMT14 English-German txt file, <strong>from</strong>: <a href="https://nlp.stanford.edu/projects/nmt/" class="inline-onebox" rel="noopener nofollow ugc">The Stanford Natural Language Processing Group</a>.<br>
So the vocabulary will be based on WMT14.</p>
<p>I already prepocess the training data into .json with the format:<br>
{“German”: “german sentence”, “English”: “English sentence”}</p>
<p>I followed the tutorial:<br>
<a href="https://pytorch.org/tutorials/beginner/translation_transformer.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/beginner/translation_transformer.html</a><br>
and think if I can build something like the Multi30k dataset and use it.</p>
<p>I struggling with the customized dataset recently and hope I can get some advice from the community.</p>
Could Dataset be used as an IterableDataset?

Sam_Lerman: <p>I was able to get a Dataset-inheriting class to work with streaming data by effectively just ignoring the index in <code>__getitem__</code>.</p>
<pre><code class="lang-auto">class MyDataset(Dataset):
                  ...
      def __getitem__(idx):
              return self.get_data()
</code></pre>
<p>This works, so I’m wondering if there’s any advantage to using IterableDataset?</p>
<p>My data is streaming and doesn’t always have a fixed length.</p>

nivek: <p>Depending on whether or not the input is <code>IterableDataset</code>, the <a href="https://github.com/pytorch/pytorch/blob/master/torch/utils/data/dataloader.py" rel="noopener nofollow ugc">DataLoader</a> performs checks to ensure that other settings are compatible with the dataset.</p>
<p>It also makes your intention clear to others who may see your code.</p>
Relation between model parameters and batchsize

ori: <p>I am trying to find relation between batch size and total parameter?</p>
<ul>
<li>Can i increase batchsize of inputs in inference by reducing the model parameters?,<br>
i tried but could not see any difference <img src="https://discuss.pytorch.org/images/emoji/apple/thinking.png?v=12" title=":thinking:" class="emoji" alt=":thinking:" loading="lazy" width="20" height="20">.</li>
</ul>
<p>Reducing parameters may refer to prunning of model, consider this topic as discussion</p>

suraj.pt: <p>What exactly are you trying to achieve? The only relation between batchsize and number of parameters is the memory they occupy (for a fixed amount of GPU memory if your model is larger, the maximum batchsize will be lower)</p>

ori: <p>i am just trying to figure out relation between these two parameters.<br>
So for example:<br>
model has parameters 100k which has batch size of 16<br>
then optimized model has 80k can i feed batch size greater than 16 here?? is it correct</p>

suraj.pt: <p>Yes I’d think so. If you have less parameters in your memory, you can pass a larger batch size.</p>
Changing the image size in DC GAN tutorial

HassanAli: <p>Hi,<br>
I am trying to generate images by following DC GAN tutorial in pytorch.<br>
<a href="https://pytorch.org/tutorials/beginner/dcgan_faces_tutorial.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/beginner/dcgan_faces_tutorial.html</a><br>
This code works fine for image size = 64. But it give me the error when I change the image size to 128. Error is given below:<br>
ValueError: Using a target size (torch.Size([128])) that is different to the input size (torch.Size([3200])) is deprecated. Please ensure they have the same size.</p>
<p>Please guide me how I can change the image size in this tutorial.</p>
<p>Best</p>

ptrblck: <aside class="quote no-group quote-modified" data-username="HassanAli" data-post="1" data-topic="147762">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/h/e9c0ed/48.png" class="avatar"> HassanAli:</div>
<blockquote>
<p>This code works fine for image size = 64. But it give me the error when I change the image size to 64.</p>
</blockquote>
</aside>
<p>I assume this is a typo and you are increasing the image size?<br>
If so, then I would guess that the <code>Discriminator</code> is not returning logits for each sample only but might return a 4-dimensional tensor with a specific spatial size which then won’t match the targets.<br>
In this case, you might need to e.g. add more layers to the <code>Discriminator</code> or use more aggressive pooling.</p>

HassanAli: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>,<br>
Thanks for your answer. Yes, it was typo. I will try after adding more layers.</p>
How to train Two models simultaneously?

ageryw: <p>I have to train two models sequentially, where the loss of one model will somehow be used by the other model. Part of the code is as follows:</p>
<pre><code class="lang-auto">  # model1
  pred1 = model1(data)
  loss1 = loss_fn1(pred1, targets)
  optimizer1.zero_grad()
  loss1.backward()
  optimizer1.step()
     
  # model2
  pred2 = model2(data)
  loss2 = loss_fn2(pred2, targets)
  kl_loss= divergence_loss_fn(
    F.softmax(pred1/t, dim=1),
    F.softmax(pred2/t, dim=1)
  )
  loss = (1-alpha) * loss2 + alpha * kl_loss
  optimizer2.zero_grad()
  loss.backward()
  optimizer2.step()
</code></pre>
<p>If I run this cod as it is I face the following error:</p>
<blockquote>
<p>RuntimeError: Trying to backward through the graph a second time (or directly access saved variables after they have already been freed). Saved intermediate values of the graph are freed when you call .backward() or autograd.grad(). Specify retain_graph=True if you need to backward through the graph a second time or if you need to access saved variables after calling backward.</p>
</blockquote>
<p>If I make the first backward function as <code>loss1.backward(retain_graph=True)</code>, then I face the following error:</p>
<blockquote>
<p>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [2048, 10]], which is output 0 of TBackward, is at version 1565; expected version 1564 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).</p>
</blockquote>
<p>Your help would be much appreciated.</p>

ptrblck: <p>In your current code snippet the <code>kl_loss</code> is created using <code>pred1</code> and <code>pred2</code> and will thus try to calculate the gradients for both models.<br>
Since <code>loss1.backward()</code> and <code>optimizer1.step()</code> were already performed, this is invalid and will raise the errors.<br>
Assuming you don’t want to train <code>model1</code> anymore, you could <code>detach()</code> <code>pred1</code> before passing it to <code>divergence_loss_fn</code>.</p>

ageryw: <p>Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a>. Adding <code>detach()</code> before passing it to <code>divergence_loss_fn</code> solved the problem.</p>
Access a plotted output image from a function directly in the "Image.open" command

danielmanu93: <p>Hi all, this may not be related to pytorch but any ideas are welcomed. I have a function that plots two images (ground truth and predicted image). I want to access these 2 images directly in the “Image.open()” command in another file when I run the function without having to save the 2 images and passing their paths.</p>
<p>So far, I tried to save the images first and pass their paths to the “Image.open()” which works, but that’s not the functionality I want. I want to access the output images from the function directly in the Image.open() command without having to save the images.</p>
Request for correlation layer

Feynman27: <p>Are there any plans to add a pytorch equivalent of the mxnet <a href="https://mxnet.incubator.apache.org/api/python/symbol.html#mxnet.symbol.Correlation" rel="nofollow noopener">correlation layer</a>? Right now I’m going to try implementing a custom layer, similar to <a href="https://github.com/onlytailei/pytorch-flownet2/blob/master/nets/flownetC.py#L45" rel="nofollow noopener">this</a> but I think it would be nice to have out-of-the-box.</p>

Feynman27: <p>Thanks, NVIDIA.<br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://assets-cdn.github.com/favicon.ico" class="site-icon" width="32" height="32">
      <a href="https://github.com/NVIDIA/flownet2-pytorch/tree/master/networks/correlation_package" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars0.githubusercontent.com/u/1728152?s=400&amp;v=4" class="thumbnail onebox-avatar" width="400" height="400">

<h3><a href="https://github.com/NVIDIA/flownet2-pytorch/tree/master/networks/correlation_package" target="_blank" rel="nofollow noopener">NVIDIA/flownet2-pytorch</a></h3>

<p>flownet2-pytorch - Pytorch implementation of FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>

fitsumreda: <p>would be nice to add the released PyTorch implementations of <a href="https://github.com/NVIDIA/flownet2-pytorch/tree/master/networks/correlation_package" rel="nofollow noopener">correlation</a> and <a href="https://github.com/NVIDIA/flownet2-pytorch/tree/master/networks/resample2d_package" rel="nofollow noopener">resample2d</a> custom layers to the standard PyTorch library</p>

jondo: <p>Is <a href="https://discuss.pytorch.org/t/could-pytorch-provide-correlation-operator/84030" class="inline-onebox">Could pytorch provide correlation operator?</a> a duplicate of this topic?</p>
In Neural Language Model,what means "No symmetry in how the inputs are processed"?

ponytai1: <p>I am learning stanford CS224n.In Neural Language Model Part. The first navie model is assuming window_size = 4,concatenate word embedding [e1,e2,e3,e4] then multiplied by a matrix W ----&gt;hidden_layer----&gt;output<br>
It says one of its disadvantage is "e1 and e2 are multiplied by completely different weights in 𝑊.No symmetry in how the inputs are processed."I am stuck in these questions.</p>
<ol>
<li>
<p>I am really confused about what means “multiplied by different weights” can some one demonstrate it with some dimensionality analysis?</p>
</li>
<li>
<p>And lecturer says “you are kind of learning some similar functions many times”!.In my understanding, <strong>concatenate</strong> is very common in neural network last layer(like some fusion tasks).So is that means what i have been doing is not good?If not,what is difference between that lecturer says and our common use of <strong>Concatenate</strong> in network?<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/1/5/15bb05696178f0afc5be2878fe67fd716e386dd6.png" alt="ooooo" data-base62-sha1="36eIvj92r47c2XfaHRbX7wtIrYO" width="343" height="197"></p>
</li>
</ol>
<p>Really hope someone can help me</p>
Very Spiky Validation Curve

anotherone_one: <p>Hello there!! I am having an issue that I really don’t know how to solve and I have been trying for weeks. I have a binary classification task with two labels “Label 1” and “Label 2”. Model training conditions described in the following topics.</p>
<ul>
<li>I am using pre-trained resnet (already tried several ones, resnet18, resnet34, resnet50, resnet101). Also already tried densenet121 and the results are more a less the same.</li>
<li>Image size: to 224x224</li>
<li>My dataset is composed of 715 train images and 174 validation images. With data augmentation (random crop, random flip and colour jitter) the dataset size increases to 1430 training images. I already double and triple checked and I don’t have wrong labels or mix data between train and validation sets.</li>
<li>Tried also two different loss functions: CrossEntropyLoss and SigmoidFocalLoss.</li>
<li>The learning rate was varied from 0.01 to 0.0001.</li>
<li>The optimizer I am using is Adam, but I also already tested SGD.</li>
<li>The batch size used was 5, 16 or 32.</li>
<li>Also, I tried to add some regularization, including the dropout and weight decay.</li>
</ul>
<p>However, the weird behaviour still happens, which is this one I show in the image. I have a very unstable validation loss that usually doesn’t show a tendency of decreasing, it just decreases randomly on some of the epochs, it seems almost like the model is not learning. These types of curves are not really what is expected. I am not even sure if we can call this overfitting. However, the accuracy is not that bad, on a set of 100 images, usually, it has an accuracy of more a less 85% or more, but at the same time the images that are failing are not corner cases and it was not expected that they would fail.<br>
Does anyone have any ideas about what I could do to improve the model or does anyone even knows the reason why this could be happening?</p>
<p>Thank you in advance.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/1/6/1622b5d2c07fb8f220447b0e6ffb77f1161e691f.png" alt="Untitled" data-base62-sha1="39ORZ0S0PQGvjpJBTA93oOjK1S7" width="545" height="358"></p>

anotherone_one: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Hello! I know that you usually can solve a lot of issues here in the forum, have any thoughts about this? <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=12" title=":smiley:" class="emoji" alt=":smiley:" loading="lazy" width="20" height="20"> Thank you in advance!</p>

Andrei_Cristea: <p>Have you double-checked that the data augmentation (specifically the RandomCrop) with your particular parameters doesn’t transform the data so much that sometimes the resulting image cannot be properly labeled anymore? You can do this by just plotting say 100 training images after applying your transforms. Just worth checking, since two aggressive of a crop could result in images that the model cannot be expected to label correctly, and will essentially learn the wrong lesson from, causing loss spikes.</p>
Tensorboard model visualization

just_started_coding: <p>I have been trying to visualize the ResNet-50 model using Tensorboard graphs. I have coded the Bottleneck in two ways: one using nn.Sequential and other using a custom forward method. Though, the structure of the network is the same, why the graph visualization different? I checked the source code from pytorch and as it is implemented with custom forward method, the graph seem to be all over the place and hard to track. Also, relu is applied after batchnorm and doesn’t seem to be represented when I print bottleneck layers.</p>
<p>Here is one bottlneck layer printed: a) from pytorch source code and b) using nn.sequential</p>
<pre><code class="lang-auto">Pytorch Source Code:

(0): Bottleneck(
      (conv1): Conv2d(64, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv3): Conv2d(64, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn3): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace=True)
      (downsample): Sequential(
        (0): Conv2d(64, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )

b) nn.Sequential implementation:

 (block): Sequential(
    (0): Conv2d(64, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
    (1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (2): ReLU(inplace=True)
    (3): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
    (4): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (5): ReLU(inplace=True)
    (6): Conv2d(64, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
    (7): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (8): ReLU(inplace=True)
  )
  (downsample): Sequential(
    (0): Conv2d(64, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
    (1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (2): ReLU(inplace=True)
  )
</code></pre>
<p>Thank you!</p>

just_started_coding: <p>Still looking for the answer!</p>
Can I create a model with some layers on CPU, some on GPU?

BruceDai003: <p>I want to validate the output of each layers running on two different platforms. In order to do so, I have registered forward/backward hooks, use same input and model weights, and dumped the data. But I noticed that there is dropout or even LSTM with dropout layers in my models. The dropout algorithm is different on the two platforms, so the output tensor after dropout is different. Thus any layer’s output after the first dropout cannot be compared directly. One way to address this issue, afaik, is maybe move those dropout layers onto my CPU, and I just checked that the two CPUs dropout can produce same results. Thus, I’m wondering if it is possible to put those dropout layers on CPU and other “more deterministic” layers still on gpus. Or do you have any other suggestions to compare the layers’ outputs?</p>

ptrblck: <p>Yes, that’s possible and you could move the layers directly to the desired device via <code>model.layer.to(device)</code> and would need to move the data in the <code>forward</code> method as well:</p>
<pre><code class="lang-python">def forward(self, x):
    x = self.gpu_layer1(x)
    x = x.cpu()
    x = self.cpu_layer1(x)
    x = self.cpu_layer2(x)
    x = x.to('cuda')
    x = self.gpu_layer2(x)
    return x
</code></pre>

BruceDai003: <p>Thanks, I will give it a try.</p>