akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
FFT returns different values compare with FFTW

hoangngan-nguyen: <p>Hello everyone,</p>
<p>I am trying to use FFT for my project, my original code is written by C++ and I want to change it to Pytorch. I met the problem that the values after transformation are different compare with FFTW. .<br>
Example:<br>
tensor([[0, 1], [2, 3], [4, 5], [6, 7]]<br>
FFT:  tensor(<br>
[[28.+0.j, -4.+0.j],<br>
[-8.+8.j,  0.+0.j],<br>
[-8.+0.j,  0.+0.j],<br>
[-8.-8.j,  0.+0.j]])</p>
<p>FFTW: [[12.000000 + 16.000000j], [-4.000000  -4.000000j], [14.000000  -6.000000j], [-4.000000 + 0.000000j], [-8.000000 -8.000000j], [0.000000 + 0.000000j], [-10.000000 + 14.000000j], [0.000000 -4.000000j]].</p>
<p>Any idea what might have caused the issue and how to fix it?</p>
Correct way to store image in pytorch

JBlanchon: <p>Hi, I’m coding an EuroSat torch VisionDataset.<br>
My <code>__getitem__()</code> method return <code>Tuple[Any, Any]</code>. My images are already store in torch.Tensor before the transformation (for RGB image I use <code>torchvision.io.read</code>_image that directly return Tensor and for full channel image I use gdal). But I see many torchvision dataset that store images as numpy array or even PIL Image. It seems to be very common as many people apply a T.ToTensor() on the dataset.</p>
<p>So what is the best practice in pytorch, directly using torch.Tensor and avoiding a ToTensor transform or using np array ?</p>

akvilonBrown: <p>Hi, I’m not knowledgeable enough about performance/size aspects. Still, if you will share your dataset with someone or reuse it later with other frameworks, it would be inconvenient to install PyTorch just to read it.</p>

OrielBanne: <p>depends what you try to do. if you are on GPU or Torch in general with no need to plot the image - keep it in torch tensor. But when you want to actually see the image, a tensor will not be the sort of image you can plot. PIL can be plotted, and various other formats as well.</p>

tom: <aside class="quote no-group" data-username="JBlanchon" data-post="1" data-topic="134202">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jblanchon/48/42986_2.png" class="avatar"> JBlanchon:</div>
<blockquote>
<p>So what is the best practice in pytorch, directly using torch.Tensor and avoiding a ToTensor transform or using np array ?</p>
</blockquote>
</aside>
<ul>
<li>If you have tensors and expect to read it in one go, I think it is very reasonable to just store them as tensors. It is a speed thing, too. Going from read from disk + CPU augmentation to store in memory + GPU augmentation can give a formidable speedup.<br>
(I did this as live coding in a <a href="https://github.com/t-vi/lod2021" rel="noopener nofollow ugc">workshop last week</a> - in the grapevine notebooks.)</li>
<li>You won’t be able to read .pt files without PyTorch. To my mind, this is not too bad. They do have the advantage that they promise libtorch compatibility when you don’t have Python.</li>
<li>You can always use torchvision’s ToPIL transform if you want to show images. Typically, you have magnitudes more images going straight to the GPU than are being output.</li>
<li>There is a small caveat: Depending on the hardware characteristics, I have seen that loading smaller files + doing some decoding (preferably on the GPU) can be faster than loading larger files that have been preprocessed. (I saw this with ct scans and int16 storage vs. fp32 and the storage interface was the main limitation, so reading half as many bytes from disk was preferable.)</li>
</ul>
<p>Best regards</p>
<p>Thomas</p>

JBlanchon: <p>Thanks Tom,</p>
<p>So I suppose even if most people and most pytorch dataset return numpy ndarray it’s a always a better practice to directly use torch tensor.</p>
<p>I looked at your workshop, it is very well written. Thanks a lot</p>
Correct Loss for bounded Multiple Regression

SumNeuron: <p>Suppose I have target vector <code>y= (y_1, y_2, ..., y_n)</code> where <code>y_i in [0, inf)</code> for all <code>i</code> e.g.</p>
<pre><code class="lang-python"># here n = 6 and n is constant for all data
y = [
  0,
  0,
  2,
  1,
  0,
  1
]
</code></pre>
<p>In general <code>y_i</code> both integers and in practice almost always are in <code>[0,10]</code>. In theory, however they can take any non negative real value.</p>
<p>What is the best way to handle this for training and which loss should I use?<br>
Should I treat it like classification and refractor <code>y</code> so the above example becomes</p>
<pre><code class="lang-python"># assuming zero indexed
 [
  [1, 0, 0, 0, 0, 0, 0, 0, 0, 0], # 0
  [1, 0, 0, 0, 0, 0, 0, 0, 0, 0], # 0
  [0, 1, 0, 0, 0, 0, 0, 0, 0, 0], # 1
  [0, 1, 0, 0, 0, 0, 0, 0, 0, 0], # 1
  [1, 0, 0, 0, 0, 0, 0, 0, 0, 0], # 0
  [0, 1, 0, 0, 0, 0, 0, 0, 0, 0], # 1
]
</code></pre>
Random batch transforms

kpedro88: <p>I have a batch of images in the usual tensor form (B, C, H, W). To apply the usual sets of rotations and flips, I’m using the construction here: <a href="https://github.com/pytorch/vision/issues/566#issuecomment-814221986" rel="noopener nofollow ugc">https://github.com/pytorch/vision/issues/566#issuecomment-814221986</a>.</p>
<p>However, this isn’t quite doing what I want. <a href="https://pytorch.org/vision/stable/transforms.html" class="inline-onebox" rel="noopener nofollow ugc">Transforming and augmenting images — Torchvision 0.16 documentation</a> states:</p>
<blockquote>
<p>Randomized transformations will apply the same transformation to all the images of a given batch</p>
</blockquote>
<p>I want a <em>different</em> random transform to be applied to <em>each</em> image in the batch, but I would still like to be able to process all the images together in a batch (avoiding an explicit loop). Is there any method available to do this?</p>

edgarriba: <p>As an alternative you can use kornia for your use case: <a href="https://kornia.readthedocs.io/en/latest/applications/image_augmentations.html" class="inline-onebox" rel="noopener nofollow ugc">Image Augmentation - Kornia</a></p>
<p>we provide <code>same_on_batch</code> flag which you can turn on/off</p>
Custom dataset and DataLoader

WAN: <p>Hello,<br>
I want to use a custom dataset with DataLoader. My dataset contains text and labels. Any suggestion?</p>

AbdulsalamBande: <p>Welcome Wan<br>
Sure, you can try the below</p>
<pre><code class="lang-auto">import torch
from torch.utils.data import Dataset
class CustomDataloader(Dataset):
    def __init__(self, text, labels):
        self.text = text
        self.labels = labels

    def __len__(self):
        return len(self.text)

    def __getitem__(self, idx):
        text = self.text[idx]
        labels = self.labels[idx]
        sample = {"text": text, "labels": labels}
        return sample

all_text = []
all_labels = []

#Generate data
for i in range(4):
    all_text.append(torch.rand(2,1))
    if i%1 ==0:
        all_labels.append(1)
    else:
        all_labels.append(0)

DataClass = CustomDataloader(all_text,all_labels)
trainloader = torch.utils.data.DataLoader(DataClass,batch_size=2)
#get the data for training
for i ,data in enumerate(trainloader):
    text = data['text']
    labels = data['labels']
    print(text)
    print(labels)
</code></pre>
Masking in torch.nn.MultiheadAttention

legoh: <p>Hi there! I am using the nn.MultiheadAttention to construct a transformer encoder layer. Suppose my queries, keys, and values, are all the same (e.g. <code>h</code>), meaning I call it via <code>h, score = MHA(h, h, h)</code>.  This means that for some of the computation, there is some form of self-attention going on. Is there a way to mask this away?</p>

AbdulsalamBande: <p>I think the <a href="https://discuss.pytorch.org/t/masking-in-torch-nn-multiheadattention/132060">torch.nn.MultiheadAttention</a> has a mask argument</p>
<blockquote>
<p><code>forward</code> (<em>query</em> , <em>key</em> , <em>value</em> , <em>key_padding_mask=None</em> , <em>need_weights=True</em> , <em>attn_mask=None</em> )</p>
</blockquote>
<p>You can watch this video for an explanation <a href="https://www.youtube.com/watch?v=_iDanMWVj98&amp;ab_channel=PytorchModulesExplained" rel="noopener nofollow ugc">https://www.youtube.com/watch?v=_iDanMWVj98&amp;ab_channel=PytorchModulesExplained</a></p>
Accuracy in the test set is highly reduced using model.eval()

tslermen: <p>Hey all</p>
<p>I’m using a EfficientNetB0 as backbone of my model and fine tuning it to three classes using a nn.Softmax() as a head with nn.Linear(). The model is performing as I expected in the training and validation sets. Even though, when I load the model and run it with the test set, the accuracy is highly decreased.</p>
<p>During the validation set, I am using</p>
<p><code>with torch.set_grad_enabled(False):     for i, data in enumerate(val_dl, 0): </code><br>
Which works fine.</p>
<p>During the test, I am using</p>
<p><code>model.eval()     for data in test_dl: </code><br>
Which works really bad.</p>
<p>When i try to use <code>with torch.set_grad_enabled(False):</code> it returns an error that has to do with the Batch norm.</p>
<p>I have no idea why it’s happening, I’ve checked the results of the validation step and it’s working well?</p>
<p>Maybe the problem is when i save the model?</p>
<pre><code class="lang-auto">save_model = True
    if save_model:
        print(f"--&gt; Saving the model at 'saves/{model_name}.pth'")
        torch.save(model, os.path.join('saves', model_name+'.pth'))
</code></pre>
<p>When I load the model?</p>
<pre><code class="lang-auto">model = torch.load('saves/' + model_name + '.pth')
</code></pre>
Mobilebert is so slow and doesn't converge

Fahimeh: <p>Has anyone had any issue with mobileBERT? I use mobileBERT and DistillBERT on Hugging face with the same code for text classification. DistillBert quickly converges and gives me a good result! But mobileBERT is so slow and doesn’t converge at all. I couldn’t find any good tip in mobileBERT paper … Is there anything else which I miss?<br>
<a class="mention" href="/u/ptrblck">@ptrblck</a> : Do you have any suggestions for that? Thanks.</p>

ptrblck: <p>I’m not deeply familiar with <code>MobileBERT</code> but by skimming through some blog posts it seems that the <code>MobileBERT</code> model needs a teacher model for the training? Could it be the case that your training script is indeed training two models (a larger teacher and the smaller <code>MobileBERT</code> model)?<br>
I’m also sure, that <a class="mention" href="/u/stas">@stas</a> might be more familiar with these implementations and could help out.</p>

stas: <p>No experience with MobileBert here, but is there any reason why you’re posting this here, instead of <a href="https://discuss.huggingface.co" rel="noopener nofollow ugc">https://discuss.huggingface.co</a> or <a href="https://github.com/huggingface/transformers/issues" class="inline-onebox" rel="noopener nofollow ugc">Issues · huggingface/transformers · GitHub</a>?</p>
<p>This is very HF-specific question and you will have a much better turnaround for getting answers by asking these to the right audiences <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=10" title=":wink:" class="emoji" alt=":wink:"></p>
<p>When you make a post there, it’d help for you to provide some details for others to know what you’re doing specifically. i.e. how you train, which versions of main components you use, etc. Think what the other person needs to know in order to understand what you’re trying to do.</p>

Fahimeh: <p>Thanks for your help.</p>
How to disable CUDNN in Libtorch

soulslicer: <p>I am using the libtorch C++ API. How can I programatically disable CUDNN for testing purposes?</p>

ptrblck: <p>I think this should work:</p>
<pre><code class="lang-auto">at::globalContext().setUserEnabledCuDNN(false);
</code></pre>

soulslicer: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="134167">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p><code>at::globalContext().setUserEnabledCuDNN(false);</code></p>
</blockquote>
</aside>
<p>I am doing a quick test with <code>torch::conv2d</code> on a simple RGB image, but whether i enable or disable CUDNN, the runtime seems to be the same.</p>
<p><code>torch::cuda::cudnn_is_available()</code> is true</p>
<p>EDIT:</p>
<p>No CUDNN is faster, just needed to run more iterations and take an average</p>

ptrblck: <p><code>cudnn_is_available()</code> would not return <code>False</code>, if you disable the usage of cuDNN as in fact it’s still available.<br>
You can profile the code via e.g. Nsight Systems or the PyTorch profiler to see the kernels.</p>
Replacing tensor values with other tensor values at specific indices

Vladimir_Blagojevic: <p>Hi everyone,</p>
<p>How does one replace tensor values in tensor A, with values from tensor B at indicated indices from tensor C? I am dealing with simple 1 dim tensors. Tensor A has shape [120], B [n] and C [n] where n is less than 120.</p>
<p>Thanks,<br>
Vladimir</p>

Vladimir_Blagojevic: <p>Found it - <code>torch.scatter</code> does exactly what I want!</p>
How can I see the architecture of a neural network

kiriakospapa: <p>Hello, I have been searching for some hours how to print the architecture of a neural network and I couldn’t find any function.</p>
<p>For example, I want to have a output like this for a linear neural network with 8 inputs 16 units in the hidden layer and 1 in the output: <code>Linear 8, Linear 16, Linear 1</code></p>
<p>Is there any function or do I have to do it manually?</p>
<p>Thank you:)</p>
Where and How to add Dropout in ResNet18

DL-ljw: <p>I want to add dropout in Resnet,but don’t know where to add.<br>
Reference to WideResnet , i put drop in the BasicBlock class,and art of my code is:</p>
<pre><code class="lang-auto">class BasicBlock(nn.Module):
    expansion = 1

    def __init__(self, inplanes, planes, stride=1, downsample=None,dropRate=0.5):
        super(BasicBlock, self).__init__()
        self.conv1 = conv3x3(inplanes, planes, stride)
        self.bn1 = nn.BatchNorm2d(planes)
        self.relu = nn.ReLU(inplace=True)
        self.conv2 = conv3x3(planes, planes)
        self.bn2 = nn.BatchNorm2d(planes)
        self.downsample = downsample
        self.stride = stride
        self.droprate = dropRate

    def forward(self, x):
        residual = x

        out = self.conv1(x)
        out = self.bn1(out)
        out = self.relu(out)

        out = self.conv2(out)
        out = self.bn2(out)

        if self.downsample is not None:
            residual = self.downsample(x)

        out += residual
        out = self.relu(out)
        **if self.droprate &gt; 0:**
**            out = F.dropout(out, p=self.droprate, training=self.training)**

        return out
</code></pre>
<p>But the question is the value of “p” had no impact on the performance,so i I suspect that the way I use dropout is wrong. So my question is where and how to add dropout in  ResNet.<br>
Really hope to get help ,thanks!</p>

singleroc: <p>Hi  I’m also interested in the problem. Is there any update on this?</p>

gabrieldernbach: <p>From what I saw it seems most common to place dropout after each relu.<br>
This code goes recursively through each block.</p>
<pre><code class="lang-auto">    model = resnet18()

    def append_dropout(model, rate=0.2):
        for name, module in model.named_children():
            if len(list(module.children())) &gt; 0:
                append_dropout(module)
            if isinstance(module, nn.ReLU):
                new = nn.Sequential(module, nn.Dropout2d(p=rate, inplace=True))
                setattr(model, name, new)


    append_dropout(model)
    print(model)
</code></pre>
CUDNN_STATUS_EXECUTION_FAILED help needed

praggya: <p>Hi there,<br>
I don’t know why I am CUDNN_STATUS_EXECUTION_FAILED error here<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/e/ee9b8ca0d6863fc717ec71e9af418cb27bf2c7e5.png" data-download-href="https://discuss.pytorch.org/uploads/default/ee9b8ca0d6863fc717ec71e9af418cb27bf2c7e5" title="Screenshot (269)"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/e/ee9b8ca0d6863fc717ec71e9af418cb27bf2c7e5_2_690x388.png" alt="Screenshot (269)" data-base62-sha1="y2OTQVZvriXt5NcXen7oNtc1rlb" width="690" height="388" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/e/ee9b8ca0d6863fc717ec71e9af418cb27bf2c7e5_2_690x388.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/e/ee9b8ca0d6863fc717ec71e9af418cb27bf2c7e5_2_1035x582.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/e/e/ee9b8ca0d6863fc717ec71e9af418cb27bf2c7e5_2_1380x776.png 2x" data-dominant-color="0A2B59"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot (269)</span><span class="informations">1920×1080 173 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Please help me resolve this issue ASAP</p>

tom: <p>My first guess would be memory problems, either GPU or CPU.</p>

praggya: <p>i am currently working on 3090 <a class="mention" href="/u/tom">@tom</a></p>
Initializing a differentiable param in pytorch

blade: <p>I’m trying to define a set of new parameters <code>B</code> in a pytorch model. I would like to initialize the new params with the current weights of the model <code>W</code>.</p>
<p><strong>Question:</strong> I want the params <code>B</code> to be differentiable, but autograd should not track their history to <code>W</code> (so <code>B</code> should have a new memory with no reference to <code>W</code>). Is <code>B = nn.Parameter(W.detach().clone())</code> the correct function to use?</p>
<p>I understand <code>B = W.clone()</code> will result in autograd tracking history of <code>B</code> to <code>W</code> while differentiating. Also I understand that <code>B = W.detach().clone()</code> will not be differentiable.</p>
'None' Type Object has no attribute data

praggya: <p>def _backward_step_unrolled(self,input_train_source, class_label_source_train,domain_label_source_train, input_train_target,class_label_target_train,domain_label_target_train,input_valid_source,target_valid_source,input_valid_target,target_valid_target, eta, network_optimizer,feature_extractor_optimizer,head_g_optimizer,init_channels):</p>
<pre><code>unrolled_model = self._compute_unrolled_model(input_train_source,input_train_target, domain_label_source_train,domain_label_target_train, eta, network_optimizer)

unrolled_feature_extractor,unrolled_head_g=self._compute_unrolled_model1(input_train_source,input_train_target,class_label_source_train,class_label_target_train,unrolled_model,eta,feature_extractor_optimizer,head_g_optimizer,init_channels)



unrolled_loss=cal_loss(unrolled_model,unrolled_feature_extractor,unrolled_head_g,input_valid_source,input_valid_target,target_valid_source,target_valid_target)

unrolled_loss.backward()

dalpha = [v.grad for v in unrolled_model.arch_parameters()]

**vector = [v.grad.data for v in unrolled_model.parameters()]**

implicit_grads = self._hessian_vector_product(vector, input_train_source, input_train_target,domain_label_source_train,domain_label_target_train)

for g, ig in zip(dalpha, implicit_grads):

  g.data.sub_(eta, ig.data)

for v, g in zip(self.model.arch_parameters(), dalpha):

  if v.grad is None:

    v.grad = Variable(g.data)

  else:

    v.grad.data.copy_(g.data)</code></pre>

praggya: <p>Can anyone please tell me ASAP,why the line within ** ** giving this error.<br>
<img src="https://discuss.pytorch.org/images/emoji/apple/cold_sweat.png?v=10" title=":cold_sweat:" class="emoji only-emoji" alt=":cold_sweat:"></p>

ptrblck: <p>Some parameters of <code>unrolled_model</code> don’t have a valid gradient and thus the <code>.grad</code> attribute is still set to <code>None</code>.<br>
This might be caused e.g. if this parameter wasn’t used in the forward pass and is thus not in the computation graph.</p>
<p>Btw. don’t use the <code>.data</code> attribute, as it could yield unwanted side effects.</p>

praggya: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> this is how I am doing the forward pass(calculating the loss):</p>
<p>def cal_loss(unrolled_model,unrolled_feature_extractor,unrolled_head_g,input_valid_source,input_valid_target,target_valid_source,target_valid_target):</p>
<p>domain_features_source,domain_logits_s=unrolled_model(input_valid_source)</p>
<p>domain_features_target,domain_logits_t=unrolled_model(input_valid_target)</p>
<p>generalized_features_source = unrolled_feature_extractor(input_valid_source)</p>
<p>generalized_features_target = unrolled_feature_extractor(input_valid_target)</p>
<p>logits1=unrolled_head_g(generalized_features_source-domain_features_source)</p>
<p>logits2=unrolled_head_g(generalized_features_target-domain_features_target)</p>
<p>crit = nn.CrossEntropyLoss()</p>
<p>loss1 = crit(logits1, target_valid_source)</p>
<p>loss2 = crit(logits2, target_valid_target)</p>
<p>loss=loss1+loss2</p>
<p>return loss</p>
How to know if mixed precision is used/useful?

Olivier-CR: <p>Hi,</p>
<p>I’m training DeepLabV3 on 1208*1920 images, batch 2 on a T4 GPU.<br>
GPU is close to 100% usage all time in nvidia-smi<br>
training time is the same with and without <a href="https://pytorch.org/docs/stable/notes/amp_examples.html" rel="noopener nofollow ugc">amp</a> (autocast + scaler). Why is there no benefit? How can I see if float16 is used? in T4 I expected a decent speed up given the presence of tensorcores, surprise that it doesn’t change anything</p>
Two methods of anchor-target matching

sunshine1: <p>In series of yolo object detector, anchors play an important role. In some code, the target box is predicted by the best anchor which match it. It means that one target box is predicted by only one anchor. But,in other code, the anchors whose iou with the target box are greater than some threshold  will predict the target box. It means that one target box may be predicted by multiply  anchors. I want to know which is better in these two methods?</p>
'tuple' object is not callable in Model.forward

laoweijianglin: <p>class Model(nn.Module):<br>
def <strong>init</strong> (self):<br>
super(Model, self).<strong>init</strong>()<br>
self.conv1 = nn.Sequential(<br>
nn.Conv2d(1, 32, kernel_size = (3,3), stride = (1,1),padding = (0,1)),<br>
nn.BatchNorm2d(num_features =32),<br>
nn.ReLU(inplace = True),<br>
nn.AvgPool2d(kernel_size = (2,2), stride =(2,2)),<br>
)</p>
<pre><code>    self.conv2 = nn.Sequential(
        nn.Conv2d(32, 64, kernel_size = (3,3), stride = (1,1), padding = (1,1)),
        nn.BatchNorm2d(num_features =64),
        nn.ReLU(inplace = True),
        nn.Dropout(p=0.10),
        nn.AvgPool2d(kernel_size = (2,2), stride =(2,2)),
    )    
        
    self.conv3 = nn.Sequential(
        nn.Conv2d(64, 64, kernel_size = (3,3), stride = (1,2)),
        nn.BatchNorm2d(num_features =64),
        nn.ReLU(inplace = True),
        nn.AvgPool2d(kernel_size = (3,3), stride =(1,1)),
    )
    
    self.Linear1 = nn.Linear(1*64*122,4),
    
def forward(self, x):
    out = self.conv1(x)
    out = self.conv2(out)
    out = self.conv3(out)
    N, C, H, W = out.size()     
    out = out.view(N, -1)
    output = self.Linear1(out)
    return output
</code></pre>
<p>model = Model()</p>
<p>When I run the code, it told me that</p>
<p>‘’’’<br>
output = self.Linear1(out)</p>
<p>TypeError: ‘tuple’ object is not callable<br>
‘’’’’</p>

tymokvo: <p>It’s a syntax problem. You have a comma after</p>
<aside class="quote no-group quote-modified" data-username="laoweijianglin" data-post="1" data-topic="15090">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/l/cab0a1/48.png" class="avatar"> laoweijianglin:</div>
<blockquote>
<p>self.Linear1 = nn.Linear(1<em>64</em>122,4),</p>
</blockquote>
</aside>
<p>The interpreter thinks that this is a tuple. Just delete it and you should be good to go.</p>
<p><img src="https://discuss.pytorch.org/images/emoji/apple/smile_cat.png?v=12" title=":smile_cat:" class="emoji only-emoji" alt=":smile_cat:" loading="lazy" width="20" height="20"></p>

laoweijianglin: <p>Yeeeaaahhhhhhhhh! It works! Thank you very much!</p>

Nikhil_Chowdary_Gutl: <p>I did the same mistake <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=9" title=":sweat_smile:" class="emoji" alt=":sweat_smile:"> and thank you for your answer</p>

parzival: <p>hero! of! the! day!!</p>
Indicies of downloaded dataset samples

ElQadasi: <p>I have this code to download CIFAR 10:</p>
<p>trainset = torchvision.datasets.CIFAR10(root=’./data’, train=True, download=True, transform=transform)<br>
trainloader = torch.utils.data.DataLoader(trainset, batch_size=batch_size, shuffle=True)</p>
<p>for indices, train_data, train_label in trainloader:<br>
**read the indices of the current batch? **</p>
<p>I do not want to design my own dataset class from scratch because I think I should store the images on the local machine to create my dataset subclass. Is there a way to get the indices directly?</p>
<p>Thanks</p>
Torch.nn.instancenorm

wei61547-jp: <p>why running_mean/var’s shape is C, not N * C<br>
according to the <a href="https://arxiv.org/abs/1607.08022" class="inline-onebox" rel="noopener nofollow ugc">[1607.08022] Instance Normalization: The Missing Ingredient for Fast Stylization</a>, the shape of mean/var is N *C , but for torch.nn.instancenorm ,<br>
at::alias(running_mean).copy_(running_mean_.view({ b, c }).mean(0, false)); }<br>
i don’t know the operation of “mean(0,false)” is use for ?<br>
<a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/Normalization.cpp#L581" class="inline-onebox" rel="noopener nofollow ugc">pytorch/Normalization.cpp at master · pytorch/pytorch · GitHub</a></p>

wei61547-jp: <p>Who can tell me ? Anyone know that?</p>
Grad value when copying tensor

torchimport: <p>I have a question about grad attribute.</p>
<p>I observed that grad value is not copied when using tensor slicing, clone(), to(), cuda() etc.</p>
<p>how can I copy tensors with grad value?</p>
<p>Below is the example:</p>
<pre><code class="lang-auto">a = torch.tensor([1., 2.], requires_grad=True)
loss = a.sum()
loss.backward()
b = a.cuda()    # change device
c = a[:1]       # slicing
d = a.clone()   # clone

print(a.grad)  # [1., 1.]
print(b.grad)  # None
print(c.grad)  # None
print(d.grad)  # None
</code></pre>
<p>Thank you.</p>

ptrblck: <p>The mentioned operations would create a differentiable operation and are thus not copying the <code>.grad</code> attribute. To do so, you could manually copy it back to the <code>.grad</code> attribute of the new tensor.</p>

torchimport: <p>I solved this issue with your help.</p>
<p>Thank you!</p>
How do I update torch::jit::IValue's value

jianlongf: <ol>
<li>First  As you know, aten use at::IValue as input &amp; output. So I wanna use layer 1 output to update layer 2 input. I don’t wanna use new created IValue, but update old one’s value by new one.(because they are the same  at::IValue).</li>
<li>Can at::IValue（for example A） store list of other at::IValues(like B C D)? If I modify other’s(B C D) at::IValue, then A would be changed coorespondly.</li>
</ol>

jianlongf: <p>The First question could be deal with at::IValue swap method. But I have no idea about second.</p>
ValueError: too many values to unpack (expected 2) by Changing num_layers to 2

Snowbow: <p>Hello, i’m new to Pytorch and nlp.<br>
I’m doing a Reinforcement Learning Project and I’m using GRU as the network. But when I changing the “num_layers” to 2, it comes a ValueError:<br>
“ValueError: too many values to unpack (expected 2)”</p>
<p>Here is my code:</p>
<pre><code class="lang-auto">class ActorNetwork(nn.Module):
    def __init__(self, input_size, alpha, hidden_size=64, num_layers=2):
        super(ActorNetwork, self).__init__()
        self.actor_rnn = nn.GRU(         
            input_size=input_size,
            hidden_size=hidden_size,         # rnn hidden unit
            num_layers=num_layers,           # number of rnn layer
            bidirectional=True,     
            batch_first=True,
            dropout=0.2
        )
        self.actor = nn.Linear(hidden_size * 2, 1)
        self.optimizer = optim.Adam(self.parameters(), lr=alpha)
        self.device = T.device('cuda:0' if T.cuda.is_available() else 'cpu')
        self.to(self.device)

    def init_weight(self):
        for name, param in self.actor_rnn.named_parameters():
            if 'bias' in name:
                nn.init.constant_(param, 1.0)
            elif 'weight' in name:
                nn.init.orthogonal_(param, gain=1)

    def forward(self, state):
        r_out, (h_n, h_c) = self.actor_rnn(state, None)  
        act_pri = self.actor(r_out)
        dist = T.squeeze(act_pri)  # from 3 dim to 1
        # if there is only 1 dim, it will be squeezed to a scale
        if dist.dim() == 0:
            dist = T.unsqueeze(dist, 0)
        dist = Categorical(logits=dist)
        return dist
</code></pre>
<p>But when the parameter num_layers is 1, the code can run normally.</p>

ptrblck: <p>As described in the <a href="https://pytorch.org/docs/stable/generated/torch.nn.GRU.html?highlight=gru#torch.nn.GRU">docs</a> <code>nn.GRU</code> yields two outputs: <code>output</code> and <code>h_n</code>.<br>
You are unwrapping the second return value into <code>h_n</code> and <code>h_c</code>, which works fine if a single layer is used, since the shape of <code>h_n</code> is defined as <code>[directions * num_layers, batch_size, H_out]</code> (<code>directions=2</code> since you are using <code>bidirectional=True</code>).<br>
However, if you are using <code>num_layers=2</code> then the first dimension will have a size of 4 instead of 2 so unwrapping this tensor to <code>h_n</code> and <code>h_c</code> won’t work and you might want to assign it to <code>h_n</code> only and split it afterwards if needed.</p>

Snowbow: <p>Thank you for the Answer! That solve my problem.</p>
How to convert a 55-color PNG segmentation mask in a [55,h,w] tensor?

Olivier-CR: <p>Hi,<br>
I’m trying to train a DeepLabV3 model on a segmentation dataset where annotations come as PNG pictures with 55 different colors.</p>
<p>When I read such an annotation PNG with PyTorch image_read I get the following tensor (shape [3,H,W])</p>
<pre><code class="lang-auto">tensor([[[135, 135, 135,  ..., 135, 135, 135],
         [135, 135, 135,  ..., 135, 135, 135],
         [135, 135, 135,  ..., 135, 135, 135],
         ...,
         [255, 255, 255,  ..., 200, 200, 200],
         [255, 255, 255,  ..., 200, 200, 200],
         [255, 255, 255,  ..., 200, 200, 200]],

        [[206, 206, 206,  ..., 206, 206, 206],
         [206, 206, 206,  ..., 206, 206, 206],
         [206, 206, 206,  ..., 206, 206, 206],
         ...,
         [  0,   0,   0,  ..., 125, 125, 125],
         [  0,   0,   0,  ..., 125, 125, 125],
         [  0,   0,   0,  ..., 125, 125, 125]],
</code></pre>
<p>I think my DeepLab model will need this as a 1-hot tensor, {55, H, W}.<br>
I cannot find anywhere any code to do that transformation from {3, H, W} to {55, H, W}. Anybody has a snippet of code to share? How segmentation datasets usually handle this?</p>

ptrblck: <p><a href="https://discuss.pytorch.org/t/training-semantic-segmentation/49275/4">This post</a> gives you an example of a color to class index mapping for the targets of a segmentation use case.</p>

Olivier-CR: <p>wow absolutely epic thanks <a class="mention" href="/u/ptrblck">@ptrblck</a> <img src="https://discuss.pytorch.org/images/emoji/apple/star_struck.png?v=10" title=":star_struck:" class="emoji" alt=":star_struck:"></p>
Get the value at a specific index in PyTorch

arneo: <p>I am stuck in this a bit for quite some time.</p>
<p>I have a ground truth label array for size 5.</p>
<p><code>    y=tensor([903,  815, 214, 282, 694])</code></p>
<p>I have the output for scores array  of shape :  [5,1000]</p>
<pre><code class="lang-auto">    scores   =  tensor([[ 1.0406,  1.1808,  4.4227,  ...,  4.6864,  8.0145,  5.2128],
            [ 6.9101,  4.6083,  6.9259,  ...,  9.7415,  9.6305,  9.3974],
            [ 7.6097,  4.0396,  4.4560,  ...,  3.4892, 11.6411, 2],
            [ 1.0693,  4.6295,  5.3638,  ..., 10.9041, 10.8380,  9.2077],
            [ 1.7085,  1.4938,  8.6876,  ..., 15.1423,  9.6055,  9.8920]],
           grad_fn=&lt;ViewBackward&gt;)
</code></pre>
<p>I want the value from scores array based on the corresponding index of y. So for y[0], which is 903, I want the corresponding value from scores[1], position 903.</p>
<p>Is there some direct Pytorch function I can use.</p>

ptrblck: <p>Assuming you would like to get:</p>
<pre><code class="lang-python">x[0, 903]
x[1, 815]
...
</code></pre>
<p>then this should work:</p>
<pre><code class="lang-python">x = torch.randn(5, 1000)
idx = torch.tensor([903, 815, 214, 282, 694])

x[torch.arange(x.size(0)), idx]
</code></pre>
Installation with conda is unbearably slow?

miturian: <p>I realize this is an odd problem, but I am trying to reinstall conda on my windows machine, using:</p>
<pre><code>conda install pytorch torchvision torchaudio cudatoolkit=11.1 -c pytorch -c conda-forge
</code></pre>
<p>but it is unbearably slow?</p>
<p>In the conda prompt, I get the following:</p>
<blockquote>
<p>Collecting package metadata (current_repodata.json): done<br>
Solving environment: failed with initial frozen solve. Retrying with flexible solve.<br>
Solving environment: done<br>
Collecting package metadata (repodata.json): done<br>
Solving environment: failed with initial frozen solve. Retrying with flexible solve.<br>
Solving environment:</p>
</blockquote>
<p>with no progress after 30 minutes. I am sitting on an excellent internet connection in northern europe (<a href="http://speedtest.net" rel="noopener nofollow ugc">speedtest.net</a> just gave me 200+ mbps).<br>
any ideas for adding mirrors or something to speed it up?</p>

zetyquickly: <p>Hey <a class="mention" href="/u/miturian">@miturian</a></p>
<p>That’s happening to me sometimes and I don’t know the answer why. Maybe the problem with inconsistent package dependencies.</p>
<p>The solution for you might be to create a new environment with <code>conda install -n new_env python=3</code> and then install <code>pytorch</code> on it along with packages you need for your research</p>

ptrblck: <p>As <a class="mention" href="/u/zetyquickly">@zetyquickly</a> explained, the issue is not your internet connection or the conda server, but the slow dependency resolution via <code>conda</code>. Using a new (and clean) environment could solve it.</p>
Trying to run cuda on AMD using ROCM

VIGNESHinZONE: <p>Im unable to run any of the usual cuda commands in pytorch like <code>torch.cuda.is_available()</code> or <code>tensor.to("cuda")</code> using the ROCM library.</p>
<p>I had installed it using the following docker image <a href="https://hub.docker.com/r/rocm/pytorch" class="inline-onebox" rel="noopener nofollow ugc">Docker Hub</a></p>
<p>Building the image-</p>
<pre><code class="lang-auto">docker pull rocm/pytorch
</code></pre>
<p>Running the container -</p>
<pre><code class="lang-auto">docker run -i -t 6b8335f798a5 /bin/bash
</code></pre>
<p>I assumed that we could directly use the usual GPU commands like we did using ROCM but doesn’t seem to work. Any help would be helpful</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.cuda.is_available()
False
&gt;&gt;&gt;torch.tensor([1.0, 2.0, 3.0]).to("cuda")

Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "/opt/conda/lib/python3.6/site-packages/torch/cuda/__init__.py", line 172, in _lazy_init
    torch._C._cuda_init()
RuntimeError: No HIP GPUs are available

</code></pre>
Multiclass classification for help

oneluxyou: <p>HEy guys,I’m a fresh learner and I have lots 5 * 1 inputs and labels(range(1-10)) like this:<br>
inputs:tensor([[ 184.3800,    5.0000, 1129.9000,    0.0000, -260.6700]]) labels:tensor([5])<br>
I want to know is there any good model to classify it to its class<br>
Here is my poor model<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/e/9/e9d8afa13ea85cfaa1ea510703104ba0f991e74d.png" alt="16341103951079" data-base62-sha1="xmHuTDsCP4D8Arn9S78E72DKK8B" width="433" height="323"></p>
Shape requirements of conv1d seems very permissive

b-cute: <p>Hi all,</p>
<p>Sorry for the potential noob post, but I encountered today what is not strictly a problem, but rather an unexpected behavior.</p>
<p>When using a <code>torch.nn.functional.conv1d()</code>, it is specified in the documentation that the input tensor shape should be of dimension <em>[minibatch, in_channels, iW]</em>. But it seems to be also working with a 4d input tensor provided your weights are defined accordingly:</p>
<blockquote>
<p>import torch<br>
import torch.nn.functional as F</p>
<p>test = torch.ones(1,3,2,2)<br>
weights = torch.randn(4,3,1,1)<br>
F.conv1d(test, weights)</p>
</blockquote>
<p>outputs</p>
<blockquote>
<p>tensor([[[[-0.8317, -0.8317],<br>
[-0.8317, -0.8317]],</p>
<pre><code>     [[-0.7135, -0.7135],
      [-0.7135, -0.7135]],

     [[ 0.3278,  0.3278],
      [ 0.3278,  0.3278]],

     [[-1.8960, -1.8960],
      [-1.8960, -1.8960]]]])
</code></pre>
</blockquote>
<p>That’s to say exactly the same as</p>
<blockquote>
<p>F.conv2d(test, weights)<br>
tensor([[[[-0.8317, -0.8317],<br>
[-0.8317, -0.8317]],</p>
<pre><code>     [[-0.7135, -0.7135],
      [-0.7135, -0.7135]],

     [[ 0.3278,  0.3278],
      [ 0.3278,  0.3278]],

     [[-1.8960, -1.8960],
      [-1.8960, -1.8960]]]])
</code></pre>
</blockquote>
<p>In other words, it seems like PyTorch is switching to conv2d under the hood because of the input tensor and weights shape. Is that true? Is there any reason not to simply always use conv1d by default?</p>
<p>Thanks for your time.</p>

ptrblck: <p>I guess some checks might not be triggered for an unexpected shape as long as it’s not invalid and based on <a href="https://github.com/pytorch/pytorch/blob/63b180beed0453141ab11d1df0527d808801e583/aten/src/ATen/native/Convolution.cpp#L568-L599">this code</a> it seems that besides a different <code>output_padding</code>? argument the internal call would be equal.<br>
I would still recommend to use the corresponding function, as internals could be changed without any notice and would thus not depend on the equal results in your code snippet.</p>
Train the same model twice in on loop

KeatsHao: <p>I want to train the same model twice in one loop.Why it still says the computation graph has been freed. What I can do to finish the logic like below. Thanks,everyone.</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

model  = nn.Linear(1,1)
optimizer = torch.optim.Adam(model.parameters(),lr=0.1,betas=(0.5, 0.999))

for i in range(200):
    optimizer.zero_grad()
    input = torch.tensor([1])
    output = model(input)
    loss_1 = 100 - output
    loss_1.backward()
    optimizer.step()

    optimizer.zero_grad()
    input = torch.tensor([2])
    output = model(input)
    loss_2 = 200 - output
    loss_2.backward()
    optimizer.step()
</code></pre>

ptrblck: <p>Your code doesn’t raise the mentioned error in my setup, but fails with a dtype mismatch since you are using <code>LongTensor</code>s as the input.<br>
Use <code>input = torch.tensor([1.])</code> and it should work.</p>
How to load complete dataset to RAM？

sunnnnnnnny: <p>I have a network training that is particularly slow now, but I have enough memory to read the completed training set at one time to improve the training speed. How should I achieve it?</p>

JuanFMontesinos: <p>If you are using pytorch dataset/dataloader, in the dataset __init__method, load all the data. If you want to load it into gpu just map it to gpu inside <strong>init</strong> too.</p>

sunnnnnnnny: <p>How do I read all the data in the <strong>init</strong> method?</p>

sunnnnnnnny: <p>Is it possible to do this: in the dataset <strong>init</strong> function, build a list, then use the for loop to append the entire data set in the list, and then in the getitem function to retrieve data from the list according to the index, this operation is to load the data Going into RAM?</p>

JuanFMontesinos: <p>It depends on. That’s the straightforward pipeline, however there are two options. The typical one is to make a list of file paths or memory map tensors which will be loaded in <code>__getitem__</code>, however, in order to load it in RAM, you need to make a list of loaded files.</p>

suyz526: <p>Hi,</p>
<p>Since my dataset is a bit large and takes a lot of time to load to RAM (But the RAM is large enougt). I’m thinking if we can load the data to RAM, when the data has been used for the first time.</p>
<p>That means, I want to create an empty data list in init() to save the loaded data. And data will be written into the list in get_item(). Since get_item() is called in dataloader with multiple workers, so in this case, the data list can be filled very fast.</p>
<p>However, this not works. I think, it is because that every worker has its own memory. So after 1 epoch, the data list will be cleaned.</p>
<p>Do you have any suggestions for this(like speed up loading the data in RAM)? Maybe like create the data list in the shared memory?</p>
<p>Thanks!</p>

ptrblck: <p>You could check <a href="https://discuss.pytorch.org/t/dataloader-resets-dataset-state/27960/4">this post</a> which uses shared arrays as a “caching” mechanism.</p>
why softmax using odds and logits?

makeHIStory: <p>Hello</p>
<p>I’d like to ask you a question about softmax today. As far as I know, softmax converts the inferred values into probability values of sum 1.</p>
<p>Many of the data explaining softmax often explain only the function of the equation (conversion to probability), and some of the data interpret the output value of the shear transmitted to softmax as a logit. As I try to understand softmax with the concept of logit, I come to think of many things.</p>
<p>If you look at the softmax equation based on the picture below, exp(y) is used. If y is logit and logit is a value that takes a natural logarithm to odds, exp(y) becomes odds. Then, softmax is a value obtained by taking exp to logit and converting it into odds, dividing each odds by the sum of odds and converting it into probabilities. But I don’t understand why the concept of odds was used.</p>
<p>So this is how I understood it. Although logit converted odds into -inf, +inf values, we focused on values in the -inf, +inf range rather than correlating with odds and simply meant raw value of linear output, widening the gap between class-specific inference values and making differential expressions easier. Is it appropriate to interpret it like this?<br>
(참고 tensorflow.org에서의 logits의 정의 : Per-label activations, typically a linear output. These activation energies are interpreted as unnormalized log probabilities.)</p>
<p>Thank you for reading it, and I ask for many people’s answers/opinions.</p>
Without using data augmentation gives results better than using data augmentation

Kais1: <p>I am a beginner to deep learning, I’m doing the image classification problem on a small self plant disease imaging dataset (400 images)composed of 7 classes. I am using transfer learning(pretrained Resnet152) model and have applied some image augmentation techniques to increase the size of training data(random rotation, flipping etc.) but the performance was worse than not using data augmentation and the model get overfitted. The first image shows the curves which are generated with data augmentation and the second one without data augmentation Can anyone help me to know what is the problem and how to solve it? Thanks<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/b/a/bacf619809cd96e04c9d729dc03230c6f0304ba0.jpeg" alt="without" data-base62-sha1="qEB7xW0LOC0AQ3epvye6L0xd3tC" width="395" height="248"><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/1/3/13c780a6664cfef1191649a6834d78da3a6928a2.jpeg" alt="acc" data-base62-sha1="2OYvfYLET1KCjlGTOJQbbi0slSq" width="407" height="248"></p>
Difference in performance if pretrained ditilBert model in different setting

vikrant06: <p>I am training a classification model using pre-trained distilBert model for which I used two methods.</p>
<p><strong>Method 1:</strong><br>
I dumped the embeddings using this snippet:</p>
<pre><code class="lang-auto">def get_encodings(model, dataloader):
    model.eval()
    model.to(device)
    embeddings_df = pd.DataFrame()
    with torch.no_grad():
        for _, data in enumerate(dataloader, 0):
            ids = data['ids'].to(device, dtype = torch.long)
            mask = data['mask'].to(device, dtype = torch.long)
            last_hidden_states = model(ids, mask) #.squeeze()
            embeddings=last_hidden_states[0][:,0,:].cpu().numpy()
            temp_df = pd.DataFrame(embeddings)
            embeddings_df = embeddings_df.append(temp_df, ignore_index=True)
    return embeddings_df
</code></pre>
<p>Then used MLP to train and got 72% accuracy on test.</p>
<p>But in <strong>Method 2</strong> I am using the model directly inside the class. With same layers and hyperparameters I am getting very poor results. Here is my model class:</p>
<pre><code class="lang-auto">class DistillBERTClass(torch.nn.Module):
    # define model elements
    def __init__(self):
        super(DistillBERTClass, self).__init__()
        self.l1 = DistilBertModel.from_pretrained("distilbert-base-uncased", cache_dir=cache_path)
        # input to first hidden layer
        self.hidden1 = Linear(768, 256)
        kaiming_uniform_(self.hidden1.weight, nonlinearity='relu')
        self.activation = ReLU()
        self.bn1 = nn.BatchNorm1d(num_features=256)
        self.dropout_1 = torch.nn.Dropout(0.5)
        # output
        self.output = Linear(256, NUM_CLASSES)
    # forward propagate input
    def forward(self, input_ids, attention_mask):
        # Extract Embeddings
        output_1 = self.l1(input_ids=input_ids, attention_mask=attention_mask)
        hidden_state = output_1[0]
        pooler = hidden_state[:, 0,:]
        # input to first hidden layer
        X = self.hidden1(pooler)
        X = self.bn1(self.activation(X))
        X = self.dropout_1(X)
        # output layer
        X = self.output(X)
        return X
</code></pre>
<p>Also I tried freezing all the 100 layers, but that further degraded the performance. The network got struck at 20-30 % accuracy. Here is my snippet for layer freezing:</p>
<pre><code class="lang-auto"># freeze transformers layers
freezed_layers = 0
for name, param in list(model.named_parameters())[:100]:  
    print(name)
    param.requires_grad = False
    freezed_layers += 1
</code></pre>
<p>Am I missing something ? do I need to do any modification in optimisers after freezing the layers?</p>
<p>The bottom line is I want to use the pre-trained model as feature extractor and only train the layers after it. The backpropagation shouldn’t go the transformer layers and their weight should remain intact so that it will give same performance as <strong>method 1</strong></p>
Correct way of implementing VGG Perceptual Loss

mohit117: <p>While computing <a href="https://arxiv.org/pdf/1603.08155.pdf" rel="noopener nofollow ugc">VGG Perceptual loss</a>, although I have not seen, I feel it is alright to wrap the computation of VGG features for the GT image inside <code>torch.no_grad()</code>.</p>
<p>So basically I feel the following will be alright,</p>
<pre><code class="lang-auto">with torch.no_grad():
    gt_vgg_features = self.vgg_features(gt)

nw_op_vgg_features = self.vgg_features(nw_op)

# Now compute L1 loss
</code></pre>
<p>or one should use,</p>
<pre><code class="lang-auto">gt_vgg_features = self.vgg_features(gt)
nw_op_vgg_features = self.vgg_features(nw_op)
</code></pre>
<p>In both approaches <code>requires_grad</code> for VGG parameters is set <code>False</code> and VGG put in <code>eval()</code> mode.</p>
<p>The first approach will save a lot of GPU resources and feel should be numerically equal to the second one as no backpropagation is required through GT images. But in most implementations, I find the second approach used for computing VGG perceptual loss.</p>
<p>So which option we should go for implementing VGG perceptual loss?</p>