akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Custom DataLoader For Audio Classification

username: achyut_mani_Tripathi
cooked: <p>Dear All,<br>
I am very new to PyTorch. I am working towards designing of data loader for my audio classification task.<br>
Training and Test Data:<br>
I have a set of audion files (.wav files). I extracted the spectrogram features from each file and saved them into a database created using .h5py file in python.<br>
Design of data loader:<br>
I want to create a custom data loader in such a way that the created mini-batch always contains 50% of data that belong to two predefined cl<code>Preformatted text</code>asses and remaining samples in minibatch are from classes other than two predefined classes.<br>
In my case I have 10 classes, I always want the minibatch such that It contains 50% of samples from class 4 and class 6 and other 50 % samples from any of the class from [1,2,3,5,7,8,910].</p>
<h1>Here is my piece of code for a custom class</h1>
<p>import os<br>
import torch<br>
import numpy as np<br>
import h5py<br>
from torch.utils.data import dataset<br>
import pandas as pd</p>
<p>class AEData_Test():<br>
def <strong>init</strong>(self,transform=None):<br>
self.annotations=pd.read_csv(“Test.csv”)<br>
self.transform=transform<br>
def <strong>len</strong>(self):<br>
return len(self.annotations)<br>
def <strong>getitem</strong>(self,index):<br>
key=self.annotations.lioc[index,0]<br>
print(index)<br>
with h5py.File(‘New_Devlopment.hdf5’, ‘r’) as f:<br>
SG_Data = f[key][()]<br>
SG_Label= torch.tensor(int(self.annotations.lioc[index,1]))<br>
if self.transform:<br>
SG_Data=self.transform(SG_Data)<br>
return (SG_Data,SG_Label)	<br>
Please guide me how to modify my code to complete the desired task.</p>
<p>Thanks<br>
Achyut</p>

username: ptrblck
cooked: <p>If the number of samples could be “approx. 50% class4 and class 6 vs. the rest”, you could use a <code>WeigthedRandomSampler</code> as described <a href="https://discuss.pytorch.org/t/how-to-handle-imbalanced-classes/11264/2">in this post</a>.<br>
The sampler would accept the sample weights and uses these to draw each sample.<br>
Since this is a random process, the exact number of class samples is not exactly defined.</p>
How to disable fp32 accumulation in tensor cores?

username: farmersrice
cooked: <p><code>nvprof</code> shows pytorch using a cuda kernel with fp32 accumulation:</p>
<p><code>turing_fp16_s1688cudnn_fp16_256x128_ldg8_relu_f2f_exp_small_nhwc_tn_v1</code></p>
<p>on code</p>
<details>
<summary>
python</summary>
<pre><code class="lang-auto">import torch

model = torch.nn.Sequential(
	torch.nn.Conv2d(128, 128, 3, stride=1, padding=1),
	torch.nn.Conv2d(128, 128, 3, stride=1, padding=1),
	torch.nn.Conv2d(128, 128, 3, stride=1, padding=1),
	torch.nn.Conv2d(128, 128, 3, stride=1, padding=1),
	torch.nn.Conv2d(128, 128, 3, stride=1, padding=1),
	torch.nn.Conv2d(128, 128, 3, stride=1, padding=1),
	torch.nn.Conv2d(128, 128, 3, stride=1, padding=1),
	torch.nn.Conv2d(128, 128, 3, stride=1, padding=1),
	torch.nn.Conv2d(128, 128, 3, stride=1, padding=1),
	torch.nn.Conv2d(128, 128, 3, stride=1, padding=1),
	torch.nn.Conv2d(128, 128, 3, stride=1, padding=1),
	torch.nn.Conv2d(128, 128, 3, stride=1, padding=1),
	torch.nn.Conv2d(128, 128, 3, stride=1, padding=1),
	torch.nn.Conv2d(128, 128, 3, stride=1, padding=1),
)

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

print(device)

x = torch.rand((200, 128, 8, 8), dtype=torch.half, device=device)

model = model.to(device).half()

print(model.forward(x))
</code></pre>
</details>
<details>
<summary>
C++ (slightly different)</summary>
<p><span class="hashtag">#include</span> &lt;bits/stdc++.h&gt;<br>
<span class="hashtag">#include</span> &lt;torch/torch.h&gt;</p>
<p>using namespace torch;</p>
<p>const int filters = 128;</p>
<p>struct Block : nn::Module {</p>
<pre><code>nn::Conv2d conv1 = nullptr;
nn::Conv2d conv2 = nullptr;


Block() {
	conv1 = register_module("conv1", nn::Conv2d(nn::Conv2dOptions(filters, filters, 3).stride(1).padding(1)));
	conv2 = register_module("conv2", nn::Conv2d(nn::Conv2dOptions(filters, filters, 3).stride(1).padding(1)));
}

Tensor forward(Tensor x) {
	at::Tensor residual(x.clone());

	x = conv1-&gt;forward(x);
	x = relu(x);

	x = conv2-&gt;forward(x);

	x += residual;
	x = relu(x);

	return x;
}
</code></pre>
<p>};</p>
<p>int main() {<br>
double sendSeconds = 0;<br>
torch::Device device = torch::kCUDA;</p>
<pre><code>nn::Sequential trunk = nn::Sequential();

for (int i = 0; i &lt; 10; i++) {
	trunk-&gt;push_back(Block());
}

NoGradGuard guard;
trunk-&gt;eval();
trunk-&gt;to(device, torch::kHalf);


for (int i = 0; i &lt; 500; i++) {
	Tensor tt = torch::rand({1, filters, 8, 8}).pin_memory();

	//auto mid1 =  std::chrono::steady_clock::now();
	tt = tt.to(device, torch::kHalf);
	//auto mid2 = std::chrono::steady_clock::now();

	tt = trunk-&gt;forward(tt);

	if (i == 499) {
		std::cout &lt;&lt; tt &lt;&lt; std::endl;
	}
}
</code></pre>
<p>}</p>
</details>
<p>I want to NOT use accumulation (example kernel: <code>turing_h1688cudnn_128x128_ldg8_relu_exp_small_nhwc_tn_v1</code> (although different dimensions)).</p>
<p>How can I turn off fp32 accumulation and go straight through with fp16? I would like a solution for C++.</p>
Repeating weights for a single axis of the pretrained model

username: MUnique
cooked: <p>Hello everyone,<br>
I have a pretrained model that accepts 50,25,25,60 inputs. Now, I want to use it for 1000,25,25,60 inputs assuming that the first axis weights can be repeated 20 times to fit the new settings (1000/50 = 20). How can I do so?</p>

username: ptrblck
cooked: <p>Usually the batch dimension would be in dim0, so that you wouldn’t have to change anything.<br>
However, if you are using a custom model without a batch dimension (or in another dim), you could use the <a href="https://pytorch.org/docs/stable/tensors.html#torch.Tensor.repeat"><code>repeat</code></a> or <a href="https://pytorch.org/docs/stable/tensors.html#torch.Tensor.expand"><code>expand</code></a> function on the parameter(s).</p>
Functional.conv2d without gradients

username: Barak_Beilin
cooked: <p>I’m using <code>torch.nn.functional.conv2d</code> to convolve an input with a custom non learning kernel, as follows:</p>
<pre><code class="lang-auto">input = torch.randn([1,3,300,300], requires_grad=False)
output = torch.nn.functional.conv2d(input, weight=torch.randn([6,3,5,5]))
</code></pre>
<p>What steps do I need to take to prevent gradient from being calculated as I’m trying to save memory.</p>
<p>I’ve declared my input tensor with <code>requires_grad=false</code>.<br>
Is this enough or was this unnecessary?</p>
<p>Thanks,<br>
Barak.</p>

username: ptrblck
cooked: <p>That should be enough, since <code>output</code> won’t have a valid <code>.grad_fn</code>, which indicates that no computation graph was created.<br>
If you try to call <code>backward()</code> on <code>output</code>, you would get the typical error:</p>
<pre><code class="lang-python">RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn
</code></pre>
<p>You could also wrap this code snippet in another <code>with torch.no_grad()</code> block just to make sure that no gradient are calculated even if you are using a <code>weight</code> parameter, which might require gradients.</p>
Batch random rotate (different rotation matrix per sample)

username: dvirginz
cooked: <p>I have a batched tensor of size <code>batch.size = [B,H,W,E]</code> (E - Euclidian coordinates) and I would like to rotate each sample differently (I cannot do this step in the dataloading part.</p>
<p>Thus I generated <code>B</code> random rotations parameters:</p>
<pre><code class="lang-auto">random_angles= torch.FloatTensor(1024).uniform_(-180, 180)
rand_cos = torch.sin(random_angles * np.pi) / 180
rand_sin = torch.cos(random_angles * np.pi) / 180
</code></pre>
<p>Now, if it wasn’t a batched tensor, I would simply do:</p>
<pre><code class="lang-auto">single_rotation_mat = [[1, 0, 0], [0, rand_cos[0], rand_sin[0]], [0, -rand_sin[0], rand_cos[0]]] # for ration along axis=0
rotated = batch[0].matmul(single_rotation_mat)
</code></pre>
<p>My problem is to somehow write the function that will take the random angles vector, and transform it to a batch of random rotation matrices. (given the <code>[[1, 0, 0], [0, cos, sin], [0, -sin, cos]]</code> logic ber matrix)</p>
Vectorize matrix-vector multiplication

username: Ursulla_Carrera
cooked: <p>I would like to multiply 3 vectors of size 2 by a 2x2 matrix multiplication without using a loop.<br>
With a loop it would look like this:</p>
<pre><code class="lang-auto">A=torch.tensor([[1.,2.],[3.,4.]])
b=torch.tensor([[3.,4.], [5.,6], [7.,8.]])
res = []
for i in range(b.shape[0]):
    res.append(torch.matmul(A, b[i]))
</code></pre>
<p>but I would like is something lile <code>torch.matmul(A, b)</code>.</p>
<p>I tried <code>torch.matmul(A.repeat((2,1,1)), b)</code> but obviously it did not work.</p>

username: chetan_patil
cooked: <p>You can do this.<br>
<code>A.matmul(b.t())</code></p>
Way to change torch tensor from 28x28 to 32x32

username: mathematics
cooked: <p>Is there any way to reshape tensor shape.</p>
<p>I ran into this problem using TensorDataset and Dataloader. I can’t use transforms here.<br>
Any directly method without changing to numpy?</p>
<p>Thanks for ideas.</p>

username: Nikronic
cooked: <p>Hi,</p>
<p>Can <code>torch.nn.functional.interpolate</code> this solve your issue?</p>
<aside class="quote" data-post="2" data-topic="52401">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nikronic/48/71873_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/resize-tensor-without-converting-to-pil-image/52401/2">Resize tensor without converting to PIL image?</a> 
  </div>
  <blockquote>
    Hi, 
You can do it using <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.functional.interpolate" rel="noopener nofollow ugc">interpolate</a> function and it supports different methods. 
Here is an example: 

import PIL.Image as Image
from torchvision.transforms import ToTensor, ToPILImage
import torch.nn.functional as F

img = Image.open('data/Places365_val_00000001.jpg')
img = ToTensor()(img)
out = F.interpolate(img, size=128)  #The resize operation on tensor.
ToPILImage()(out).save('test.png', mode='png')

Bests 
Nik
  </blockquote>
</aside>

<p>Bests</p>

username: mathematics
cooked: <p>Yay, thanks.<br>
As solution i have converted<br>
<code>[20,784]</code> i.e <code>[20,1,28,28]</code> to <code>[20,1,32,32]</code><br>
using interpolate as<br>
<code>xb = F.interpolate(xb, size=32,32)  #The resize operation on tensor.</code></p>
Does torch.autograd.grad save gradient?

username: Juna
cooked: <p>Hi,<br>
I’m trying to get a gradient of some tensor with respect to a loss function by using torch.autograd.grad<br>
but the problem is that I don’t want the gradient of other tensors or model parameters to be saved in their .grad.<br>
I’m not really sure if torch.autograd.grad does that.<br>
Thanks in advance</p>

username: albanD
cooked: <p>Hi,</p>
<p><code>autograd.grad</code> does not touch the <code>.grad</code> fields. It returns directly all the gradients it computes. And it only computes gradients that are needed.</p>

username: Juna
cooked: <p>Thanks a lot It helped!</p>
How to cache intermediate interactions of model parameters that are not input-dependent?

username: Zhaofeng_Wu
cooked: <p>Assume that I have a model that has two parameters <code>A</code> and <code>B</code>, takes as input a matrix <code>C</code>, and returns <code>f(g(A, B), C)</code>. I could compute this during every forward fun, but we really should be able to cache <code>g(A, B)</code>. I tried to compute <code>self.D = g(A, B)</code> in <code>__init__</code>, but the device doesn’t match when I do <code>f(self.D, C)</code>. Alternatively, if I do <code>f(self.D.to(C.device), C)</code>, the forward pass works but the gradient device is wrong during the optimizer step. I think I could also somehow always keep <code>self.D</code> in GPU memory at all times, but is that necessary? What’s the best way to accomplish this?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Zhaofeng_Wu" data-post="1" data-topic="87679">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zhaofeng_wu/48/14563_2.png" class="avatar"> Zhaofeng_Wu:</div>
<blockquote>
<p>but the gradient device is wrong during the optimizer step.</p>
</blockquote>
</aside>
<p>What kind of error are you getting?<br>
The <code>to()</code> operation is differentiable, thus the parameters of different models can be on different devices.</p>

username: Zhaofeng_Wu
cooked: <p>My apologies. The error is actually coming from a 3rd party <code>transformer</code> library. I saw the error was coming from <code>optimizer.step()</code> so I didn’t look in more detail. The error stack trace is pasted below, but I will ask in their repo. Thanks!</p>
<pre><code class="lang-auto">File "file.py", line 71, in optimizer_step
optimizer.step()
File ".../lib/python3.7/site-packages/torch/optim/lr_scheduler.py", line 67, in wrapper
return wrapped(*args, **kwargs)
File ".../lib/python3.7/site-packages/transformers/optimization.py", line 155, in step
exp_avg.mul_(beta1).add_(grad, alpha=1.0 - beta1)
RuntimeError: expected device cuda:0 but got device cpu
</code></pre>

username: ptrblck
cooked: <p>Sure, let us know, if you figured out the issue.<br>
Based on the error message my best guess is that some internal buffers using in the optimizer are not pushed to the right device, so maybe an <code>optimizer.cuda()</code> method was implemented in this repository?<br>
If not, try to push the model to the device before passing it to the optimizer.</p>
How to use embedding layers for batched multilabel categories?

username: Abhishaike_Mahajan
cooked: <p>So, I have time-series data where there could be multiple categories per time-point. I would like to use an embedding lookup table to get each individual categories embedding, and take the average of them for each time point. I dug a little, and found a neat little EmbeddingBag class seemingly made exactly for this type of setup.</p>
<p>So, below, I have 4 timepoints, each composed of a max number of 2 categories. The 2nd-to-last datapoint is half-padded, and the last datapoint is completely padded. I use the ‘max’ functionality to ensure that the function is interpreting my data correctly.</p>
<pre><code class="lang-auto">embedding_sum = nn.EmbeddingBag(10, 3, mode='max')
input = torch.LongTensor([[1,2], [1,1], [2,2], [0,0]])
print(embedding_sum(input))

tensor([[-0.2409,  0.4761,  0.0832],
        [-1.9867,  0.4761, -0.5098],
        [-0.2409,  0.0551,  0.0832],
        [-0.5566, -0.3860,  0.6931]], grad_fn=&lt;EmbeddingBagBackward&gt;)
</code></pre>
<p>This works just fine. However, I do not know how to extend this to a batched case. For example, where I have something like two data points or in code form…</p>
<pre><code class="lang-auto">embedding_sum = nn.EmbeddingBag(10, 3, mode='max')
input = torch.LongTensor([[[1,2], [1,1], [2,2], [0,0]], [[2,2], [1,1], [0,0], [0,0]]])
print(embedding_sum(input))

ValueError: input has to be 1D or 2D Tensor, but got Tensor of dimension 3
</code></pre>
<p>I get a value error. Any ideas on what I’m missing?</p>
Expected input batch_size to match target batch_size for Linear Layer after RNN

username: huzzm
cooked: <p>I am getting an error when trying to evaluate a model with Cross Entropy Loss after calling the forward function of my model which is a simple RNN with a linear layer afterwards.</p>
<p>This is my model. As you will see, input size, hidden size and output size are all 1, while only the sequence length is 80. The sequences are just 80 floating point numbers, and the label is one floating point value (the next value for the whole sequence that I am trying to learn).</p>
<pre><code class="lang-auto">class RNN(nn.Module):
    def __init__(self, input_size, hidden_size, output_size, seq_length):
        super(RNN, self).__init__()
        self.hidden_size = hidden_size
        self.input_size = input_size
        self.seq_length = seq_length
        self.num_layers = 1
        self.batch_size = 1  

        self.rnn = nn.RNN(input_size, hidden_size)
        self.linear = nn.Linear(hidden_size, output_size)
        
        self.hidden_state = self._init_hidden()
    
    def forward(self, input_seq):
        input_seq = input_seq.reshape(self.seq_length, self.batch_size, self.input_size)
        out, self.hidden_state = self.rnn(input_seq, self.hidden_state)
        print(out.shape)
        out = self.linear(out)
        return out
    
    def _init_hidden(self):
        hidden = torch.zeros(1, self.batch_size, self.hidden_size)
        return hidden

rnn_model = RNN(input_size=1, hidden_size=1, output_size=1, seq_length=80)
loss_function = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(rnn_model.parameters(), lr=lr)
</code></pre>
<pre><code class="lang-auto">for i in range(epochs):
    for seq, label in train_sequences:
        optimizer.zero_grad()
        output = rnn_model(seq)
        loss = loss_function(output, label) # &lt;-- ERROR
        loss.backward() 
        optimizer.step() 
</code></pre>
<p>The error message that I get is: <strong>Expected input batch_size (80) to match target batch_size (1).</strong></p>
Too many different loss and too times backward

username: Selcuk_Caglar
cooked: <p>Enviroment:</p>
<pre><code class="lang-auto"># OCR optimizer
ocr_loss.backward(retain_graph=True)
self.ocr_optimizer.step()

# Total generator loss
loss_G = loss_content + self.opt.lambda_adv * loss_GAN + self.opt.lambda_pixel * loss_pixel
loss_G += ocr_loss
loss_G.backward(retain_graph=True)
self.optimizer_G.step()
</code></pre>
<p>…<br>
Error:<br>
RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time.</p>
Compute gradients when using BERT classifier on top of image captioning model

username: taivanbat
cooked: <p>Hi,</p>
<p>I have an image captioning model, and I want to ensure that it generates sentences with correct facts by using a pretrained BERT classifier on top of it (the model generates a sentence --&gt; we extract “facts” from the sentence using a pretrained classifier --&gt;  we calculate the loss on the classification).</p>
<p>I want to use the Huggingface Transformers library, but because their input is a sentence that they then tokenize, I am unsure of how to handle or compute the gradients.</p>
<p>Does anyone have any ideas how I can compute the gradients?</p>
Load Data / Models to a specific CPU

username: t326wang
cooked: <p>I’m implementing an algorithm which requires a lot of model evaluations. I want to parallelize model evaluations by using CPU. The code segment I want to parallelize is here (simplified for readability):</p>
<pre><code class="lang-python">def test_img(network, datatest):
    network.eval()

    data_loader = DataLoader(datatest, batch_size=args.bs)

    for idx, (data, target) in enumerate(data_loader):
        result = network(data)
</code></pre>
<p>And for one evaluation it takes around 1.5s on CPU. The networks for evaluation are different, but the dataset is the same. I tried to use joblib.Parallel to parallelize this process:</p>
<pre><code class="lang-auto">results = Parallel(n_jobs=num_cpu, prefer="threads")(delayed(test_img)(network_lst[i], dataset) for i in range(N))
</code></pre>
<p>However, it seems that there are no improvement by using this method (it will take 15s for 10 evaluations). I specifically count the runtime for <code>result = network(data)</code> line, and it takes 8 seconds. Therefore I think neither the network evaluation nor the dataloader are parallelized. Are there any way for us to load network and data to a specific CPU core, like <code>data.to('cpu:0')</code>? Is it even possible to use CPU for model evaluation parallelization?</p>
<p>Any suggestions are appreciated!</p>

username: iffiX
cooked: <p>By default pytorch will use multiple cpu cores to calculate:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/6/1/61e1c2b098ebfef2656624c76ff933dc5e4a0cca.png" alt="image" data-base62-sha1="dXU0guhBB81uRxjQmVWz4NArasi" width="677" height="61"></p>
<pre><code class="lang-auto">import time
import multiprocessing as mp
import torch as t

def subproc():
    # keep process showing in "top"
    begin = time.time()
    while time.time() - begin &lt; 10:
        a = t.ones([1000, 1000]) * t.ones([1000, 1000])


if __name__ == "__main__":
    p = mp.Process(target=subproc, args=())
    p2 = mp.Process(target=subproc, args=())
    p.start()
    p2.start()
    print("started")

    p.join()
    p2.join()
    print("joined")
</code></pre>
<p>Seems that you are using “threads”, not good for python, you must use processes:</p>
<pre><code class="lang-auto">results = Parallel(n_jobs=num_cpu, prefer="threads")(delayed(test_img)(network_lst[i], dataset) for i in range(N))
</code></pre>
Installing PyTorch results in an older version

username: blade
cooked: <p>I’m trying to update PyTorch on a Conda environment on a remote machine. I enter:</p>
<p><code>conda install pytorch torchvision -c pytorch</code></p>
<p>The current stable version is <code>1.5.1</code> but above command installs <code>1.4.0</code>. I was wondering what is the issue and how to fix it.</p>

username: harsha_g
cooked: <p><code>conda install pytorch=1.5.1 torchvision=0.6.1 -c pytorch</code></p>

username: blade
cooked: <p>I tried that and I received the following message:</p>
<blockquote>
<p>Collecting package metadata (current_repodata.json): done<br>
Solving environment: failed with initial frozen solve. Retrying with flexible solve.<br>
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.<br>
Collecting package metadata (repodata.json): done<br>
Solving environment: failed with initial frozen solve. Retrying with flexible solve.<br>
Solving environment: /<br>
Found conflicts! Looking for incompatible packages.<br>
This can take several minutes.  Press CTRL-C to abort.<br>
Examining conflict for tensorboard tensorflow-estimator tensorflow-base grpcio tensorflow absl-py:   2%|▏         | 2/98 [00:05&lt;04:22,  2.74sExamining conflict for tensorboard tensorflow-estimator tensorflow-base grpcio tensorflow absl-py:   3%|▎         | 3/98 [00:05&lt;03:08,  1.99s</p>
</blockquote>
<p>and it continued on and on.</p>

username: harsha_g
cooked: <p>What version of Python does your anaconda have? I am on 3.7.7 and (in the past) when I used the command you posted, the farthest I could get to was <code>PyTorch 1.3.1</code> and <code>torchvision 0.4.2</code>. I manually upgraded the modules using the command I posted. However, in the interest of your conda environment stability, I’d just create a new environment and install libraries as needed.</p>

username: albanD
cooked: <p>Hi,</p>
<p>This is most likely due to either an old version of conda that does not support latest pytorch (you just need to update conda in that case). Or other packages that have dependencies that conflict with the latest version of pytorch.</p>
<p>As <a class="mention" href="/u/harsha_g">@harsha_g</a> suggested, I would try to install it in a new environment to make sure it is not due to conflict with other libs.</p>

username: blade
cooked: <p><a class="mention" href="/u/harsha_g">@harsha_g</a> <a class="mention" href="/u/alband">@albanD</a> That makes sense. Thank you both!</p>
Freezing in shared weight

username: ad6398
cooked: <pre><code class="lang-auto">class modelA(Module):
    def __init__(self):
        super().__init__()
        self.base= ...
        self.headA= ...
    
    def forward(self,input):
        x= self.base(input)
        outA= self.headA(x)

        return outA

class modelB(Module):
    def __init__(self):
        super().__init__()
        self.base=...
        self.headB= ...

    def forward(self,input):
        x= self.base(input)
        outB= self.headB(x)
        return outB
</code></pre>
<p>this model was supposed to work in way that weight of base layer should be shared, so as per suggestion <a href="https://discuss.pytorch.org/t/how-to-create-model-with-sharing-weight/398/2">here</a> by <a class="mention" href="/u/apaszke">@apaszke</a> I defined a combined model as follow</p>
<pre><code class="lang-auto">class model_combined(Module):
    def __init__(self):
        super().__init__()
        self.base=...
        self.headA= ...
        self.headB= ...

    def forward(self,input):
        x= self.base(input) # this need to be freezed
        outA= self.headA(x)
        y=self.base(input) #this neeeds to backpropagte
        outB= self.headB(y)
        return outA, outB
</code></pre>
<p>in training loop I am calculating loss as</p>
<pre><code class="lang-auto">total_loss= loss(outA)+ loss(outB)
total_loss.backward()
</code></pre>
<p>but after some epoch, I need to freeze modelA (i.e self.headA in model_combined)<br>
My questions are:</p>
<ol>
<li>since <code>base</code> is common layer here, so how to freeze <code>headA</code> (weight of headA should not change) such that <code>base</code> is updated due to backpropagation of <code>loss(outB)</code> but not due to <code>loss(outA)</code>
</li>
<li>will it be a bad practice if instead of freezing I just don’t backpropagate loss through <code>headA</code> i.e change <code>total_loss= loss(outB)</code>
</li>
</ol>

username: albanD
cooked: <p>Hey,</p>
<p>If you don’t want gradients corresponding to <code>headA</code>, then yes I think the best thing to do is to just not backrprop for it.</p>
<p>Note that if you share the weights between A and B, then the weights in A will still change due to the update on B.</p>

username: ad6398
cooked: <p>Hi, but won’t this be a problem , as graph of <code>headA</code> will keep growing without a loss.backward() step for <code>headA</code> (back-propagation frees the graph created during epoch) ?  by sharing of weight do you mean manually copying of weight from one layer to another or something else.</p>

username: albanD
cooked: <blockquote>
<p>by sharing of weight do you mean manually copying of weight from one layer to another or something else.</p>
</blockquote>
<p>You mention that " since <code>base</code> is common layer here" so I assume that you have a single set of parameters for both. But maybe I misunderstood this.</p>
Imbalanced dataset and image augmentation

username: sn07
cooked: <p>Hi,</p>
<p>I am doing the image classification with 3 classes using Transfer Learning(Resnet-18). The issue with my dataset is I have a fair imbalance dataset, in a way, that the number of positive samples is very less as compared to the other 2 classes. So , there are 3 questions:</p>
<ol>
<li>Should I use weighted Random Sampler to oversample the minority class(the positive one) and undersample the rest 2 classes? (For instance, in my case, I have around 500 images of class 1 and &gt;2k images of class 2 and class 3), so in the very beginning , should I select the fairly equal samples like 500 odd of each class. I am rather doubtful over this, because I am using transfer learning, and it may lead to overfitting.<br>
2). So, if I goes with the case that I select 500 images of class 1 and the rest(&gt;2k) images of other classes, then along with weighted random sampling, should I also augment the images for all 3 classes in transform sequence .</li>
</ol>
Multiclass problem with cross entropy

username: netaglazer
cooked: <p>How should I insert the labels to the loss?<br>
should it be one-hot encoded?<br>
or can it be tensor of class’s num:<br>
tensor([1,2,5,3,5,3,…])</p>

username: Nikronic
cooked: <p>Hi,</p>
<p>Actually, you need to use indices as you have mentioned. No need to convert to one hot.<br>
Here is the <a href="https://pytorch.org/docs/master/generated/torch.nn.CrossEntropyLoss.html" rel="nofollow noopener">official doc</a>.</p>
<p>Bests</p>
What loss function should I use?

username: maudzung
cooked: <p>Hello everyone,</p>
<p>I’m doing a task that has targets within <em>a range of 0 and 1 <strong>(like probabilities)</strong></em>.<br>
I thought about <strong>Binary Cross-Entropy</strong>, but the task is not the binary classification problem. So from my point of view, the <strong>BCE</strong> loss may not be fit.<br>
I have tried the <strong>MSE</strong> loss function but the results were not good.</p>
<p>Can you please suggest me other loss functions? I’d like to try them.</p>
<p>Thank you so much.</p>

username: dejanbatanjac
cooked: <p>You have a regression problem. But not logistic regression because logistic regression is a classification problem as well. So you need some loss functions  for real regression problem. L1, L2 respectable called MAE, and MSE for instance. If you tried MSE, maybe you should try more. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Implement Convpoint with Pytorch-geometric structure

username: shangguan91
cooked: <p>Dear all,</p>
<p>i want to implement different methods on my point cloud dataset. i have pretrained model such as Edgecov, splineconv. Now i intend to reimplement with Convpoint, However there is no code for Convpoint in PyG, only in this repo&gt; <a href="https://github.com/aboulch/ConvPoint" rel="nofollow noopener">https://github.com/aboulch/ConvPoint</a>.<br>
Shall Clone this repo and inplant the Architecture to achieve what i want?</p>
Creating dataloader for multivariate time series dataset

username: achyut_mani_Tripathi
cooked: <p>Dear All,</p>
<p>I am trying to classify the multivariate time series (MTS) data. Let’s say the data set contains 10000 data samples. Out of the 10000 data samples, training data contains 6000 data samples, 2000 data samples for validation data, and the remaining 2000 data samples in the test dataset.<br>
I want to write PyTorch based script that uses CNN to classify the MTS. For this purpose, I need to create a data loader to load the data from the CSV files that contain MTS data. I am very new to PyTorch. Can you please help me with the steps  to create a data loader from the MTS.</p>
<p>Thanks<br>
Achyut</p>
Resize sanity check

username: newuser1
cooked: <p>I’m new to torch, coming from TF/Keras, and I’m just trying to do some basic examples loading pre-trained models and generating some predictions for image classification problems.</p>
<p>To do this I’ve been using the transform.Resize() function.<br>
For instance, I’m using an Inception model that expects a (299,299,3) shaped input.<br>
I see a lot of places making it seem as if transform.Resize(299) and transform.Resize((299,299)) should give the same results, but that is not the case from my tests.</p>
<p>Can someone clarify if this is expected behavior? Some other sanity checks I ran were to resize to (299,300), which should generate an error, but the network still makes a prediction. How is this possible?</p>
<p>Thank you in advance for any help.</p>

username: tom
cooked: <aside class="quote no-group" data-username="newuser1" data-post="1" data-topic="87965">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/n/f1d935/48.png" class="avatar"> newuser1:</div>
<blockquote>
<p>I see a lot of places making it seem as if transform.Resize(299) and transform.Resize((299,299)) should give the same results, but that is not the case from my tests.</p>
</blockquote>
</aside>
<p>Yeah, the  <a href="https://stackoverflow.com/questions/50002543/what-are-transforms-in-pytorch-used-for" rel="noopener nofollow ugc">StackOverflow explanation</a> is wrong. In general, it seems that there aren’t many places to get better PyTorch advice than on these forums here.</p>
<aside class="quote no-group" data-username="newuser1" data-post="1" data-topic="87965">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/n/f1d935/48.png" class="avatar"> newuser1:</div>
<blockquote>
<p>Can someone clarify if this is expected behavior?</p>
</blockquote>
</aside>
<p>The <a href="https://pytorch.org/docs/stable/torchvision/transforms.html#torchvision.transforms.Resize" rel="noopener nofollow ugc">TorchVision documentation</a> explains:</p>
<ul>
<li>If size is a sequence like (h, w), output size will be matched to this.</li>
<li>If size is an int, smaller edge of the image will be matched to this number. i.e, if height &gt; width, then image will be rescaled to (size * height / width, size)</li>
</ul>
<p>The idea of the latter is to not ruin the aspect ratio of the image and follow by a cropping operation to the final size.</p>
<p>Best regards</p>
<p>Thomas</p>
Exporting backpropagation graph

username: Sumat_Purewal
cooked: <p>Is there any way to export/save the graph used for backprop as a regular pytorch feedforward network?</p>
<p>I have a network with fixed weights that computes a loss function. I need just a small number of backprop iterations to modify my input into a desired result. If I could export the backprop graph then I could effectively unroll the small number of iterations to create one larger network that computes the result in one shot.</p>

username: tom
cooked: <p>No, that isn’t possible.<br>
The backward computations are not even necessarily visible as functions. (Personally, I think it would be worth making all backwards available to be more consistent, but that sentiment hasn’t caught on, possibly because most potential use cases seem to have limited attraction.)</p>
<p>Best regards</p>
<p>Thomas</p>
Targets attribute ImageFolder

username: ganLover
cooked: <p><strong>While trying to access the Targets attribute of the ImageFolder-</strong></p>
<pre><code class="lang-auto">valdata = datasets.ImageFolder("../input/chest_xray/chest_xray/val", transform = transform)
valloader = torch.utils.data.DataLoader(valdata, shuffle = True, batch_size = 64 )

print(valdata.class_to_idx)
print(valdata.classes)
print(valdata.targets)
for i in valdata.samples:
    print(i)
</code></pre>
<p><strong>I get this error-</strong></p>
<pre><code class="lang-auto">{'NORMAL': 0, 'PNEUMONIA': 1}
['NORMAL', 'PNEUMONIA']
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
&lt;ipython-input-46-a7c7858221ef&gt; in &lt;module&gt;()
     20 print(valdata.class_to_idx)
     21 print(valdata.classes)
---&gt; 22 print(valdata.targets)
     23 for i in valdata.samples:
     24     print(i)

AttributeError: 'ImageFolder' object has no attribute 'targets'
</code></pre>

username: ptrblck
cooked: <p>It looks like the <code>targets</code> attribute was added in <a href="https://github.com/pytorch/vision/pull/501" rel="nofollow noopener">this PR</a> 9 months ago. Could you try to update <code>torchvision</code> and try your code again?</p>

username: ganLover
cooked: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>Actually, I was using Pytorch on a Kaggle Kernel.<br>
Is there any permanent fix- so that I need not update the libraries every time I use a Kaggle Kernel?</p>

username: ptrblck
cooked: <p>I’m not sure if Kaggle Kernels stay updated or if you would have to update certain packages in each run.<br>
Could you try it out and let us know?</p>
<p>Alternatively, you might just get all targets using <a href="https://github.com/pytorch/vision/blob/1f8f7ea71c8807408a7860b8ae9ca63177975b64/torchvision/datasets/folder.py#L96" rel="nofollow noopener">this line of code</a> manually:</p>
<pre><code class="lang-python">targets = [s[1] for s in valdata.samples]
</code></pre>

username: ganLover
cooked: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a> for help</p>

username: SANTOSH_S
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>,<br>
how do we get length of each class in this example?</p>

username: ptrblck
cooked: <p>If your targets are all stored in a tensor and you would like to get the count for each class, you could use <code>torch.unique(targets, return_counts=True)</code>.</p>