akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Basic quick question, size of dataset object

username: Ian_Scott
cooked: <p>Hi</p>
<p>sorry this is a really basic question but I genuinely can’t figure it out.  I have a custom dataset object and had to limit my batch size so would all fit on the GPU ram.  But what I can’t figure out is what size the dataset object actually is?  I tried this:</p>
<p>sys.getsizeof(myDataset)</p>
<p>and it returned-</p>
<p>56</p>
<p>As I understand that 56 bytes?  This doesn’t seem right as its a ton of images.  Can anyone enlighten me?</p>
<p>Ian</p>

username: ptrblck
cooked: <p>Depending how you are loading the data (eager or lazily) you might be able to check the number of elements the entire data holds and calculate the size with it.<br>
Currently you might be checking the size of the Python object itself.</p>
<p>If I understand your use case correctly, you are interested to see how much memory a batch would use?<br>
If so, this should work:</p>
<pre><code class="lang-python"># get the batch from the DataLoader
batch = torch.randn(16, 3, 224, 224)
print('{}MB'.format(batch.nelement() * batch.element_size() / 1024**2))
&gt; 9.1875MB
</code></pre>
What functions in PyTorch do I use to interpolate within existing image without resizing

username: csailnadi
cooked: <p>I want to <em>get replace zero-values using cubic interpolation within an image without resizing</em>. Is it possible to achieve it with PyTorch?</p>
<p>It seems like <code>torch.nn.functional.interpolate</code> only interpolates for resizing.</p>

username: ptrblck
cooked: <p><code>F.interpolate</code> will most likely call <a href="https://github.com/pytorch/pytorch/blob/ad2305e5562120b624b9857efc7bdf60a7557493/aten/src/ATen/native/UpSampleBicubic2d.cpp#L9-L37">this method</a> and just copy the inputs to the output tensor.<br>
I think it’s just possible to really trigger the interpolation by changing the size slightly.</p>
<p>Do you need to backpropagate through this interpolation? If no, then you could use a numpy or scipy implementation. I’m not sure how general this use case is, but it might be worth to create a feature request on GitHub and discuss your use case there.</p>

username: Shivang_Patel
cooked: <p><a class="mention" href="/u/csailnadi">@csailnadi</a> Did you able to do it? It would be a great help if you can share.</p>
Question about pytorch KL divergence loss

username: jhp
cooked: <p>Hi I have a quick question about kl divergence loss in Pytorch</p>
<ol>
<li>
<p>Is it okay to use sigmoid instead of softmax for input? Most of the case I noticed that softmax probability distribution is used, but I wonder does that make sense to use kl divergence loss for multi-label target.</p>
</li>
<li>
<p>I want to use KL divergence loss by giving more than two dimension input and target. In this case, does KL div loss compare the distritbution between last dimension of input and target? Or does it compare all the dimension except for the batch dim?</p>
</li>
</ol>
<p>thanks</p>

username: KFrank
cooked: <p>Hi Jhp!</p>
<aside class="quote no-group" data-username="jhp" data-post="1" data-topic="133196" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jhp/48/31228_2.png" class="avatar"> jhp:</div>
<blockquote>
<ol>
<li>Is it okay to use sigmoid instead of softmax for input?</li>
</ol>
</blockquote>
</aside>
<p>Not really.</p>
<aside class="quote no-group">
<blockquote>
<ol start="2">
<li>does KL div loss compare the distritbution between last dimension of input and target?</li>
</ol>
</blockquote>
</aside>
<p><code>KLDivLoss</code> doesn’t care what any of the dimesions – including<br>
the batch dimension – are.  It simply performs an element-wise<br>
computation and takes the mean (but see its <a href="https://pytorch.org/docs/1.9.1/generated/torch.nn.KLDivLoss.html" rel="noopener nofollow ugc">documentation</a> for<br>
how its <code>reduction = 'batchmean'</code> and deprecated <code>reduce</code>  and<br>
<code>size_average</code> constructor arguments affect its treatment of the<br>
batch dimension).</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.9.0'
&gt;&gt;&gt; _ = torch.manual_seed (2021)
&gt;&gt;&gt; input = -torch.randn (3, 5).abs()
&gt;&gt;&gt; target = torch.rand (3, 5)
&gt;&gt;&gt; torch.nn.KLDivLoss() (input, target)
&lt;path_to_pytorch_install&gt;\torch\nn\functional.py:2742: UserWarning: reduction: 'mean' divides the total loss by both the batch size and the support size.'batchmean' divides only by the batch size, and aligns with the KL div math definition.'mean' will be changed to behave the same as 'batchmean' in the next major release.
  "reduction: 'mean' divides the total loss by both the batch size and the support size."
tensor(0.3705)
&gt;&gt;&gt; torch.nn.KLDivLoss() (input.T, target.T)
tensor(0.3705)
&gt;&gt;&gt; torch.nn.KLDivLoss (reduction = 'none') (input, target).mean()
tensor(0.3705)
</code></pre>
<p>Whether the result of <code>KLDivLoss</code> represents a proper Kullback–Leibler<br>
divergence across certain dimensions depends on whether your<br>
<code>input</code> and <code>target</code> are themselves proper (log-)probability distributions<br>
across those dimensions.</p>
<p>Best.</p>
<p>K. Frank</p>

username: jhp
cooked: <p>Thank you for your kind answer. My concerns all clear!</p>
[POLL][RFC] DataParallel Deprecation

username: cbalioglu
cooked: <p>Hey everyone,</p>
<p>We wanted to let you know that we are considering the deprecation of DataParallel (<code>torch.nn.DataParallel</code>, a.k.a. DP) module with the upcoming v1.11 release of PyTorch. Our plan is to keep DataParallel in maintenance mode for the 12 months following the v1.11 release and afterwards completely remove it from our code base. No matter whether we follow this plan or not, we still highly encourage everyone relying on DataParallel to onboard with the Distributed Data Parallel (<code>torch.distributed</code>, a.k.a. DDP) module as we consider it the future of PyTorch Distributed.</p>
<p>Your feedback is very important to us, so please let us know your thoughts on our <a href="https://github.com/pytorch/pytorch/issues/65936" rel="noopener nofollow ugc">GitHub post</a>.</p>

username: albanD
action_code: pinned_globally.enabled
Doubt with torch.from_numpy with uint16 and how to_tensor manage these kinds of images

username: Dhorka
cooked: <p>Hi,</p>
<p>I have a doubt related to the function torch.from_numpy. I’m trying to convert a numpy array that contains uint16 and I’m getting the following error:</p>
<blockquote>
<p>TypeError: can’t convert np.ndarray of type numpy.uint16. The only supported types are: float64, float32, float16, int64, int32, int16, int8, uint8, and bool.</p>
</blockquote>
<p>I suppose one way to solve that is to convert my uint16 numpy array to a numpy array with int32. Is there another way to solve this issue? Am I doing something wrong?</p>
<p>On the other hand, I was wondering how <strong>to_tensor method from torchvision</strong> is able to transform uint16 images. I checked the <a href="https://github.com/pytorch/vision/blob/cd2b7f071cbfb207400e70deaf4b1b2c46c3e47c/torchvision/transforms/functional.py#L72" rel="noopener nofollow ugc">source code</a>  and I realized that there is a conversion to int16… However, in the official <a href="https://pillow.readthedocs.io/en/stable/handbook/concepts.html#concept-modes" rel="noopener nofollow ugc">documentation</a> of PIL library specifies that the I;16 mode indicate that the pixel contains an unsigned int16 pixel. Is there a bug on this spcific line? Am I missing something?</p>
<p>Thanks,</p>

username: albanD
cooked: <p>Hi,</p>
<p>I think the answer here depends on the use case.<br>
If you want to be use not to loose any precision, then yes you want to convert it to an int32 before converting to pytorch.<br>
If you don’t want to use double the space, you can use int16 and it should work just fine as long as your numbers are not too big.</p>
<p>Do you encounter any issue with the choice made by torchvision?</p>

username: Dhorka
cooked: <aside class="quote no-group" data-username="albanD" data-post="2" data-topic="86410">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar"> albanD:</div>
<blockquote>
<p>Do you encounter any issue with the choice made by torchvision?</p>
</blockquote>
</aside>
<p>Yes, it my case my numbers are big and I need all the space.</p>

username: albanD
cooked: <p>In that case, I guess the simplest is going to be make sure your dataset returns Tensors directly. And convert them to int32 before converting them to torch Tensors.</p>

username: Dhorka
cooked: <p>I’m going to do that. Thanks. However, maybe It will be a good idea to document this decision about how it is managed uint16 by ToTensor() method.</p>

username: albanD
cooked: <p>cc <a class="mention" href="/u/fmassa">@fmassa</a> what do you think?</p>

username: Arun_Kumar4
cooked: <p>i have to change uint16 to int64 but i have range values of  0 -9 for y varaible inj neural network<br>
tell me solution for this please</p>

username: Arun_Kumar4
cooked: <p>I have uint16 target image between 0  and 9 values how to use in pytorch dataloader</p>
Best loss for super ressolution?

username: Samuel_Bachorik
cooked: <p>Hi I want to ask if any of you have experience with training super resolution model. Can you please recommend to me loss for it ? I know MSE loss is used for it but if you have any tuned loss I would like to try <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=10" title=":smiley:" class="emoji" alt=":smiley:"></p>
How inplace sin op work? It really save memory when requires_grad is False?

username: wyg1997
cooked: <p>Hi, I have a question how inplace sin could use output_value to calculate input_grad?</p>
<p>As the function: y = sin(x)，x_grad = grad * sign * sqrt(1 - sin^2(x)). But we don’t know that the sign is positive or negative.</p>
<p>It looked as though it must save the value of original x tensor. But it can’t save memory in this way.</p>
<p>Can we do the inplace operator and save memory in the same time?</p>

username: albanD
cooked: <p>Hi,</p>
<p>I am afraid you can’t no. You have to save something of the size of the input. (either the sign or the input itself). So in core, we save a clone of the input.</p>
How to use dynamically changed customized dataset with DistributedDataParallel?

username: warrenyuan
cooked: <p>Hi, I am new to DistributedDataParallel. Recently, I want to use DDP with the customized dataset, and the dataset will be dynamically changed after each epoch. I want to know, how to meet these needs. The following is the pseudo-code I wrote, I don’t know if it is correct.</p>
<pre><code class="lang-auto">def train(gpu, args):
    ############################################################
    rank = args.nr * args.gpus + gpu	                          
    dist.init_process_group(                                   
    	backend='nccl',                                         
   		init_method='env://',                                   
    	world_size=args.world_size,                              
    	rank=rank                                               
    )                                                          
    ############################################################
   model = MyModel()
   torch.cuda.set_device(gpu)
   model.cuda(gpu)
   batch_size = ...
   criterion = nn.CrossEntropyLoss().cuda(gpu)
   optimizer = torch.optim.SGD(model.parameters(), 1e-4)
    
    ###############################################################
   # Wrap the model
   model = nn.parallel.DistributedDataParallel(model,
                                                device_ids=[gpu])
   train_set = MyDataset()
   train_sampler = torch.utils.data.distributed.DistributedSampler(...)
   train_loader = torch.utils.data.DataLoader(
    	dataset=train_dataset,
       batch_size=batch_size,
      sampler=train_sampler)    # 
   for i in range(args.epoch):
      # the training code
     ...
      # the validation code
     ...
     # I modified the dataset. so, should I reuse the DistributedSampler? I am confused if it is correct with the following a few lines.
    new_trainset = MyDataset()
    new_train_sampler = torch.utils.data.distributed.DistributedSampler(...)
    new_train_loader = torch.utils.data.DataLoader(
    	dataset=train_dataset,
       batch_size=batch_size,
      sampler=train_sampler)    # 
   ...
</code></pre>
<p>The confusing thing for me is when I modified the original dataset, should I reuse the DistributedSampler? or it will automatically recognize the change? If I need to re-new a sampler, should I use barrier()?</p>
<p>Thank you very much for your time and your help!</p>
BUG Shared weights CPU/CUDA

username: Patte_d_Encre
cooked: <p>I am trying to share weights between two layers <code>d1</code> and <code>d2</code> (with a <code>transpose</code>), and I get strange behavior.<br>
The weights are shared on the <code>bloc</code> .</p>
<ul>
<li>bloc on position 1:
<ul>
<li>using <code>device('cpu')</code>: the weights are shared</li>
<li>using <code>device('cuda')</code>: the weights are not shared, the assert fail after the first <code>optimizer.step</code>
</li>
</ul>
</li>
<li>bloc on position 2:
<ul>
<li>using <code>device('cpu')</code>: the weights are shared</li>
<li>using <code>device('cuda')</code>: the weights are shared</li>
</ul>
</li>
</ul>
<p>=&gt; I would like to have the bloc on position 1 however. In a clean way.</p>
<pre><code class="lang-auto">import random
import torch
from torch.nn import Linear, Sequential, Sigmoid, Parameter

device = torch.device('cuda')  # 'cuda' 'cpu'

d1 = Linear(5, 2, True)
d2 = Linear(2, 5, True)

# POSITION 1
# start bloc
d1_w = d1.weight
d2.weight = Parameter(d1_w.transpose(0, 1))
d2_w = d2.weight
# end bloc

D = Sequential(
    *[Sequential(d1, Sigmoid()),
    Sequential(d2, Sigmoid()),]
).to(device)

# POSITION 2



# check if the weights are shared as the start
assert (d1_w == d2_w.transpose(0, 1)).all()
print(d1_w)
print(d2_w.transpose(0, 1))

optimizer = torch.optim.Adam(D.parameters(), 0.001)
loss_func = torch.nn.MSELoss()

for _ in range(40):
    # just random training function
    optimizer.zero_grad()

    input = torch.Tensor([random.randint(1, 40) for _ in range(10)]).reshape([2, 5]).to(device)
    target = torch.Tensor([random.randint(1, 40) for _ in range(10)]).reshape([2, 5]).to(device)
    hat_target = D(input)

    loss = loss_func(hat_target, target)
    loss.backward()
    optimizer.step()

    # check if the weight are still equals
    d1_w = D[0][0].weight
    d2_w = D[-1][0].weight
    print("====")
    print(d1_w)
    print(d2_w.transpose(0,1))
    assert (d1_w == d2_w.transpose(0, 1)).all()
</code></pre>
Torchaudio bug when saving mp3-files?

username: paulrev
cooked: <p>Hi,<br>
I am using torchaudio to load and save audio files but the number of samples seems to be wrong.</p>
<p>Here is my code:</p>
<pre><code class="lang-auto">path_audio = 'example.mp3'
save_path = 'example_new.mp3'

#show info about file
print(torchaudio.info(path_audio))
Output: AudioMetaData(sample_rate=16000, num_frames=90432, num_channels=1, bits_per_sample=0, encoding=MP3)

#load example file
audio_tuple = torchaudio.load(path_audio)
audio = audio_tuple[0]
samplerate = audio_tuple[1]

print(audio.shape[1])
Output: 89856 #why is this different from info?

#cut audio to 1 second @ 16kHz
audio = audio[:,0:16000]

#check
print(audio.shape[1])
Output: 16000

#save audio and get info of saved file
torchaudio.save(save_path, audio, samplerate)
print(torchaudio.info(save_path))
Output: AudioMetaData(sample_rate=16000, num_frames=17280, num_channels=1, bits_per_sample=0, encoding=MP3) #why is this not 1600?

#load saved file:
reloaded_audio_tuple = torchaudio.load(save_path)
reloaded_audio = reloaded_audio_tuple[0] 
print(reloaded_audio.shape)
Output: torch.Size([1, 16704]) #this should be 16000! 
</code></pre>
<p>As you can see the output file has too many samples. They have not been added to the end of the file so i can’t just cut the file again. Can someone help me?</p>
<p>Thx!</p>
Multprocessing inference uses more memory of gpu0

username: kaka_zhao
cooked: <p>I use 2 GPUs for multi-process inference. I load models in the main process and put them to different GPUs in the sub-processes. However, I find that the processes of gpu1 always occupy 153M memory of gpu0.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/f/0f5a491115fefe9e68c6ab11da5829ee2acf3805.png" data-download-href="https://discuss.pytorch.org/uploads/default/0f5a491115fefe9e68c6ab11da5829ee2acf3805" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/f/0f5a491115fefe9e68c6ab11da5829ee2acf3805.png" alt="image" data-base62-sha1="2bOB3rBvqu2G23ged8W0kqxBepn" width="487" height="500" data-dominant-color="262725"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">727×745 23 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<pre data-code-wrap="python"><code class="lang-python">    import torch.multiprocessing as multiprocessing
    multiprocessing.set_sharing_strategy('file_system')

    ...

    def main(self):
            ...

            self.create_model()

            multiprocessing.set_start_method('spawn', force=True)
            print('&gt;&gt;&gt; multi-processes running &lt;&lt;&lt;')

            param_list = [([video_idx, video_name, im_list, gt_list, lang, len(seqs)],
                           self.tester_cfg.num_gpu)
                          for video_idx, (video_name, im_list, gt_list, lang) in enumerate(seqs)]

            with multiprocessing.Pool(processes=self.tester_cfg.num_process) as pool:
                pool.starmap(self.sub, param_list)        


    def sub(self, seq_info, num_gpu):
        video_idx, video_name, im_list, gt_list, lang, seq_num = seq_info

        worker_name = multiprocessing.current_process().name
        worker_id = int(worker_name.split('-')[1]) - 1
        gpu_id = worker_id % num_gpu
        torch.cuda.set_device(gpu_id)
        rank = worker_id
        print('start rank {} on gpu {}'.format(rank, gpu_id))

        self.model_to_device()
        
        ...
</code></pre>
No "RuntimeError: mat1 and mat2 shapes cannot be multiplied" when using GPU

username: Sohrabbeig
cooked: <p>Hi,</p>
<p>When running this piece of code using GPU on google colab, I expect to receive “RuntimeError: mat1 and mat2 shapes cannot be multiplied”, however, I do not. Obviously, the shape of those consecutive Linear layers mismatch, and I get that RuntimeError while using CPU. Could anyone explain why?</p>
<pre><code class="lang-auto">import os
import torch
from torch import nn

device = 'cuda' if torch.cuda.is_available() else 'cpu'
</code></pre>
<p>output:</p>
<blockquote>
<p>Using cuda device</p>
</blockquote>
<pre><code class="lang-auto">class NeuralNetwork(nn.Module):
    def __init__(self):
        super(NeuralNetwork, self).__init__()
        self.flatten = nn.Flatten()
        self.linear_relu_stack = nn.Sequential(
            nn.Linear(28*28, 513),
            nn.ReLU(),
            nn.Linear(512, 520),
            nn.ReLU(),
            nn.Linear(512, 10),
        )

    def forward(self, x):
        x = self.flatten(x)
        logits = self.linear_relu_stack(x)
        return logits

model = NeuralNetwork().to(device)
print(model)
</code></pre>
<p>output:</p>
<blockquote>
<p>NeuralNetwork(<br>
(flatten): Flatten(start_dim=1, end_dim=-1)<br>
(linear_relu_stack): Sequential(<br>
(0): Linear(in_features=784, out_features=513, bias=True)<br>
(1): ReLU()<br>
(2): Linear(in_features=512, out_features=520, bias=True)<br>
(3): ReLU()<br>
(4): Linear(in_features=512, out_features=10, bias=True)<br>
)<br>
)</p>
</blockquote>
<pre><code class="lang-auto">X = torch.rand(1, 28, 28, device=device)
logits = model(X)
pred_probab = nn.Softmax(dim=1)(logits)
y_pred = pred_probab.argmax(1)
print(f"Predicted class: {y_pred}")
</code></pre>
<p>Output:</p>
<blockquote>
<p>Predicted class: tensor([5], device=‘cuda:0’)</p>
</blockquote>

username: ptrblck
cooked: <p>You might be hitting <a href="https://discuss.pytorch.org/t/pytorch-is-allowing-incorrect-matrix-multiplication-when-using-cuda/130664">this known and already fixed issue</a> so please update to the latest release (1.9.1).</p>
Testing model for ImageNet Dataset

username: KanZa
cooked: <p>I want to train this model (<a href="https://github.com/razvancaramalau/Sequential-GCN-for-Active-Learning/blob/master/load_dataset.py" class="inline-onebox" rel="noopener nofollow ugc">Sequential-GCN-for-Active-Learning/load_dataset.py at master · razvancaramalau/Sequential-GCN-for-Active-Learning · GitHub</a>) for the ImageNet dataset. I have already downloaded the dataset and added the following lines but It is giving errors and loading the ImageNet dataset from the root folder.</p>
<p>elif dataset == ‘imagenet’:<br>
data_train = ImageNet(root=’./data/imagenet’, train=True, download=True, transform=train_transform)<br>
data_unlabeled = MyDataset(dataset, True, test_transform)<br>
data_test  = ImageNet(root=’./data/imagenet’, train= False, download=True, transform=test_transform)<br>
NO_CLASSES = 1000<br>
adden = 3000<br>
no_train = NUM_TRAIN</p>

username: ptrblck
cooked: <p>What kind of error are you seeing? Could you post the stacktrace here by wrapping them into three backticks ```?</p>
Segmentation masks

username: FilipAndersson245
cooked: <p>i have a mask file saved as a .tiff file ranging from 0…255.<br>
I would like to split it into <em>X</em> classes so my tensor look something like this [batch, classes, width, height].<br>
My network currently does Binary classification and outputs [9, 2, 256, 256].</p>
<p>So i would like to match this with the segmentation mask?, and understand how i should do if i wannt to increase the number of classes?</p>

username: ptrblck
cooked: <p>The segmentation mask in a multi-class classification use case should have the shape <code>[batch_size, height, width]</code> and contain the class indices in the range <code>[0, nb_classes-1]</code>.<br>
Does your target image contain 256 unique classes? If not, you should map each color value to a class index.</p>
<p>The output of your model should have the shape <code>[batch_size, nb_classes, height, width]</code>.</p>

username: Arun_Kumar4
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="53914">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>nb_classes</p>
</blockquote>
</aside>
<p>i get done same but i get same size output image<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/8/285775b899b46f23bc932a96bf0e860780984e4a.png" data-download-href="https://discuss.pytorch.org/uploads/default/285775b899b46f23bc932a96bf0e860780984e4a" title="img"><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/8/285775b899b46f23bc932a96bf0e860780984e4a.png" alt="img" data-base62-sha1="5KStNJ476JNVYAVqHr8U5tucfge" width="415" height="500" data-dominant-color="0C0C0C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">img</span><span class="informations">490×589 14.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
i get wrong image<br>
i want to get like this image below<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/5/f5acd97cd85b9721aedc0307f32120bdd0905fd4.png" data-download-href="https://discuss.pytorch.org/uploads/default/f5acd97cd85b9721aedc0307f32120bdd0905fd4" title="img2"><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/5/f5acd97cd85b9721aedc0307f32120bdd0905fd4.png" alt="img2" data-base62-sha1="z3lj8MG4HSSvQ9XvgzmsqkHd2ao" width="485" height="500" data-dominant-color="181818"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">img2</span><span class="informations">503×518 1.49 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Compute per sample gradient for a normal model

username: abstrcode
cooked: <p>Hello, How to compute a per sample gradient for a usual model using Opacus? In this case, we don’t care about the privacy issue.</p>

username: ashkan_software
cooked: <p>Hello,</p>
<p>Excellent question. We recently wrote a blog about this: <a href="https://medium.com/pytorch/differential-privacy-series-part-2-efficient-per-sample-gradient-computation-in-opacus-5bf4031d9e22" class="inline-onebox" rel="noopener nofollow ugc">Differential Privacy Series Part 2 | Efficient Per-Sample Gradient Computation in Opacus | by PyTorch | PyTorch | Sep, 2021 | Medium</a></p>
kaldi.fbank alternative in librosa?

username: Rida_Saleem
cooked: <p>Hi everyone,<br>
I would really appreciate if someone could let me know how to replicate <strong>compliance.kaldi.fbank()</strong> function in <strong>librosa</strong> ? I’ve gone through alot of literature and forums but haven’t really found a way to replicate their parameters. Any help would be appreciated.</p>
Computing gradient w.r.t the network parameters on each input sample

username: alldbi
cooked: <p>Hi,</p>
<p>I need to compute the norm of the grads w.r.t. the network parameters on each input sample. Right now I am doing this as follows:</p>
<pre><code class="lang-auto">pred = net(input)
for i in range(len(input)):

    loss = criterion(pred[i:i + 1], targets[i:i + 1])  # only consider the loss on i-th data
    net.zero_grad()
    loss.backward(retain_graph=True)

    # gather gradients wrt the weights on the current sample
    grad_l = []
    for param in net.parameters():
        # check if the param has grad
        if param.grad is not None:
            grad_l.append(torch.clone(param.grad.view(-1)))
    grad_l = torch.cat(grad_l, 0)
    norm = grad_l.norm(2)
    grad_w_norm_batch.append(norm.detach())
</code></pre>
<p>However, it is notably slow. Is there any other way/trick to do so in a faster manner?</p>
Train the input for a frozen network

username: smu226
cooked: <p>Hello! I have a network that takes as input 3 numbers and outputs 2 and I trained it on a given data set, to predict the 2 numbers. Now I would like to freeze all the layers of the network, and make the input trainable. Such that, for a given output (consisting of 2 numbers), the gradient descent will not touch the NN, but it will adjust the input such that I get the output I want. I am not sure how to do this in practice. After I trained the network I tried this:</p>
<pre><code class="lang-auto">for param in model.parameters():
    param.requires_grad = False

input = Variable(torch.randn(bs, 3).cuda(), requires_grad=True)

lrs = 1e-2
optimizer = optim.Adam(input.parameters(), lr = lrs)
</code></pre>
<p>But it doesn’t allow me to call the optimizer like that for the input. How can I do what I need? Thank you!</p>

username: JuanFMontesinos
cooked: <p>first of all, Variable was deprecated long ago and no longer used.<br>
You can achieve that by passing a list of tensors<br>
[input]<br>
to the optimizer (or something really similar to that), read optimizer docs.</p>
Multi-target not supported at ClassNLLCriterion.c:22

username: Shar
cooked: <p>Hi,<br>
I’m trying to use the function:<br>
<code>loss = criterion(outputs, labels)</code><br>
where the inputs are:<br>
labels=<br>
<code>tensor([[2]])</code><br>
outputs=<br>
<code>tensor([[274.3095, 842.6060, -52.4284]], grad_fn=&lt;AddmmBackward&gt;)</code></p>
<p>but I get the error:</p>
<p>multi-target not supported at /Users/distiller/project/conda/conda-bld/pytorch_1570710797334/work/aten/src/THNN/generic/ClassNLLCriterion.c:22</p>
<p>Any help<br>
Thanks</p>

username: KFrank
cooked: <p>Hi Shar!</p>
<aside class="quote no-group" data-username="Shar" data-post="1" data-topic="133085" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/e274bd/48.png" class="avatar"> Shar:</div>
<blockquote>
<p>labels=<br>
<code>tensor([[2]])</code></p>
</blockquote>
</aside>
<p>Your <code>labels</code> has one too many dimensions.  It appears that<br>
you are using a cross–entropy-type loss criterion.  Pytorch’s<br>
<code>CrossEntropyLoss</code> (and related loss criteria) expect your<br>
<code>outputs</code> to have shape <code>[nBatch, nClass]</code> and your labels<br>
to have shape <code>[nBatch]</code> (with no <code>nClass</code> dimension).  If, for<br>
some reason, your <code>labels</code> naturally has a trailing singleton<br>
dimension, you can <code>squeeze()</code> it away.</p>
<p>Thus:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.9.0'
&gt;&gt;&gt; labels = torch.tensor ([[2]])
&gt;&gt;&gt; outputs = torch.tensor ([[274.3095, 842.6060, -52.4284]])
&gt;&gt;&gt; criterion = torch.nn.CrossEntropyLoss()
&gt;&gt;&gt; labels.shape
torch.Size([1, 1])
&gt;&gt;&gt; outputs.shape
torch.Size([1, 3])
&gt;&gt;&gt; loss = criterion (outputs, labels.squeeze (1))
&gt;&gt;&gt; loss
tensor(895.0344)
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

username: Shar
cooked: <p>It worked. Many thanks!</p>
(myerror)TypeError: forward() takes 2 positional arguments but 3 were given

username: bingqing_liu
cooked: <pre><code class="lang-auto">import torch.nn as nn
import torch
class structural_attention_layer(nn.Module):
    def init(self):
        super(structural_attention_layer,self).init()

    def forward(self,a,b):
        return torch.randn(a,b)
    
s_att=nn.Sequential()
s_att.add_module(name="structural_attention_layer",module=structural_attention_layer())
s_att(1,2)
</code></pre>

username: ptrblck
cooked: <p>Is this a double post from <a href="https://discuss.pytorch.org/t/forward-takes-2-arguments-but-3-were-given/132877">here</a> with an already provided solution or what would be the difference in this code snippet?</p>
Dataset for Dataloader

username: Leo2
cooked: <p>Hi<br>
I have a huge dataset that needs to be loaded into colab in segments. I managed to load the whole dataset as a one off in order to do the necessary processing/cleaning and put it into a giant pandas dataframe but i don’t want to keep it in RAM when i do my training. So I then used to_pickle to save 10 seperate dataframes, (obviously containing many samples each) into files.</p>
<p>I want to use Dataloader because i’ve got a collate_function, and it requires a Dataset.</p>
<p>I designed a custom Dataset to load a single dataframe at a time.</p>
<p>class Custom_dataset(torch.utils.data.Dataset):<br>
def <strong>init</strong>(self):<br>
self.dataframes = os.listdir(’/content/drive/MyDrive/processed_data’)</p>
<p>def <strong>getindex</strong>(self, idx):<br>
df= pd.read_pickle(self.dataframes[idx])<br>
return df</p>
<p>def <strong>len</strong>(self):<br>
return len(self.dataframes)</p>
<p>Is there anyway I can design a custom dataset that will load one file at a time and also one sample from that file at a time. The one i’ve got at the moment obviously loads the entire dataframe</p>
Pytorch Model ios deployment

username: IrfanUmar
cooked: <p>I have made a pytorch model using resnet architecture, I made a model.pt file and I want to use this model file in iOS application. Model is imaga classification model and use CUDA.</p>
<p>Whenever I load this file in iOS my application crashes and gives me this error.</p>
<p>“Could not run ‘aten::empty_strided’ with arguments from the ‘CUDA’ backend. This could be because the operator doesn’t exist for this backend, or was omitted during the selective/custom build process (if using custom build). If you are a Facebook employee using PyTorch on mobile, please visit <a href="https://fburl.com/ptmfixes" class="inline-onebox" rel="noopener nofollow ugc">Internal Login</a> for possible resolutions. ‘aten::empty_strided’ is only available for these backends: [CPU, BackendSelect, Named, ADInplaceOrView, AutogradOther, AutogradCPU, AutogradCUDA, AutogradXLA, UNKNOWN_TENSOR_TYPE_ID, AutogradMLC, Tracer, Autocast, Batched, VmapMode].”</p>
<p>What is work around for it. Also I want to know if models are using CUDA gpu then can we use them in mobile deployments ?</p>

username: ptrblck
cooked: <p>I guess you might have stored the model directly via <code>torch.save</code> while it was still on the GPU?<br>
If so, you might need to save the scripted model from the CPU so that your iOS application could load it without any CUDA dependency.</p>
Is it possible to updated optimizer based on loss from a "lookup table"

username: chtapodi
cooked: <p>I am working on a GAN where the generator outputs a one-hot tensor.<br>
Each index in the tensor has an associated array of characteristics that are not involved with the generator. I am trying to figure out if it is possible to calculate loss based on the sum of these characteristics e.g. <code>sum(characteristic_tensor[where(onehot_tensor&gt;0)])</code>.</p>
<p>My current approach is training one discriminator on onehot tensors, and the other on summed characteristics. My goal is to update the generator based on the loss for both of these discriminators.</p>
<p>The problem I am hitting is that I cannot determine if it is possible to convert the onehot tensor to a summed tensor and still have its graph connected to the generator, or if this is even something that it makes sense to do.</p>
<p>The other approach I can think of is detaching the tensor, summing it, calculating its loss, and then updating the optimizer based on this gradient. However I cannot figure out how to do this if it is possible.</p>
<p>Is what I am trying to do feasible with how autograd works?</p>
<pre><code class="lang-auto">#This is how I am trying to update the generator after the discriminators have been trained
    noise = torch.randn(b_size, 100, device=device)
    
    generator.zero_grad()
    
    #generate fake data (b_size,46)
    fake_batch=generator(noise)
    
    #get onehot loss
    true_labels=torch.empty((b_size,), dtype=torch.float, device=device).uniform_(0.9, 1.00)
    
    e_output = e_discriminator(fake_batch).view(-1)
    e_err=criterion(e_output,true_labels)
    
    #get sum loss
    #convert the onhot tensor to a sum of attributes (b_size,30)
    sum_batch=torch.from_numpy(controller.encoded_batch_to_sum(fake_batch.detach().cpu().numpy())).float().to(device)
    
    f_output = f_discriminator(sum_batch).view(-1)
    f_err=criterion(f_output,true_labels)
    
    err=e_err+f_err
    err.backward()
    g_opt.step()
</code></pre>

username: JuanFMontesinos
cooked: <p>No, once you call the lookuptable and get a tensor from there it diesconnets as there is no differentiable operation between both.<br>
The only coherent thing you can do is to get that tensor by a multiplication or some sort of attention.<br>
you can set values&lt;0 to zero. Thus, no gradient will flow through there.</p>
<p>Another plausible option is you to impose a loss on the probability of the lookup. The  use the lookup and use a layer to combine those characteristics with your output.</p>

username: chtapodi
cooked: <p>Thank you. I was concerned that this would be the case.</p>
Transforms Error: TypeError: pic should be Tensor or ndarray. Got <class 'torch.Tensor'>

username: Wilan
cooked: <p>I’m trying to pre-process an array of images, <code>frames_tensor</code>, of shape <code>(8393, 3, 224, 224)</code> and type <code>torch.Tensor</code> using this transformation sequence:</p>
<pre><code class="lang-auto">class Frames_Dataset(Dataset):

            def __init__(self, frames):
                self.preprocess = transforms.Compose([
                    transforms.ToPILImage(),
                    transforms.Resize(256),
                    transforms.CenterCrop(224),
                    transforms.ToTensor(),
                    transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
                ])

                self.frames = self.preprocess(frames)

            def __getitem__(self, index):
                return self.frames[index]

            def __len__(self):
                return len(self.frames)

frames_dataset = Frames_Dataset(frames_tensor)
</code></pre>
<p>I’m getting error:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "Features/main.py", line 33, in &lt;module&gt;
    extract_features(file, subdir, features_name, fps, args.overwrite_features)
  File "/media/wilsonchan/Giratina/Projects/Badi Highlights/v2 AceOne/SoccerNetv2-DevKit/Features/Features.py", line 153, in extract_features
    frames_dataset = Frames_Dataset(frames_tensor)
  File "/media/wilsonchan/Giratina/Projects/Badi Highlights/v2 AceOne/SoccerNetv2-DevKit/Features/Features.py", line 143, in __init__
    self.frames = self.preprocess(frames)
  File "/home/wilsonchan/anaconda3/envs/AceOne-Features/lib/python3.7/site-packages/torchvision/transforms/transforms.py", line 49, in __call__
    img = t(img)
  File "/home/wilsonchan/anaconda3/envs/AceOne-Features/lib/python3.7/site-packages/torchvision/transforms/transforms.py", line 110, in __call__
    return F.to_pil_image(pic, self.mode)
  File "/home/wilsonchan/anaconda3/envs/AceOne-Features/lib/python3.7/site-packages/torchvision/transforms/functional.py", line 103, in to_pil_image
    raise TypeError('pic should be Tensor or ndarray. Got {}.'.format(type(pic)))
TypeError: pic should be Tensor or ndarray. Got &lt;class 'torch.Tensor'&gt;.
</code></pre>
<p>My array is a Tensor, but  <code>F.to_pil_image</code> is telling me it doesn’t recognize this?</p>
<p>Note: I’ve seen <a href="https://github.com/pytorch/vision/issues/408" rel="noopener nofollow ugc">this post</a>, but would like to keep my images as 3-channels, instead of 1-channel.</p>
<p>If anyone has any idea what the issue is, I’d greatly appreciate the help.</p>

username: Keyv_Krmn
cooked: <p>you can keep your 3-channel, just double check if your input is 3-dimensional [channel, H, W]. if you are using grayscale, then reshape your tensor to [1, H, W]</p>
DataLoader glitches every once in a while when num_workers > 0

username: stevenwjy
cooked: <p>I seem to encounter the following behavior for DataLoader:</p>
<ul>
<li>If I set num_workers &gt; 0, a lot of iterations seem to take very fast to load (perhaps due to the caching) but then every X iterations it will sort of have a glitch and take quite some time to load (perhaps due to the preprocessing).</li>
<li>Meanwhile, if I set num_workers = 0, all iterations seem to take slightly longer to load but the time is consistent.</li>
</ul>
<p>For example, to put some benchmark (avg. data loading time every 100 iterations):</p>
<ul>
<li>num_workers = 8:
<ul>
<li>up to 800-th step: avg. 0.0004s / step</li>
<li>800-900th step: avg. 0.65s / step (note that the glitch prob happen in only one of the steps)</li>
</ul>
</li>
<li>num_workers = 0:
<ul>
<li>avg. 0.015s / step</li>
</ul>
</li>
</ul>
<p>If we do some math for 900 steps:</p>
<ul>
<li>num_workers = 8: 0.0004 * 800 + 0.65 * 100 =  0.32 + 65  = 65.32s</li>
<li>num_workers = 0: 0.015 * 900 = 13.5s</li>
</ul>
<p>I’m aware that the huge spike in time is probably because it tries to cache for future iterations, but still the difference in time is quite significant and the inconsistency is quite surprising to me.</p>
<p>Is this kind of behavior expected?</p>

username: ptrblck
cooked: <p>If the workers aren’t able to pre-load the batches in time, you would see spikes once in a while (or every <code>num_workers</code> iterations), but your issue sounds a bit strange, since the data loading speed tanks towards the end. Are you seeing the same slowdown in each epoch?</p>

username: stevenwjy
cooked: <p>Yeah, it seems to happen consistently every 800 steps upon loading data using the dataloader in my case</p>
ONNX export after QAT

username: kazimpal87
cooked: <p>Hi,</p>
<p>I’m just wondering if there is a way to export a model trained using quantisation aware training to onnx? There seem to be conflicting answers in various places saying that its not supported, and others that it is now supported. Is there some true answer for this?</p>
<p>If it is supported, is there an example somewhere? Do we export the “prepared” model, or the “converted” model?</p>
<p>Thanks</p>

username: Vasiliy_Kuznetsov
cooked: <p>hi <a class="mention" href="/u/kazimpal87">@kazimpal87</a> , currently we do not officially support exporting quantized models via ONNX.  We would definitely welcome contributions in this area.</p>
Reshape tensor with RGB value

username: cbd
cooked: <p>In following program “tensor_m” with torch.Size([1, 3, 3, 3]) reshape to “tensor_n” with torch.Size([1, 9, 3]). Assume “tensor_m” of 3 RGB channel with 3x3 value with first value of each channel 0.63, 0.22 and 0.77 respectively.</p>
<p>I need “tensor_n” with reshape as RGB value. For an example first value should be [ 0.63, 0.22, 0.77 ] and so on. (first value of each channel instead of each channel converted in series as shown in output)</p>
<pre><code class="lang-auto">m = torch.rand(3,3,3)
m=m.unsqueeze(0)
print(m)
print(m.shape)
n=torch.reshape(m,[1,9,3])
print(n)

Output: tensor([[[[0.6372, 0.9886, 0.0787],
          [0.1122, 0.5088, 0.9846],
          [0.7976, 0.7544, 0.9155]],

         [[0.2244, 0.4629, 0.4907],
          [0.5833, 0.3484, 0.0845],
          [0.8927, 0.8919, 0.5495]],

         [[0.7770, 0.3146, 0.4091],
          [0.6119, 0.8719, 0.4748],
          [0.9458, 0.8806, 0.9648]]]])
torch.Size([1, 3, 3, 3])
tensor([[[0.6372, 0.9886, 0.0787],
         [0.1122, 0.5088, 0.9846],
         [0.7976, 0.7544, 0.9155],
         [0.2244, 0.4629, 0.4907],
         [0.5833, 0.3484, 0.0845],
         [0.8927, 0.8919, 0.5495],
         [0.7770, 0.3146, 0.4091],
         [0.6119, 0.8719, 0.4748],
         [0.9458, 0.8806, 0.9648]]])</code></pre>
Pip install torch==1.9.1+cu111 -f https://download.pytorch.org/whl/torch_stable.html is killed

username: Arturas_Druteika
cooked: <p>I am trying to install newest stable torch 1.9.1 witch uses cuda 11.1 (cuda 11.1 and cudnn is already installed and set up). For some reason when I run <code>pip3 install torch==1.9.1+cu111 torchvision==0.10.1+cu111 torchaudio==0.9.1 -f https://download.pytorch.org/whl/torch_stable.html</code> I get the output as follows:</p>
<pre><code class="lang-auto">Looking in links: https://download.pytorch.org/whl/torch_stable.html
Collecting torch==1.9.1+cu111
Killed
</code></pre>
<p>For some reason this process is killed. I have tried installing torch 1.9.1 which uses cuda 10.2 and it installed successfully. I have also tried installing torch 1.9.1 + cu111 using pycharm’s package handler (using GUI to install) and it worked as well.</p>
<p>Why it only doesn’t work when I try to install it using pip install?</p>

username: albanD
cooked: <p>Hi,</p>
<p>This “Killed” means that your computer ran out of ram and so the “pip” command was killed by the OS to save memory.<br>
You should make sure you have enough ram free before running the command.</p>
How to install PyTorch on ppc64le with pip?

username: Mahdi_Amrollahi
cooked: <p>I am going to build a virtual environment (python -m venv) and install PyTorch on a ppc64le machine. But there is no package in pip to install PyTorch, however it is available in conda. But I wanted not to use conda because I need to install some specific packages and versions. So, how can I install PyTorch on a ppc64le(IBM Power8) machine?</p>
InstanceNorm1d returns 0-filled tensor to 2D tensor.This is because InstanceNorm1d reshapes inputs to(1, N * C, ...) from (N, C,...) and this makesvariances 0

username: jpainam
cooked: <p>Hi. I’m using a unit batch-size and following this discussion, [<a href="https://discuss.pytorch.org/t/error-expected-more-than-1-value-per-channel-when-training/26274/2" class="inline-onebox">Error: Expected more than 1 value per channel when training - #2 by ptrblck</a>] (<a href="https://discuss.pytorch.org/t/error-expected-more-than-1-value-per-channel-when-training/26274/2" class="inline-onebox">Error: Expected more than 1 value per channel when training - #2 by ptrblck</a>)  I can switch to <code>InstanceNorm</code>.<br>
But this is the error i get when using</p>
<pre><code class="lang-python">x =  torch.randn(1, 512)
batch_norm = nn.BatchNorm1d(512)
inst_norm = nn.InstancceNorm1d(512)
batch_norm(x)
&gt;&gt; ValueError: Expected more than 1 value per channel when training, got input size torch.Size([1, 512])
inst_norm(x)
&gt;&gt;  InstanceNorm1d returns 0-filled tensor to 2D tensor.This is because InstanceNorm1d reshapes inputs to(1, N * C, ...) from (N, C,...) and this makesvariances 0.
x = torch.randn(32, 512)
out = batch_norm(x)
out.shape
&gt;&gt; torch.Size([32, 512])
</code></pre>
<pre><code class="lang-python">     File "/miniconda3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/miniconda3/lib/python3.9/site-packages/torch/nn/modules/instancenorm.py", line 56, in forward
    self._check_input_dim(input)
  File "//miniconda3/lib/python3.9/site-packages/torch/nn/modules/instancenorm.py", line 132, in _check_input_dim
    raise ValueError(
ValueError: InstanceNorm1d returns 0-filled tensor to 2D tensor.This is because InstanceNorm1d reshapes inputs to(1, N * C, ...) from (N, C,...) and this makesvariances 0.
</code></pre>

username: zetyquickly
cooked: <p>Hi <a class="mention" href="/u/jpainam">@jpainam</a></p>
<p>The manual <a href="https://pytorch.org/docs/stable/generated/torch.nn.InstanceNorm1d.html#torch.nn.InstanceNorm1d" rel="noopener nofollow ugc">says</a> that <code>nn.InstanceNorm1d</code> works with 3d tensors, your example of <code>x</code> has two dimensions.</p>
<p>Have you tried 3d tensors?</p>

username: zetyquickly
cooked: <p>Seems like it doesn’t work with 2d inputs but according to manual it should.</p>
<p>I’ve found this <a href="https://github.com/pytorch/pytorch/issues/60585" rel="noopener nofollow ugc">issue</a> that says <code>nn.InstanceNorm1d - Should be updated to support 1D or 2D inputs</code> without checkbox filled. It might be the case that this operation doesn’t work with 2d inputs for now.</p>

username: paulrev
cooked: <p>I did a training with an nn.InstanceNorm1d-layer and when I run it with a batch of inputs it works fine.:<br>
for example [5,40,100]<br>
but when I test it with a batch containing only one sample [1,40,100] it shows the same error message as you.</p>
<p>This is probably a bug</p>
Copy best model from GPU to CPU

username: Randall_Smith
cooked: <p>Hello all,</p>
<p>I want to keep around the best model while training without writing it to disk every time a new best model is encountered, which is what I’m currently doing.</p>
<p>Can I copy the model that is on the GPU over to the CPU somehow?  I don’t want to make an extra copy on the GPU since I need that memory.</p>
<p>Thanks!</p>

username: LeviViana
cooked: <p>You can keep a copy of its <code>state_dict()</code>. This should work:</p>
<pre><code class="lang-bash">from collections import OrderedDict
best_model_state_dict = {k:v.to('cpu') for k, v in model.state_dict().items()}
best_model_state_dict = OrderedDict(best_model_state_dict)
</code></pre>
<p>Later, you will be able to load that model by doing <code>model.load_state_dict(best_model_state_dict)</code></p>

username: Randall_Smith
cooked: <p>Thanks a bunch.  This worked for me.</p>

username: Chen_Zejun
cooked: <p>Nice method to copy model. Thank you all <img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=10" title=":grinning:" class="emoji" alt=":grinning:"></p>