akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Why does pytorch dataloader add a "._" infront of some file names?

rishabh135: <p>I am running a pretty basic model and loading the input data (images in this case ) using torchvision.datasets , I am facing an issue in that the  script is somehow inserting a “._” as a prefix on some of the names of the image files . Does any body knows why or a workaround ?</p>
<pre><code class="lang-auto">    raise batch.exc_type(batch.exc_msg)                              [1/1927]
IOError: Traceback (most recent call last):
  File "/opt/conda/envs/pytorch-py27/lib/python2.7/site-packages/torch/utils$data/dataloader.py", line 40, in _worker_loop
    samples = collate_fn([dataset[i] for i in batch_indices])
  File "/opt/conda/envs/pytorch-py27/lib/python2.7/site-packages/torchvision$datasets/folder.py", line 116, in __getitem__
    img = self.loader(path)
  File "/opt/conda/envs/pytorch-py27/lib/python2.7/site-packages/torchvision$datasets/folder.py", line 63, in default_loader
    return pil_loader(path)
  File "/opt/conda/envs/pytorch-py27/lib/python2.7/site-packages/torchvision/
datasets/folder.py", line 45, in pil_loader
    with Image.open(f) as img:
  File "/opt/conda/envs/pytorch-py27/lib/python2.7/site-packages/PIL/Image.py
", line 2519, in open
    % (filename if filename else fp))
IOError: cannot identify image file &lt;open file './anime-faces/aqua_eyes/._dan
booru_2559693_dc628b766d7142f2d2d9c75559e36eb5.jpg', mode 'rb' at 0x7f16dee79
810&gt;

</code></pre>

abdelkrim_zitouni: <p>I’m facing the same problem, did you solve it?</p>
Implementing batch gradient descent in neural network

Jeet: <p>Hi, I am trying to train a neural network with the help of batch gradient descent. I am not sure if I am doing it correctly because I am confronting a lot of errors and an incorrect output. The code is as mentioned -</p>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self, in_features):
        super().__init__()
        self.disc = nn.Sequential(
            nn.Linear(in_features, 4),
            nn.Sigmoid(),
            nn.Linear(4, 1),
            nn.Sigmoid(),
        )

    def forward(self, x):
        return self.disc(x)

net = Net(image)

for epoch in range(num_epochs):
    totalloss=torch.empty_like(lossNet)
    net.zero_grad()
    for sample in dataset:
         lossNet= xxxxx
         totalloss=totalloss+lossNet
   
    totalloss=torch.div(totalloss,5) #Since there are 5 samples in the dataset so taking the average
    net.zero_grad()    
    totalloss.backward(retain_graph=True)
    opt_net.step()
</code></pre>
<p>I would like to know if this is the correct way of solving batch gradient descent? If not then what is the alternative method to apply batch gradient descent in Pytorch?</p>

eqy: <p>I believe the typical approach is to capture the model’s parameters using the optimizer and to zero_grad on the optimizer. Please take a look at the MNIST example for more details:</p>
<p><a href="https://github.com/pytorch/examples/blob/main/mnist/main.py" rel="noopener nofollow ugc">examples/main.py at main · pytorch/examples (github.com)</a></p>
Preparing multi-track audio data for Dataset and DataLoader

brudal: <p>I’m currently creating a Wave-U-Net model for automatic mixing of audio and I’m stuck on building the dataset. The data I’m using is comprised of 8 audio tracks (corresponding to separate pieces in a drum kit) and a mixdown of these tracks (some recordings only have 7 audio tracks so in those instances I’ve created a tensor of zero values for silence – is this the right way to go about this?).</p>
<p>I’ve written the tensors to hdf format leaving me with a test.hdf5, val.hdf5 and train.hdf5. The audio tracks range from ~10s to 60s long, so my main question is; do I need to pad all of the data such that every single audio track is as long as the longest one (60s), or do I just need to verify that the tracks and mix that make up each recording are as long as each other? Are there any other additional steps needed in preparing my data? (e.g. do I need to stack tensors from the multi-tracks or something?)</p>
<p>Any help is appreciated</p>

JuanFMontesinos: <p>The default batch fun (the function that creates batches out of the individual output of each worker from the dataloader) stack tensors.<br>
This means all your tensors within the same category need to be the same length (ie, you need to pad)</p>
<p>WRT steps, it depends a lot on the problem, if you want to mix automatically I guess you may want to have control over the loudness of each track?</p>

brudal: <p>Hi <a class="mention" href="/u/juanfmontesinos">@JuanFMontesinos</a>, I’ve padded shorter tracks and cut longer ones so that all tracks in the dataset are 20s long to avoid long data preparation and training times. I’m not sure if I’m approaching the task the wrong way and was wondering if you could clarify some things. Currently I have converted all tracks to melspectrograms and stacked them together into one input tensor of size (32, 8, 64, 1536) and a target tensor (32, 2, 64, 1536) - is combining into one input tensor the correct way to format a mixing task or would separate tensors for each track as input be more suited? (I don’t know how multiple inputs work with torch networks or how to implement them)</p>

JuanFMontesinos: <p>Well first of all if you are using wave-unet u should be using raw waveforms.</p>
<p>Also, I find 20s of audio too large (does it fit the in the gpu?. There are very few papers working with such duration (maybe it’s ur case). I don’t understand what your target is, soz.</p>
<p>Lastly, for multiple inputs there are many approaches. You can pass those many inputs as channels (which sounds good to have to show the network all the info at once).</p>
A problem on torch.bmm()

hilbert_deng: <p>I got the correct result with torch1.2, while the result from torch1.8 is not reasonable. If there are  something wrong with torch.bmm()?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/4/948db3b513520fd23038de0461e3a3715d043b4b.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/948db3b513520fd23038de0461e3a3715d043b4b" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/4/948db3b513520fd23038de0461e3a3715d043b4b_2_690x345.jpeg" alt="image" data-base62-sha1="lcagadouJRm9HUoNyWFAUERLjK3" width="690" height="345" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/4/948db3b513520fd23038de0461e3a3715d043b4b_2_690x345.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/9/4/948db3b513520fd23038de0461e3a3715d043b4b_2_1035x517.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/9/4/948db3b513520fd23038de0461e3a3715d043b4b_2_1380x690.jpeg 2x" data-dominant-color="191919"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1497×750 139 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>Could you update to the latest stable release and check, if you would still be seeing unexpected results?</p>
GCP's AI explainability service for pytorch models (images)

Prachi: <p>Can we use the AI explainability service of the google cloud platform to deploy PyTorch models? I am unable to find resources relevant to this topic (similar to <a href="https://colab.research.google.com/github/GoogleCloudPlatform/ai-platform-samples/blob/master/notebooks/samples/explanations/tf2/ai-explanations-image.ipynb#scrollTo=mHF9VCProKJN" rel="noopener nofollow ugc">TensorFlow</a>).</p>
<ul>
<li>E.g. AI explainability service for an image data type that highlights input (feature) attributes (eg pixels important to predict an image’s class) using various algorithms.</li>
</ul>
<p>Any directing resources would be appreciable,<br>
thanks!</p>
Is is thread-safe to do `Tensor.to(device)` from multiple threads to the same GPU device?

Fei_Liu: <p>I’m reading many small files (tensors) to be loaded to the GPU memory. In this case, can I open a thread pool and use <code>.to</code> to move all of them to GPU? I have 8 GPUs in total, each will take ~2000 such tensors, and this is done in python with a <code>multiprocessing.pool.ThreadPool(32)</code>.</p>
<p>A more general question: is it safe to access the same GPU device through different threads but with thread local objects (still in python, e.g. using a thread pool) ? E.g. if I do <code>x = torch.rand(2, 2)</code> in two different threads and manipulate each thread’s <code>x</code> separately, will I possibly run into any errors / undefined behaviors?</p>

mrshenli: <aside class="quote no-group" data-username="Fei_Liu" data-post="1" data-topic="157783">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/fei_liu/48/36219_2.png" class="avatar"> Fei_Liu:</div>
<blockquote>
<p>A more general question: is it safe to access the same GPU device through different threads but with thread local objects (still in python, e.g. using a thread pool) ?</p>
</blockquote>
</aside>
<p>It should be safe, as long as you use CUDA stream/events to make sure that subsequent ops do not access the data before it’s ready. See the doc below.</p>
<p><a href="https://pytorch.org/docs/stable/generated/torch.cuda.Event.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.cuda.Event.html</a></p>
<p>BTW, if you only wanna avoid blocking Host-to-Device (H2D) copies, you can use pinned CPU memory and then set <code>non_blocking=True</code> in <code>Tensor.to()</code>, which will make the copy async.</p>
<blockquote>
<p>E.g. if I do <code>x = torch.rand(2, 2)</code> in two different threads and manipulate each thread’s <code>x</code> separately, will I possibly run into any errors / undefined behaviors?</p>
</blockquote>
<p>This should fine, because x in these two threads will use different CUDA memory blocks. However, one thing to note is that, Python GIL might kill perf for multi-threading. You should still be able to get some speedup though, as PyTorch drops GIL when entering the C++ side.</p>

Fei_Liu: <p>Thank you for the education! These completely answers my questions.</p>
DDP on 2 GPUs and singe GPU have different loss

sqiangcao: <p>In my case, the performance of DDP is lower than single card training. I guess the biggest difference between the two might be the data being fed into the network, not in the same order. So, I tried to control the data fed into the network is exactly the same at the first iteration, but the loss is still different. Can someone please tell me why there is a training difference between DDP and single GPUs?</p>

mrshenli: <p>Hey <a class="mention" href="/u/sqiangcao">@sqiangcao</a></p>
<p>DDP’s loss is local to the input of each process, and it keeps model replicas in sync by averaging gradients across all processes. So in each iteration, each process will have a different loss and depending on the loss function, you might need to tune things like learning rate when moving local training to DDP.</p>
<p>See more discussions here: <a href="https://discuss.pytorch.org/t/average-loss-in-dp-and-ddp/93306" class="inline-onebox">Average loss in DP and DDP</a></p>
Got stuck using dist.barrier

ojipadeson: <p>In order to get the precise result on test dataset, I use the following code:</p>
<pre><code class="lang-auto">val_stat = self.evaluate() if utils.is_main_process() else None
dist.barrier()
update(val_stat)
</code></pre>
<p>And it got stuck.</p>

mrshenli: <p>Hey <a class="mention" href="/u/ojipadeson">@ojipadeson</a>, is there any collective communication in <code>self.evaluate()</code>? All processes must launch the same collective communication in the same order. If you are not sure if  <code>self.evaluate()</code> launched any collective, you can try remove <code>val_stat = self.evaluate() if utils.is_main_process() else None</code> and see if running <code>dist.barrier()</code> alone passes?</p>
Is DDP faster than DP in inference time?

Giang_Nguyen: <p>Hi,</p>
<p>I am wondering is DDP faster than DP in inference time?<br>
DDP is normally faster than DP in training because of the way it handles the gradient, but how about <strong>inference</strong> where no gradient is needed?</p>

mrshenli: <p>Hey <a class="mention" href="/u/giang_nguyen">@Giang_Nguyen</a>, for inference, you don’t need DDP or DP, because their main feature is synchronizing gradients which only applies to training.</p>
<p>If you have to use DDP or DP for inference, then yes, DDP will be faster. This is because DP will broadcast model weights to all devices in the beginning of every iteration, while DDP maintains a separate model replica on each process.</p>
Intended usage of PyTorch buffers

david-macleod: <p>I can see two benefits of registering tensors as buffers:</p>
<ol>
<li>They get broadcast from rank 0 to other ranks in DPP (useful for keeping statistics in sync for BatchNorm etc)</li>
<li>They are part of the state_dict and so get cast to the same dtype and device as the model with <code>model.to()</code>
</li>
</ol>
<p>For me scenario 2 is perhaps most useful as there are often cases where I have some non-learnable constants that I want to ensure mirror the dtype/device of the model commands (to avoid additional copies at runtime when casting / changing device). However this sometimes clashes with scenario 1 with implicit synchronisation across ranks which is sometimes not appropriate.</p>
<p>This can be avoided with <code>broadcast_buffers=False</code> in DDP, but then I have to manually set collectives for the buffers I do want to sync.</p>
<p>Is there a more canonical approach to decoupling these behaviours, and getting the benefits of scenario 2 without falling victim to scenario 1?</p>

david-macleod: <p>Is the alternative here to use a Parameter instead of a buffer, but set requires_grad=False? Could there be other gotchas here to using a non-trainable parameter for a constant?</p>
Some stupid question about tensor operations in Pytorch

lsw980303: <p>Hi there, I ran into a problem during the recent project. In the “forward” part, the “for” I use slowed the program a lot which I cannot accept.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/8/48b9d34b053f174e57572726f01401b95e1f1b67.png" data-download-href="https://discuss.pytorch.org/uploads/default/48b9d34b053f174e57572726f01401b95e1f1b67" title="截屏2022-08-01 02.24.29"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/8/48b9d34b053f174e57572726f01401b95e1f1b67_2_690x301.png" alt="截屏2022-08-01 02.24.29" data-base62-sha1="anmvqr53pl15WoxuyhksmdBjt5B" width="690" height="301" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/8/48b9d34b053f174e57572726f01401b95e1f1b67_2_690x301.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/8/48b9d34b053f174e57572726f01401b95e1f1b67_2_1035x451.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/8/48b9d34b053f174e57572726f01401b95e1f1b67_2_1380x602.png 2x" data-dominant-color="2C2E38"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">截屏2022-08-01 02.24.29</span><span class="informations">2062×900 129 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>As the sample shows above, how could I convert the “for” loop into matrix operation in pytorch or einsum making the program run faster ?  Or how to think about problem like this ?</p>
<p>Sorry for my poor English.</p>
<p>Thanks a lot!</p>

ptrblck: <p>Could you post an executable code snippet with the missing definitions of e.g. <code>couplings_lookup</code> by wrapping it into three backticks ```, please?</p>

lsw980303: <pre><code class="lang-auto">couplings_lookup = torch.tensor(range(1,901)).reshape(6,6,5,5)


fccl = torch.tensor([1,2,3,4,5,0])
xx = torch.tensor([3,2,2,2,1,0])
coupling = []
for idx, pos in enumerate(fccl):
    ccoupling_mask = torch.zeros(couplings_lookup.shape)
    ccoupling_mask[fccl[:idx],pos,xx[:idx],:] = 1
    # ccoupling_mask[fccl[:idx],pos,:,:] = 1
    # ccoupling_mask[:,pos,xx[:idx],:] = 1
    ccoupling = torch.sum(couplings_lookup * ccoupling_mask, axis=(0,1,2))
    print(ccoupling)
    coupling.append(ccoupling)
</code></pre>
<p>Thanks for the reply!</p>

lsw980303: <p>Maybe give me some hits about how to use “scatter” to accomplish this? Many thanks.</p>
<p>Sorry for the bother .</p>
Two types of transformations

komandant: <p>lets say i have these transformations</p>
<pre><code class="lang-auto"> transform_all = transforms.Compose([transforms.ToTensor(),
                                     transforms.Resize((500,500))]) 
 minor_class_transform = transforms.Compose([transforms.RandomPerspective()])
</code></pre>
<p>.</p>
<p>I’m aware that  the transformations will be applied on the fly for every sample. I want that transform_all is applied to every single datapoint since its doing normalization etc. Then i want that minor_class_transform is applied for every datapoint which is in an minority class. When i apply minority class transformation will non-transformed datapoints also added to the sample?</p>
<p>This is how i apply transformations in my custom dataset class.</p>
<pre><code class="lang-auto">    def __getitem__(self, idx):
        x, y_1, y_2  = self.samples[idx]
        x = cv2.imread(x)
        x = self.transform_all(x)
        if y_2 == 0:
            x = self.minor_class_transform(x)
        return x, y_1, y_2
</code></pre>
<p>I want that transform_all is applied to every single datapoint and that not a single image is without transform_all. Then im using minor_class_transform to do data augmenting for minority class images, but i also want that when  minor_class_transform  is not applied then such images are in the sample.</p>

knitemblazor: <p>Answer according to my understanding</p>
<p>self.transform_all will be used for transformation of all images even if y2==0 but for y2==0 an additional transform will also take place ie the self.minor_class_transform</p>

komandant: <p>Thus we have common understanding how the code snippet works. However lets say i have a picture X which belongs to a minority class. Then obviously self.minor_class_transform will be applied to the X. Will the non-transformed X be in the same sample as transformed X, since i want to increase amount of datapoints in the minority class.</p>
Why change orders of weights_init can get different weights？

fzy139: <p>Hi,</p>
<p>I’m trying to reproduce the result after refactoring my model’s codes. I find that the when I change orders of module initialization, e.g. changing</p>
<pre><code class="lang-auto">module1.apply(weights_init)
module2.apply(weights_init)
</code></pre>
<p>to</p>
<pre><code class="lang-auto">module2.apply(weights_init)
module1.apply(weights_init)
</code></pre>
<p>, the weights of them change either. If I run the same code again, they won’t change.<br>
<code>weights_init</code> is a random initialization function like <code>nn.init.kaiming_normal_()</code>.</p>
<p>But I’ve already fixed random seed like</p>
<pre><code class="lang-auto">torch.manual_seed(seed)
np.random.seed(seed)
random.seed(seed)
torch.cuda.manual_seed_all(seed)
os.environ['PYTHONHASHSEED'] = str(seed)
</code></pre>
<p>Is that normal because of some underterminstic cuda ops?</p>
<h1>
<a name="update-1" class="anchor" href="#update-1"></a>Update:</h1>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import random
import numpy as np

def set_random_seed(random_seed):
    np.random.seed(random_seed)
    random.seed(random_seed)
    if torch.cuda.is_available():
        torch.cuda.manual_seed(random_seed)
        torch.cuda.manual_seed_all(random_seed)
    torch.manual_seed(random_seed)
    torch.backends.cudnn.deterministic = True
    torch.backends.cudnn.benchmark = False

set_random_seed(0)

class Model(nn.Module):
    def __init__(self) -&gt; None:
        super().__init__()
        self.lin1 = nn.Linear(1, 16)
        self.lin2 = nn.Linear(16, 16)
        # swap the above two lines will get different weights
        print(self.lin1.bias)

model = Model()
</code></pre>

Aayush_Garg: <p>I usually initialize in the see in the following manner to reproduce the results in PyTorch:</p>
<pre><code class="lang-python"># Set the random seed

def set_random_seed(random_seed):
    np.random.seed(random_seed)
    random.seed(random_seed)
    if torch.cuda.is_available():
        torch.cuda.manual_seed(random_seed)
        torch.cuda.manual_seed_all(random_seed)
    torch.manual_seed(random_seed)
    torch.backends.cudnn.deterministic = True
    torch.backends.cudnn.benchmark = False
</code></pre>
<p>Maybe try this?</p>

fzy139: <aside class="quote no-group" data-username="Aayush_Garg" data-post="2" data-topic="158019">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/aayush_garg/48/23363_2.png" class="avatar"> Aayush_Garg:</div>
<blockquote>
<pre><code class="lang-auto">def set_random_seed(random_seed):
    np.random.seed(random_seed)
    random.seed(random_seed)
    if torch.cuda.is_available():
        torch.cuda.manual_seed(random_seed)
        torch.cuda.manual_seed_all(random_seed)
    torch.manual_seed(random_seed)
    torch.backends.cudnn.deterministic = True
    torch.backends.cudnn.benchmark = False
</code></pre>
</blockquote>
</aside>
<p>Thanks for the fast reply. I’ve tried it but the weights still change.</p>
<p>I’ve added a minimal code to show this issue.</p>
TorchScript deployment in cpp with wrong results

fel1x: <p>Hello,</p>
<p>I’ve recently been trying to deploy a V-net which I’ve implemented in Python using PyTorch. I’ve tested the model in Python and it seems to work fine. Then I exported the model via torch.jit.script, which also worked after some adjustments of the used types and adding some additional typing annotation. I then imported the model for deployment in cpp, loaded the exact same inputs (I double checked), but the model produces some entirely different output. The one thing that I found curious was, that when I checked the weights using the named_parameters method, everything seemed to match, but when I called dump_to_str to debug the entire module, some entirely different weights were printed. I’ve also exported the module from cpp to python and checked it again, everything worked fine again. What I suspect is that somehow the weights of the submodules and the saved parameter weights are not correctly matched / loaded in the cpp backend. Did anyone else encounter this problem or knows how to manually set the weights?</p>
<p>Thank you.</p>
Virtual batches of SGD optimization?

tal123: <p>I have a rather small GPU that can compute ImageNet classification loss on up to 32 images at once, and I’d like to simulate batch sizes of 256 images.</p>
<p>Since I use SGD optimizer, the loss gradient on the whole batch is simply the sum of the image loss gradients.</p>
<p>So theoretically, if I had access to the gradient vector, then I could compute the loss gradient for 256 images by computing it for 8 batches of 32 images and adding the results to a single vector, and only finally call <code>optimizer.step()</code>.</p>
<p>Is there a way to do it in PyTorch?</p>

ptrblck: <p>Yes, you could use the approaches described <a href="https://discuss.pytorch.org/t/why-do-we-need-to-set-the-gradients-manually-to-zero-in-pytorch/4903/20">here</a>.</p>

tal123: <p>That’s great, thank you.</p>
Torch.jit.trace receives fixed size input to track operations. But How traced modules handle different batch size?

may1: <p>we can convert pytorch modules to torchscript with torch.jit.trace().<br>
and this trace function needs fixed size input to track operations of the module.</p>
<p>However, the last batch size can be different from the batch size if the number of samples is not multiple of the batch size.<br>
ex) 118 samples with 32 batch size =&gt; 32/ 32 / 32 / 22</p>
<p>Then how does this traced module handle different batch size?<br>
I thought it would raise an error but traced module could inference with different batch size input.</p>
<p>Also, when I use larger batch size, I could see that inference speed becomes slow.<br>
I am using CNN model and I usually use 201 samples.<br>
I calculated the model inference time using the below code and could see that 16 batch size (6 seconds average in my example) makes the inference speed faster than 128 batch size (8 seconds average). (1 batch size results in 15 seconds)</p>
<pre><code class="lang-auto">            torch.cuda.synchronize()
            start = time.perf_counter()
            with torch.no_grad():
                output = torch_model(sample_batch_tensor)
            torch.cuda.synchronize()
            stop = time.perf_counter()
            curr_time = stop - start
</code></pre>
<p>I am using cuda parallel computation so I thought larger batch size will accelerate the model inference speed. I didn’t include model or data loading time. Is this also related to Tracing module with fixed size?</p>
How do I access my Conv/Linear weights without touching other learnable parameters

Andrew_Legend: <p>(I am relatively new to PyTorch so sorry if it is too basic)<br>
I want to access my convolutional and linear weights for regularization. However I have some PReLU parameters in my model that If I apply my regularization there, it will affect negatively my results.</p>
<p>Here is a toy model example:</p>
<pre><code class="lang-auto">#The Model
class TestModel(nn.Module):
    def __init__(self):
        super(TestModel, self).__init__()
        self.input_conv= nn.Conv2d(3, 64, 5, padding=2)
        self.i_activation= nn.PReLU(64)
        
        self.downconv1= nn.Conv2d(64, 64, 3, padding=1)
        self.d_activation1= nn.PReLU(64)
    def forward(self, x):
        x= self.input_conv(x)
        x= self.i_activation(x)
        
        x= self.downconv1(x)
        x= self.d_activation1(x)
        return x

#Instance of the model
model=TestModel()
</code></pre>
<p>Itterating with model.parameters does not work since PReLU stands in the way. Is there a kind of loop/function that I can use so that I can go through this? Thanks in advance</p>

ptrblck: <p>You could use <code>model.named_parameters()</code> which will return the name as well as the parameter and then use a condition on the name. Alternatively, you could also directly access the desired weight parameter via <code>model.input_conv.weight</code>.</p>
Where can I find the implementation of the declaration from `model.h`?

tmaric: <p>The <code>torch::Module</code> is</p>
<blockquote>
<p>22 /// The base class for all modules in PyTorch.</p>
</blockquote>
<p>I found its declaration in <code>include/torch/csrc/api/include/torch/nn/module.h</code>, where can I find its definition/implementation?</p>

ptrblck: <p>Check the corresponding <code>.cpp</code> file <a href="https://github.com/pytorch/pytorch/blob/a54c9a419e1f025546b69b9c393b5f96dc26539f/torch/csrc/api/src/nn/module.cpp">here</a>.</p>
How to understand temporal, spatial and volumetric words

blackbirdbarber: <p>When you deal with tensor(s), I know spatial coordinates are X, Y, and Z if it is present. I also know the batch size is not temporal.</p>
<p>Are image channels volumetric or temporal coordinates because both has sense to me.</p>

ptrblck: <p>I would claim it depends more on the operations you are applying to the input than the input itself.<br>
E.g. in a conv layer the channel dimension could be seen as “volumetric” as each conv filter would use all input channels in each window position and would not “slide” through it.<br>
However, you could certainly reshape the input and pass the channels as the “temporal” dimension to an RNN. I don’t know if it makes sense for your use case, but nothing stops you from treating the data as you wish <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"></p>
Set channels_last to be default in PyTorch

blackbirdbarber: <p>Is it possible to set <code>memory_format=torch.channels_last</code> in the notebook code?</p>

ptrblck: <p>No, you cannot set the memory format globally (as it wouldn’t make sense to create all intermediates/internal tensors in this format) and would need to call it on the model and the input tensors.</p>

blackbirdbarber: <p>Yeah, thank you <a class="mention" href="/u/ptrblck">@ptrblck</a>.</p>
<ol start="5">
<li>
<p>Interesting decision, since I read recently (y-day) channel last format may have some faster execution on GPU and TensorFlow also uses this channel last format probable for speed but I don’t know if this is really a huge gain. Maybe just some 20ish percents.</p>
</li>
<li>
<p>But you know other imaging libraries such as PIL (and numpy) use channel first format so maybe this is the reason.</p>
</li>
</ol>

ptrblck: <p>The speedup depends on the actual model, the used precision, the device, and especially the used libraries.<br>
While <code>channels_last</code> should give you a speedup for CNNs in mixed-precision, the actual performance depends on the kernel availability and the heuristics. E.g. if you are using the latest cuDNN (with the beta v8 API) you should see better performance in <code>channels_last+amp</code> for a variety of models compared to older releases. However, some configs such as grouped convs could still run into performance cliffs, so it’s unfortunately still not a clear decision.</p>
<p>The preference of the memory layout changes from time to time, so it really comes down to which format is now the “best” one <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"></p>

blackbirdbarber: <p>Yeah, I was reading y-day the <a href="https://pytorch.org/blog/tensor-memory-format-matters/" rel="noopener nofollow ugc">Efficient PyTorch: Tensor Memory Format Matters</a> and the conclusion is:</p>
<blockquote>
<p>The Memory Layout of an input tensor can significantly impact a model’s running time. For Vision Models, prefer a <strong>Channels Last</strong> memory format to get the most out of your PyTorch models.</p>
</blockquote>
<p>But how you can define the Vision Models nowadays when Transformer is probable today also a vision model.<br>
<img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji only-emoji" alt=":wink:" loading="lazy" width="20" height="20"></p>

blackbirdbarber: <p>And now just one simple ask. If I work with input tensor in channel-last memory format do other tensors need to be channel-last as well?</p>
How to get the short name of a torch::Module

tmaric: <p>I would like to get the shorthand name of a torch::Model, in <code>include/torch/csrc/api/include/torch/nn/module.h</code>, I found</p>
<pre><code class="lang-auto">   87   /// Returns the name of the `Module`.
   88   ///
   89   /// A `Module` has an associated `name`, which is a string representation of
   90   /// the kind of concrete `Module` it represents, such as `"Linear"` for the
   91   /// `Linear` module. Under most circumstances, this name is automatically
   92   /// inferred via runtime type information (RTTI). In the unusual circumstance                                                                                        
   93   /// that you have this feature disabled, you may want to manually name your                                                                                          
   94   /// `Module`s by passing the string name to the `Module` base class'                                                                                                 
   95   /// constructor.                                                                                                                                                     
   96   const std::string&amp; name() const noexcept;
</code></pre>
<p>but when I use it for example for <code>Tanh</code>, I get two different string representations, one by <code>name()</code> another by the <code>ostream&amp; &lt;&lt;</code> ,</p>
<pre><code> auto tanh = torch::nn::Tanh();
std::cout &lt;&lt; tanh-&gt;name() &lt;&lt; std::endl;
std::cout &lt;&lt; tanh &lt;&lt; std::endl;  
</code></pre>
<p>namely,</p>
<pre><code class="lang-auto">torch::nn::TanhImpl
torch::nn::Tanh()
</code></pre>
<p>I would just like to have “Tanh”, is this readily available without resorting manipulating strings myself?</p>
Decouple the weights and the cod/logic in a torch package

Erotemic: <p>I opened a discussion last year: <a href="https://discuss.pytorch.org/t/packaging-pytorch-topology-first-and-checkpoints-later/129478" class="inline-onebox">Packaging pytorch topology first and checkpoints later</a> that didn’t get much traction. It was suggested that I repost it here.</p>
<p>The details are in the previous discussion, so I’ll keep this short. But the gist is I want to:</p>
<ul>
<li>Deploy my model as a torch package as normal</li>
<li>Update the weights in that file (perhaps by unzipping it, adding in a new file with the pickled weight dictionary, and then re-zipping it).</li>
</ul>
<p>After using torch.package for awhile I’m thinking this isn’t possible given the current design of it, but if that is true I’d like to understand why, and I’d also like to see if this is a feature for torch.package where there might be some interest in developing it.</p>
<p>As mentioned in my previous post, I wrote a torch packager (before torch.package existed) that did something similar to what I’m saying, so I know it’s possible.</p>
Get Sample ROIs from torchvision's Faster RCNN

sidnetopia: <p>Hi, I want to get the sample ROIs in <a href="https://github.com/pytorch/vision/blob/main/torchvision/models/detection/faster_rcnn.py" rel="noopener nofollow ugc">torchvision’s Faster RCNN</a> (Please see red blob in the attached image). How do you go about it?</p>
<p>Apologies for a novice question.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/4/04745699eb71c19176a6501e2cef43735895fac8.png" data-download-href="https://discuss.pytorch.org/uploads/default/04745699eb71c19176a6501e2cef43735895fac8" title="Screenshot 2022-07-30 234221"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/4/04745699eb71c19176a6501e2cef43735895fac8_2_690x365.png" alt="Screenshot 2022-07-30 234221" data-base62-sha1="Dpa8x4QwrlRqcgLxVI66dOz3MQ" width="690" height="365" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/4/04745699eb71c19176a6501e2cef43735895fac8_2_690x365.png, https://discuss.pytorch.org/uploads/default/original/3X/0/4/04745699eb71c19176a6501e2cef43735895fac8.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/0/4/04745699eb71c19176a6501e2cef43735895fac8.png 2x" data-dominant-color="DFDBDE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2022-07-30 234221</span><span class="informations">1002×531 288 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

sidnetopia: <p>The sample rois is controlled by the <code>box_batch_size_per_image</code> parameter</p>
Prunning yolov5 custom model (trained from scratch)

ariel.solomon: <p>Hi,<br>
Was trying to prune custom trained yolov5 model with torch.nn.utils.prune as shown in tutorial (<a href="https://pytorch.org/tutorials/intermediate/pruning_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Pruning Tutorial — PyTorch Tutorials 1.12.0+cu102 documentation</a>), using the API.<br>
Prunning goes with no errors yet the model raise error as None, meaning it does not exists after prunning. None prunned model runs correctly (.pt model)<br>
My questions:</p>
<ol>
<li>Is it possible to prune yolov5 custom model?</li>
<li>if 1 is positive, can any one indicate possible cause?<br>
Error:"  File “main.py”, line 65, in main<br>
inference = model(img)<br>
TypeError: ‘NoneType’ object is not callable"<br>
It was done in torch container environment (<a href="https://catalog.ngc.nvidia.com/orgs/nvidia/containers/pytorch/tags" class="inline-onebox" rel="noopener nofollow ugc">PyTorch | NVIDIA NGC</a> the 21.09-py3)</li>
</ol>
Autograd + memory freeing + garbage collection

srishti-git1110: <p>Continuing the discussion from <a href="https://discuss.pytorch.org/t/autograd-garbage-collection/157523">Autograd + garbage collection</a>:</p>
<p>I just realized this should be a part of the autograd cateogry and not a separate cateogry.<br>
Could anyone please answer their thoughts on it?</p>
RuntimeError: mat1 and mat2 shapes cannot be multiplied (2x786432 and 100x8192)

joel2: <p>Getting that error everytime i run my training model for the discriminator and generator am using 512*512</p>

ptrblck: <p>Answered <a href="https://discuss.pytorch.org/t/struggling-with-a-dcgan-while-trying-to-train-my-dataset/157933/4">here</a>.</p>
[torchvision's Faster RCNN implementation] Extend the task and update weights of backbone

sidnetopia: <p>Hi, I want to extend the torchvision’s Faster RCNN<br>
(<a href="https://github.com/pytorch/vision/blob/main/torchvision/models/detection/faster_rcnn.py" class="inline-onebox" rel="noopener nofollow ugc">vision/faster_rcnn.py at main · pytorch/vision · GitHub</a>). My plan is to have an additional classification task where features from the <code>feature extractor</code>  are used to determine the class of the image. Then the classification loss will be computed and backpropagated to the <code>feature extractor</code>. How do you go about it?</p>
<p>Apologies for a novice question.</p>

ptrblck: <p>I assume you are interested in using <a href="https://github.com/pytorch/vision/blob/3db30442a52e250c569ba131485fc432c8300f16/torchvision/models/detection/generalized_rcnn.py#L101">these features</a>. If so, you could try to return this <code>features</code> tensor additionally and use it directly in your new classification task or alternatively you could also try to store this tensor using a forward hook and pass it to the classificator after the forward pass was performed.</p>

sidnetopia: <p>thanks <a class="mention" href="/u/ptrblck">@ptrblck</a>  <img src="https://discuss.pytorch.org/images/emoji/apple/grin.png?v=12" title=":grin:" class="emoji" alt=":grin:" loading="lazy" width="20" height="20"><img src="https://discuss.pytorch.org/images/emoji/apple/grin.png?v=12" title=":grin:" class="emoji" alt=":grin:" loading="lazy" width="20" height="20"><img src="https://discuss.pytorch.org/images/emoji/apple/grin.png?v=12" title=":grin:" class="emoji" alt=":grin:" loading="lazy" width="20" height="20">!</p>
Categorical to Numerical labels

ahmedoumar: <p>Hey Everyone, I’m returning to pytorch since a while, and i’m tackling a multi class classification problem for categorical data.<br>
I have 18 categorical labels(names of countries) and I’m trying to convert them to one hot labels?<br>
I’m now Using the sklearn class (LabelEncoder) but this method is just converting categories to numbers, and i want to one hot encode them?<br>
Any method to do this in pytorch?<br>
Please note that i have Train and val in two separate csv files.</p>
<p>Thank you so much for your help</p>

ptrblck: <p>You could use <code>F.one_hot</code> to create one-hot encoded targets from target tensors containing class indices.<br>
Note however, that built-in loss functions for multi-class classification use cases (e.g. <code>nn.CrossEntropyLoss</code> and <code>nn.NLLLoss</code>) expect targets containing class indices and not one-hot encoded targets (in newer versions <code>nn.CrossEntropyLoss</code> would accept these and would treat them as probabilities).</p>
Is there a loss that is too small?

Max_Unhold: <p>Hello,<br>
I have a loss function, that has values around 5.0 at the beginning of the training and goes down to 0.0003 during training. Then I get the error</p>
<pre><code class="lang-auto">RuntimeError                              Traceback (most recent call last)
\lib\site-packages\torch\autograd\__init__.py:173, in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables, inputs)
    170 # The reason we repeat same the comment below is that
    171 # some Python versions print out the first line of a multi-line function
    172 # calls in the traceback and some print out the last line
--&gt; 173 Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
    174     tensors, grad_tensors_, retain_graph, create_graph, inputs,
    175     allow_unreachable=True, accumulate_grad=True)

RuntimeError: Function 'MseLossBackward0' returned nan values in its 0th output.
...
    170 if isinstance(error, str):
    171     error = AssertionError(error)
--&gt; 172 raise error

AssertionError: 
</code></pre>
<p>Neither the loss function nor the prediction or the label have nan values. That’s why I assume, the loss function is too small and creates nan values in the backward.</p>
<p>Is that true<br>
and when is the loss function to small?<br>
Should I multiply the loss by some factor if I want to train further?</p>

ptrblck: <p>There shouldn’t be a limit on the loss value for <code>nn.MSELoss</code> as it can be negative as well (it might not make sense, but it won’t error out or should not create NaN values for zero or negative values).<br>
Could you check the model output as well as the target in the iteration creating this error?</p>
How to set different learning rates for different part of the same tensor?

Qwwq: <p>Is it possible to set different learning rates for different part of a tensor?</p>

ptrblck: <p>That’s not possible as you would be hitting the same limitations explained in your <a href="https://discuss.pytorch.org/t/set-a-portion-of-a-tensor-to-have-requires-grad-false/157856/2">other topic</a>.</p>

Qwwq: <p>So, the only way to have a different learning rates is to manually scale the gradients on the backward pass. Right?</p>

ptrblck: <p>No, you could also use different (sub-)tensors and stack/concatenate them in the actual forward pass as explained e.g. <a href="https://discuss.pytorch.org/t/want-to-optimize-only-some-variables-but-getting-cant-optimize-a-non-leaf-tensor-error/139714/6">here</a>.</p>
How would one create a Conditional Variational Autoencoder with Convolutional layers?

PyTorch2603: <p>I was trying to find an example of a Conditional Variational Autoencoder that first uses convolutional layers and then fully connected layers, which would be necessary if dealing with larger images (I created a CVAE on the MNIST dataset which only used fully connected layers). Does anyone know of any CVAE which also uses convolutional layers before the fully connected layers that acts on larger images, and/or any tips as to how I should adapt my code? Thanks.<br>
Link to original code: <a href="https://github.com/StructuredProgramming/CVAE-using-MNIST-dataset/blob/main/cvae.py" class="inline-onebox" rel="noopener nofollow ugc">CVAE-using-MNIST-dataset/cvae.py at main · StructuredProgramming/CVAE-using-MNIST-dataset · GitHub</a></p>
Estimating the amount of VRAM that will be used by inference?

joeyballentine: <p>Hello. I have tried a few ways of estimating how much VRAM will be used by inference. Specifically different combinations of using the number of params of the model and number of elements of the image, but nothing really seems to work quite right.</p>
<p>I am wondering if there’s a known accurate way to perform this estimation.</p>
<p>For context, I am trying to prevent the users of my application from getting an out-of-memory error when using a model that is too large for their machine. I currently do this by essentially just try/catching the cuda out-of-memory error and tiling/chunking the image recursively, so it keeps tiling until it is able to infer (this is for super-resolution where the entire information of the image does not matter, so tiling is acceptable). However, for some users, this approach has caused issues, mainly where PyTorch sits right on the edge of usable VRAM and does not cause an out-of-memory error, but is instead just extremely slow. Plus, even though PyTorch causes an out-of-memory error fairly quickly, it still does take extra time to determine how much it needs to tile the image by just catching the errors.</p>
<p>So, I want to switch to an approach where I can just tell beforehand how much VRAM will be used by the model and image combination, and both show the user how much VRAM will be required as well as determine the correct amount for tiling beforehand.</p>
<p>If there is any way to do this accurately, please let me know. This is the formula I’m currently playing around with:</p>
<pre data-code-wrap="py"><code class="lang-nohighlight">img_bytes = img_tensor.numel() * img_tensor.element_size()
model_bytes = sum(
    p.numel() * p.element_size() for p in model.parameters()
)
mem_required_estimation = (model_bytes / (1024 * 52)) * img_bytes
</code></pre>
<p>However, it’s a bit arbitrary. The 52 in that formula was chosen just by seeing what number got me closest to accurately determining what would get me out of memory. Surely there must be a better and more accurate way of doing this?</p>
<p>Thank you in advance for your time.</p>