akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to broadcast in arbitrary dimensions?

tsp_t: <p>By default, broadcasting works in the last dimension, but I want to make broadcasting work in an arbitrary dimension.<br>
For example, in a matrix of 4 and 1 dimensions, the broadcast multiplication for the first dimension can be written as<br>
torch.einsum(“ijkl,i-&gt;ijkl”, X, Y)</p>
<p>But is there a suitable function that broadcasts in the n( 1&lt;=n&lt;=K )th dimension in a matrix of K and 1 dimensions?<br>
(The broadcast behavior I want is enough to behave like <code>view(1, 1, -1, 1, ..., 1)</code> )</p>

AlphaBetaGamma96: <p>Hi <a class="mention" href="/u/tsp_t">@tsp_t</a>,</p>
<p>If you’re trying to multiply across all dimensions you can just use <code>*</code>, if you get a mismatch on dimensions error, you can just unsqueeze your 2nd tensor so that it has the same shape as the 1st tensor. For example,</p>
<pre><code class="lang-auto">x1 = torch.randn(4,)
x2 = torch.randn(4, 5)

x1 * x2 #fails 
#RuntimeError: The size of tensor a (4) must match the size of tensor b (5) at non-singleton dimension 1
</code></pre>
<p>However,</p>
<pre><code class="lang-auto">x1[:, None] * x2  #works, returns tensor of shape [4,5]
</code></pre>
<p>In your case, you just need to unsqueeze to a tensor of size 4 and multiply, i.e. <code>x1[:, None, None, None] * x2</code>.</p>
Kernal died during permute operation

Suryalok1: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/f/3f49bc2c76889cf387a8dff1809ba01771afb0ae.png" data-download-href="https://discuss.pytorch.org/uploads/default/3f49bc2c76889cf387a8dff1809ba01771afb0ae" title="Screenshot 2022-10-29 at 1.25.09 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/f/3f49bc2c76889cf387a8dff1809ba01771afb0ae_2_690x448.png" alt="Screenshot 2022-10-29 at 1.25.09 PM" data-base62-sha1="91S3y74zqct9HHULeuIqfa5bpbE" width="690" height="448" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/f/3f49bc2c76889cf387a8dff1809ba01771afb0ae_2_690x448.png, https://discuss.pytorch.org/uploads/default/optimized/3X/3/f/3f49bc2c76889cf387a8dff1809ba01771afb0ae_2_1035x672.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/3/f/3f49bc2c76889cf387a8dff1809ba01771afb0ae_2_1380x896.png 2x" data-dominant-color="8C8D8D"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2022-10-29 at 1.25.09 PM</span><span class="informations">2940×1912 441 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Can weight_decay in AdamW be greater than zero?

Roman_Koshkin: <p>I just learned that it is possible to set <code>weight_decay</code> in <code>AdamW</code> to a value greater than 1. I thought that weight decay must be between 0 and 1, or am I missing something?</p>
<pre><code class="lang-auto">def make_optimizer(model, decoder_weight_decay):
    groups = []
    for n, p in model.named_parameters():
            if n.startswith('decoder.weight'):
                weight_decay = decoder_weight_decay
    
            group = {
                "params": p,
                "weight_decay": weight_decay,
                "name": n}
    groups.append(group)
    return AdamW(groups)
</code></pre>
Help with Deepspeed

promach: <p>Any Deepspeed user or expert out there ?</p>
<p>Could anyone help with <a href="https://github.com/microsoft/DeepSpeed/issues/2302#issuecomment-1279607938" class="inline-onebox" rel="noopener nofollow ugc">DeepSpeed still gives CUDA-out-of-memory issue · Issue #2302 · microsoft/DeepSpeed · GitHub</a> ?</p>
Torch not compiled with CUDA enabled Error

Lolo1: <p>I have the Following error raised<br>
None<br>
False<br>
1.13.0<br>
loading test data in memory: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5821/5821 [00:00&lt;00:00, 9485.06it/s]<br>
The output (*args, **kwargs)<br>
The second output (self, /, *args, **kwargs)<br>
Traceback (most recent call last):<br>
File “/home/lambda-quad2/t.py”, line 317, in <br>
main(args)<br>
File “/home/lambda-quad2/t.py”, line 171, in main<br>
torch.cuda.synchronize()<br>
File “/home/lambda-quad2/anaconda3/envs//lib/python3.9/site-packages/torch/cuda/<strong>init</strong>.py”, line 564, in synchronize<br>
_lazy_init()<br>
File “/home/lambda-quad2/anaconda3/envs/pedgraph2/lib/python3.9/site-packages/torch/cuda/<strong>init</strong>.py”, line 221, in _lazy_init<br>
raise AssertionError(“Torch not compiled with CUDA enabled”)<br>
AssertionError: Torch not compiled with CUDA enabled</p>
<p>I did install the pytorch version with cuda enable and I am unsure why I have this error raised I used the Following Link to install pytorch<br>
conda install pytorch torchvision torchaudio cudatoolkit=11.6 -c pytorch -c conda-forge</p>
<p>and when I run the nvidia-smi, I have GPU enabled</p>

ptrblck: <p>Based on the error message you’ve installed a CPU-only version. Check if you have multiple PyTorch installs in your current environment, uninstall them all, and reinstall the desired one.</p>
Can we continue training a model after we stop training it as if we did not stopped it?

jmlipman: <p>I originally assumed that, in order to continue the training where it stopped, we just needed to store the weights, the optimizer state, current epochs/iterations, and re-adjust the learning rate if there was any scheduling. However, the order in which the samples are drawn from the training set are going to be drawn in a different order, regardless of the seed you use.</p>
<p>This happens because when “iter(tr_loader)” is called, a new random seed is used, which I guess is an efficient way to shuffle the data after every epoch. Is there any good way to continue sampling the data in the way it was supposed to, besides running Tensor.random_ n times where n is the number of times you previously called “iter(tr_loader)” and “iter(val_loader)”?</p>
<p>You can read more about this is-not-a-bug-it’s-a-feature issue here: <a href="https://github.com/fepegar/torchio/issues/991" class="inline-onebox" rel="noopener nofollow ugc">Order of the elements in the queue changes unexpectedly · Issue #991 · fepegar/torchio · GitHub</a></p>
<p>Long story short, this:</p>
<pre><code class="lang-auto">_ = enumerate(val_loader)
_ = enumerate(val_loader)
for i, patch in enumerate(tr_loader):                                           
    print(patch["info"])
</code></pre>
<p>will draw the samples in a different order than if you do this</p>
<pre><code class="lang-auto">for i, patch in enumerate(tr_loader):                                           
    print(patch["info"])
</code></pre>
<p>because the Tensor.random_() used in the for loop is not the same anymore, as it was called before.</p>
Exploding Loss with GradScaler and frozen Batchnorm Layers

SBensberg: <p>Hi,<br>
I today noticed that when I freeze my batchnorm2d layers and using torch.cuda.amp.GradScaler my losses exploding after just 3 or 4 batches. The same code and parameters are giving very good results with not frozen bn layers. I have to scale down the learning rate to get a functioning training process again.</p>
<p>This is obviously not a bug report, I just cannot come up with the reason behind this? I am using a DeepLabV3 with 107 batchnorm layers. For freezing them I simply set them to eval. Maybe someone can help me understand why this is happening.</p>
<p>Thanks for reading!</p>

ptrblck: <p>Could you post a minimal, executable code snippet which shows the loss explosion in amp, please?</p>

SBensberg: <p>Sure!</p>
<pre><code class="lang-auto">         learning_rate = 0.01
         weight_decay = 0.0005
         momentum = 0.9
         optimizer = optim.SGD(
                    model.parameters(), learning_rate, momentum=momentum, 
                    weight_decay=weight_decay, nesterov=True)
         grad_scaler = torch.cuda.amp.GradScaler()
         freeze_bn = True

         with model.train() as model:
            if freeze_bn:
                model.freeze_bn()
            optimizer.zero_grad(set_to_none=False)

            for batch_idx, batch in enumerate(self.train_loaders):
                batch = self._batch_to_device(batch)

                outputs = model(batch)

                targets = batch['targets']
                preds = outputs['segmentation_logits']
                loss_seg = functional.cross_entropy(preds, targets, ignore_index=255)

                if self.grad_scaler is not None:
                    # https://pytorch.org/tutorials/recipes/recipes/amp_recipe.html
                    grad_scaler.scale(loss_seg).backward()
                    grad_scaler.step(self.state.optimizer)
                    grad_scaler.update()
                    optimizer.zero_grad(set_to_none=False)
</code></pre>
<p>Freezing bn:</p>
<pre><code class="lang-auto">            for m in self.modules():
                if isinstance(m, (BatchNorm2d, SynchronizedBatchNorm2d)):
                    m.eval()
                    m.weight.requires_grad = False # probably redundant
                    m.bias.requires_grad = False # probably redundant
</code></pre>

ptrblck: <p>Your code is unfortunately not executable as the majority of object definitions is missing.</p>

SBensberg: <p>Sorry for that. I added most definitions. Are model and loaders definition needed too? Is something else still missing?</p>
<p>Edit: My guess is the explosion is because of too high learning rate. But why is it too high with frozen bn but perfectly okay without them frozen?</p>

ptrblck: <p>Yes, the model is also needed as I won’t be able to execute the code otherwise.<br>
Your current code snippet doesn’t show an <code>autocast</code> usage at all, so I’m currently also unsure why you are scaling the gradients in the first place.</p>
How to call function after forward function of a PyTorch model

kabbas570: <p>I am running a code where they have another function after the forward function (load_from) in the PyTorch model, as below. How to call this function or is it not automatically called? THANK yOU</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/3/a3c6a782aac1a6cd273edd8a56f70e832cc5aebc.png" data-download-href="https://discuss.pytorch.org/uploads/default/a3c6a782aac1a6cd273edd8a56f70e832cc5aebc" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/3/a3c6a782aac1a6cd273edd8a56f70e832cc5aebc.png" alt="image" data-base62-sha1="nmPrrv9zErchYAjbNcvqiEiWrqQ" width="690" height="417" data-dominant-color="273038"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1002×607 37 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

srishti-git1110: <p>You can call the method on an instance of the class the method is in.</p>

kabbas570: <p>thanks for the suggestion; I did something like this, and it worked.</p>
In matplot, how to set the color in this case?

BobKim: <p>Hello!<br>
I would like to make animation usinig this class.<br>
The label list is [‘A’, ‘B’, ‘C’, ‘D’].<br>
For now, the line colors are applied by default.<br>
However the default value has poor visibility, so I want to change each color. (‘grey’, ‘black’, ‘blue’, ‘limegreen’)</p>
<p>How to adjust the code below?</p>
<hr>
<p>class TrajectoryAnimation3D(animation.FuncAnimation):</p>
<pre><code>def __init__(self, *paths, zpaths, labels=[], fig=None, ax=None, frames=None,
             interval=60, repeat_delay=5, blit=True, **kwargs):
    if fig is None:
        if ax is None:
            fig, ax = plt.subplots()
        else:
            fig = ax.get_figure()
    else:
        if ax is None:
            ax = fig.gca()
    self.fig = fig
    self.ax = ax
    self.paths = paths
    self.zpaths = zpaths
    if frames is None:
        frames = max(path.shape[1] for path in paths)
    self.lines = [ax.plot([], [], [], '-',label=labels, alpha=1.0, lw=2.0)[0]
                  for _, label in zip_longest(paths, labels)]
</code></pre>

AlphaBetaGamma96: <p>Hi <a class="mention" href="/u/bobkim">@BobKim</a>,</p>
<aside class="quote no-group" data-username="BobKim" data-post="1" data-topic="164618">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/bobkim/48/52081_2.png" class="avatar"> BobKim:</div>
<blockquote>
<pre><code class="lang-auto">self.lines = [ax.plot([], [], [], '-',label=labels, alpha=1.0, lw=2.0)[0]
                  for _, label in zip_longest(paths, labels)]
</code></pre>
</blockquote>
</aside>
<p>Could you not edit this line to zip over your list of <code>[‘grey’, ‘black’, ‘blue’, ‘limegreen’]</code> as well?</p>
<p>So, something like,</p>
<pre><code class="lang-auto">colors = [‘grey’, ‘black’, ‘blue’, ‘limegreen’]
self.lines = [ax.plot([], [], [], '-', color=color, label=labels, alpha=1.0, lw=2.0)[0]
                  for _, label, color in zip_longest(paths, labels, colors)]
</code></pre>
GA and PSO as an optimizer

Peterwisu: <p>Hi, I implementing both Genetic algorithms and Particle Swarm as  a optimizer for my network. I  just curious that since GA and PSO dont need gradient for searching for an optimum that mean the tensor of weight in model also should be the tensor that<code> requires_grad</code> is False right and do I still need to put <code>with torch.no_grad()</code> ??  and do is there anything i should be aware of when using non-gradient method in pytorch?Thank you.</p>
Fix random seed per validation sample but unfix it in training

manurare: <p>Hi I have a custom dataset class with the following structure. Both the the validation and training datasets are created using this class.</p>
<pre><code class="lang-auto">class CustomDataset(data_utils.Dataset):
    def __init__(self, seed_per_sample=False):
        self.image_files = []
        self.mask_files = []
        self.seed_per_sample = False if not seed_per_sample else True 
        ...

    def __getitem__(self, index):
        img = Image.open(self.image_files[index])
        if self.seed_per_sample:
            random.seed(index)
        else:
            random.seed()
        mask_files = random.sample(self.mask_files, 3)
        ...

train_dataset = CustomDataset(seed_per_sample=False)
val_dataset = CustomDataset(seed_per_sample=True)
</code></pre>
<p>I want the sampled masks in the validation to always be the same at each sample. However I want random masks during training. This is the only solution I could come with. However, this makes reproducibility impossible because I am always re-setting the seed on each call to <code>__getitem__</code>. How could I avoid this and randomize the sampling in training while keeping it the same in validation?</p>
How many num_samples in WeightedRandomSampler?

Micha: <p>Does it make sense to choose a num_samples larger than the actual sample size, in order to be sure that the weighted random sampling actually gets “enough” samples from each class?</p>
<p>Based on the example <a href="https://github.com/ptrblck/pytorch_misc/blob/master/weighted_sampling.py" rel="noopener nofollow ugc">here</a> the num_samples is set to the sample size. But I am trying:</p>
<pre><code class="lang-auto">        weights = [1.0 / x for x in class_counts]
        label_wts = [weights[x] for x in labels]
        label_wts = torch.from_numpy(np.array(label_wts))
        num_samples = int(
            2 * class_counts['minimum_class'] * len(class_counts))
</code></pre>
<p>Is that enlarged num_samples necessary?</p>
<p>Thanks, Micha</p>

ptrblck: <p>I don’t think it’s necessary since increasing the number of epochs should be the same effect, shouldn’t it?</p>

Micha: <p>Thanks, that certainly clarifies.</p>
How to compute grad w.r.t. an single element of tensor

phantom90: <p>Hi there,</p>
<p>Consider this scenario.</p>
<pre><code class="lang-auto">y = model(x)  # shape of y: B * C
gradd = torch.autograd.grad(y[3, 2], x)
</code></pre>
<p>What I want to do is to take the grad of y[3, 2], w.r.t. x. However when I write as above, it raises</p>
<pre><code class="lang-auto">RuntimeError
element 0 of tensors does not require grad and does not have a grad_fn
</code></pre>
<p>Any suggestions. Thank you so much!</p>

tom: <p>This likely has nothing to do with the single element indexing but that <code>model</code> breaks the graph somewhere (if x requires grad), to check, see if y.requires_grad is True.</p>
<p>P.S.: Note that the terminology in the title is slightly off relative to the code, as it attempts to compute the grad <em>of</em> the single element tensor <code>y[3, 2]</code> <em>w.r.t.</em> <code>x</code>.</p>

srishti-git1110: <p>Hi <a class="mention" href="/u/phantom90">@phantom90</a>,<br>
Please see the following :</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
class Model(nn.Module):
  def __init__(self):
    super().__init__()
    self.fc1 = nn.Linear(2, 4)

  def forward(self, x):
    return self.fc1(x)

net = Model()
input_x = torch.tensor([1.0, 2], requires_grad=True) # But for requires_grad=True, it will produce your error
out_y = net(input_x)
gradient = torch.autograd.grad(out_y[1], input_x)
print(gradient) # (tensor([0.3005, 0.3967]),)
</code></pre>
<p>This shows that you can calculate the gradient of a single element of a tensor (by indexing) wrt some other tensor.</p>
<p>Edit : Just noticed <a class="mention" href="/u/tom">@tom</a> has already suggested the possible reasons as I was writing a reply.</p>
<p>As they’ve suggested, make sure the graph isn’t breaking anywhere.</p>
Loading weights for specific layer from .pt file

BobKim: <p>I would like to monitor the change of weights in every epoch.</p>
<p>So i use torch.save.<br>
torch.save({‘epoch’: epoch,<br>
‘net_state_dict’: net.state_dict(),<br>
‘optimizer_state_dict’: optimizer.state_dict()},<br>
‘save_weights/weights.pt’)</p>
<p>Then, i train ResNet-34 for CIFAR-10 image classification.</p>
<p>If the above function is used, a large amount of tensors are recorded, making it difficult to measure.<br>
So, I want to monitor the weight of a specific layer or measure the mean of a specific tensor.</p>
<p>Any advice would be appreciated!</p>

ptrblck: <p>In case you want to print some stats of a specific parameter you could access it directly via <code>model.layer.weight</code> and compute the stats of it. Would this work or what exactly are you looking for?</p>

BobKim: <p>Thanks!! Yes that’s what i’m looking for!</p>
`nn.Module` init update module with loops

1chimaruGin: <pre><code class="lang-auto">...
for key, info in self.layers_info.items():
            self.convs[key] = ConvKXBN(info, no_create=no_create, **kwargs)
            self.convs[key].add_module('relu', nn.ReLU(inplace=True))
            self.block_list.append(self.convs[key])
...
</code></pre>
<p>I want -<br>
<code>print(model)</code></p>
<pre><code class="lang-auto">HardConvK1KXK1(
  (conv1): ConvKXBN(
    (conv1): Conv2d(96, 24, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
    (bn1): BatchNorm2d(24, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (relu): ReLU(inplace=True)
  )
)
</code></pre>
<p>But I’m getting -</p>
<pre><code class="lang-auto">HardConvK1KXK1()
</code></pre>
How can I replace the convolution of a parallel model with a custom convolution?

ydc: <p>I want to use custom convolutional layers, but I don’t want to modify the model definition file (such as resnet.py), instead I want to override the forward method of each convolutional layer for a given nn.Module.<br>
I think the apply function seems to be helpful for my purpose. My implementation is as follows：</p>
<blockquote>
<p>def channel_pruning(m):</p>
<pre><code>def new_forward(self, x):
    y = F.conv2d(x, self.weight, self.bias, self.stride, self.padding, self.dilation, self.groups)
    k = int(m.out_channels * m.rate)
    if k &gt; 0:
        s = F.adaptive_avg_pool2d(torch.abs(x), (1, 1)).view(x.size()[0], -1)
        g = F.relu(self.gate(s))
        i = (-g).topk(k, 1)[1]
        t = g.scatter(1, i, 0)
        t = t / torch.sum(t, dim=1).unsqueeze(1) * self.out_channels
        y = y * t.unsqueeze(2).unsqueeze(3)
    return y

name = m.__class__.__name__
if 'Conv' in name:
    m.rate = args.rate
    m.gate = nn.Linear(in_features=m.in_channels, out_features=m.out_channels, bias=True).to(device)
    nn.init.constant_(m.gate.bias, 1)
    nn.init.kaiming_normal_(m.gate.weight)
    m.forward = types.MethodType(new_forward, m)
</code></pre>
<p>model.apply(channel_pruning)</p>
</blockquote>
<p>This code runs fine under CPU and single GPU. Unfortunately, when I try to parallelize the model, the program reports an error “RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:1 and cuda:0! (when checking argument for argument weight in method wrapper__cudnn_convolution)”<br>
The reason for the error is obvious, the model and the input are assigned to different GPUs. What should I do to make this code support multi-GPU parallelism?</p>

ptrblck: <p>I guess the <code>to(device)</code> is causing the issues, since <code>device</code> seems to be a global variable.<br>
Try to use the <code>.device</code> attribute of a local parameter or buffer of the used module and create the newly initialized layer on this <code>device</code>.</p>

ydc: <p>I have fixed this bug by replacing DataParallel to DistributedDataParallel.</p>
LSTM many2many and one2one

NoWay2Guess: <p>Hello, I am new to LSTM. I am currently a bit confused about one2one and many2many. For my understanding, many2many consists of multiple one2one neurons, is this correct? Here is the project I am currently working on, would you please go through the code and check out whether this model is a one2one or many2many, I am so confused about these concepts. The model that I am currently using is a 2-layer LSTM, and I plan to replace this model by a seq2seq model, is  this a good choice and what is the main difference between a many2many and a seq2seq?</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://colab.research.google.com/drive/1NJ1l17ulJ9gGREIndst6FRWL9zwhtuRL?usp=sharing">
  <header class="source">
      <img src="https://colab.research.google.com/img/favicon.ico?vrz=colab-20221025-060104-RC00_483625937" class="site-icon" width="128" height="128">

      <a href="https://colab.research.google.com/drive/1NJ1l17ulJ9gGREIndst6FRWL9zwhtuRL?usp=sharing" target="_blank" rel="noopener nofollow ugc">colab.research.google.com</a>
  </header>

  <article class="onebox-body">
    <img src="https://colab.research.google.com/img/colab_favicon_256px.png" class="thumbnail onebox-avatar" width="260" height="260">

<h3><a href="https://colab.research.google.com/drive/1NJ1l17ulJ9gGREIndst6FRWL9zwhtuRL?usp=sharing" target="_blank" rel="noopener nofollow ugc">Google Colaboratory</a></h3>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
ImageFolder with Custom Data

manudwd: <p>I’m working with a custom data set in the format</p>
<pre><code class="lang-auto">folder
│     
│
└--train
    └--──class1
    |         │   file011
    |         │   file012
    |
    |
    └───--class2
          │   file021
          │   file022

└--val
    └--──class1
    |         │   file011
    |         │   file012
    |
    |
    └───--class2
          │   file021
          │   file022
</code></pre>
<p>When trying to load the dataset</p>
<pre><code class="lang-auto">data_dir = r'PATH_TO_DATA/train'

dataset = datasets.ImageFolder(data_dir, ...)

FileNotFoundError: Found no valid file for the classes Cat, Deer, Dog, Human. Supported extensions are: .jpg, .jpeg, .png, .ppm, .bmp, .pgm, .tif, .tiff, .webp
</code></pre>
<p>The only issue I found similar to this was <a href="https://stackoverflow.com/questions/68229246/pytorch-torchvision-datasets-imagefolder-filenotfounderror-found-no-valid-file" rel="noopener nofollow ugc">here</a>, however in their case there seemed to be a <code>.ipynb_checkpoints</code> file which was causing the issue. It doesn’t appear to be the case here.</p>
<p>I also checked for hidden files, and made sure the extensions are acceptable.</p>
<p>Edit:  An important piece of information that I didn’t realize was the issue seems to be the issue. I am hosting this data on a remote using Rclone, mounting my onedrive to access the data. When accessing the data directly, the dataset is read just fine. It seems to be an issues of ImageFolder accessing data via the remote access more than anything else.</p>

ptrblck: <p>Which data type are you using?<br>
Based on the error message no valid file extension can be used to read the files, so maybe you would need to use a custom <code>loader</code> in the <code>ImageFolder</code> dataset in case your extension differs.</p>

manudwd: <p>The dtype is <code>torch.FloatTensor</code>, I think I should remove this discussion because when using google drive to access the data, it works fine through Colab. Only when hosting the data through RClone, there seems to be some data corruption.</p>

ptrblck: <p>If you’ve stored tensors directly, the file extension should still fail as these types are supported:</p>
<pre><code class="lang-python">Supported extensions are: .jpg, .jpeg, .png, .ppm, .bmp, .pgm, .tif, .tiff, .webp
</code></pre>
<p>which are all image formats.</p>
Detect optimizer state variable names without step

lshamis_fb: <p>Hey folks,</p>
<p>I’m trying to find the state variables names an optimizer uses without stepping the optimizer.</p>
<p>For example, given an instance of <code>optim.SGD</code>, I’d like to know that it tracks <code>momentum_buffer</code>. Given an instance of <code>optim.Adam</code>, I’d like to know that it tracks <code>step</code>, <code>exp_avg</code>, and <code>exp_avg_sq</code>.</p>
<p>This is easy to get from <code>state_dict()['state']</code> once a step has been taken, but initially <code>state_dict()['state']</code> is empty.</p>
<p>Is there another way to get the state variables names?</p>
<p>Thanks!</p>
<p>Sample code:</p>
<pre><code class="lang-auto">import torch
from torch import nn
from torch import optim

class MyModel(nn.Module):
    def __init__(self) -&gt; None:
        super(MyModel, self).__init__()
        self.layer_0 = nn.Linear(2, 3)
        self.layer_1 = nn.Linear(3, 4)

    def forward(self, x):
        return self.layer_1(self.layer_0(x))


mdl = MyModel()
opt = optim.Adam(mdl.parameters())

print(opt.state_dict())
# {'param_groups': [...], 'state': {}}

nn.MSELoss()(mdl(torch.rand(2)), torch.rand(4)).backward()
opt.step()

print(opt.state_dict())
# {'param_groups': [...], 'state': {
#     0: {'step': ..., 'exp_avg': tensor([...]), 'exp_avg_sq': tensor([...])},
#     1: {'step': ..., 'exp_avg': tensor([...]), 'exp_avg_sq': tensor([...])},
#     2: {'step': ..., 'exp_avg': tensor([...]), 'exp_avg_sq': tensor([...])},
#     3: {'step': ..., 'exp_avg': tensor([...]), 'exp_avg_sq': tensor([...])}}}

opt = optim.SGD(mdl.parameters(), lr=0.01, momentum=0.9)

print(opt.state_dict())
# {'param_groups': [...], 'state': {}}

nn.MSELoss()(mdl(torch.rand(2)), torch.rand(4)).backward()
opt.step()

print(opt.state_dict())
# {'param_groups': [...], 'state': {
#     0: {'momentum_buffer': tensor([...])},
#     1: {'momentum_buffer': tensor([...])},
#     2: {'momentum_buffer': tensor([...])},
#     3: {'momentum_buffer': tensor([...])}}}
</code></pre>

ptrblck: <p>No, I don’t think that’s possible since the states are lazily initialized as seen <a href="https://github.com/pytorch/pytorch/blob/6cc4ae3d2d64b10d7104c4a0cc4083a644ef8e54/torch/optim/adam.py#L210-L223">here</a> in the <code>step()</code> function.</p>
Inference multiple models simultaneously

Philipp_Singer: <p>I am trying to find a simple way to run a forward on a batch on two models on two GPUs at the same time. That means I do not want to distribute the batch of the same model across devices, but two models across devices. I thought that simple python multiprocessing can work, but running into issues due to pickling of the models and memory-related things.</p>
<p>So I want to do:</p>
<pre><code class="lang-auto">model1 = model1.to("cuda:0")
model2 = model2.to("cuda:1")

# run simultaneously
model1(batch)
model2(batch)
</code></pre>
<p>Any idea how to do this elegantly? In best case in a notebook environment.</p>

rvarm1: <p>In the simplest version, you don’t need multiprocessing and can take advantage of the async nature of CUDA.</p>
<p>In particular, the following will run the CUDA kernels corresponding to each model forward pass in parallel, so only CPU-side launches will not be parallelized:</p>
<pre><code class="lang-auto">model1 = model1.to("cuda:0")
model2 = model2.to("cuda:1")

# run simultaneously
0_batch = batch.to(0)
1_batch = batch.to(1)
# CUDA kernels will run in parallel as they execute async, and GPU 0 kernels don't block GPU 1 kernels.
model1(0_batch)
model2(1_batch)
</code></pre>
Which Forum Template was used to build discuss.pytorch.org

Elias_Vansteenkiste: <p>Which template was used to build this PyTorch forum?<br>
I see it here and there around the web, e.g forums.fast.ai is using the same template.</p>

ptrblck: <p>The forum is based on <a href="https://en.wikipedia.org/wiki/Discourse_(software)"><code>Discourse</code></a>.</p>
How to split a tensor dimension in 2 with JIT?

lostmsu: <p>Basically I am trying to JIT this op:</p>
<pre data-code-wrap="python"><code class="lang-python">x.view(*x.shape[:-2], x.shape[-2] // 2, 2)
</code></pre>
<p>The idea it to split the last dimension of tensor in 2, and then apply a linear layer with shape <code>2 -&gt; 16</code>.</p>
<p>Problem is JIT complains about this because it can not compute <code>*x.shape[:-2]</code>:</p>
<blockquote>
<p>cannot statically infer the expected size of a list in this context</p>
</blockquote>
<p>Is this generally impossible? Perhaps there’s another way to compute the same thing?</p>
<p>In the end I am trying to inline a trained activation like this:</p>
<pre data-code-wrap="python"><code class="lang-python">class Activation2(nn.Module):
    def __init__(self, dtype=None):
        super().__init__()
        self.l1 = activation.activation[2]
        self.l2 = activation.activation[4]
        self.l3 = activation.activation[6]
        self.dtype = dtype

    def forward(self, x: Tensor):
        x = x.unsqueeze(-1)
        x = x.view(*x.size()[:-2], x.size(-2) // 2, 2)
        x = x.to(torch.float32)
        x = self.l1(x)
        x = geglu(x)
        x = self.l2(x)
        x = geglu(x)
        x = self.l3(x)
        x = x.to(self.dtype)
        return x.squeeze(-1)

module = Activation2(dtype)
scripted_module = torch.jit.script(module.eval())
</code></pre>
Model parallelism for pretrained model

wasabi: <p>I’m trying to follow <a href="https://pytorch.org/tutorials/intermediate/model_parallel_tutorial.html" rel="noopener nofollow ugc">https://pytorch.org/tutorials/intermediate/model_parallel_tutorial.html</a> for model parallelism, is there a way to make this process easy for pretrained models? For example, in this post, I can create a paralleled ModelParallelResNet50 from scratch. However, if I have a pretrained ResNet50, I cannot directly load model state into ModelParallelResNet50, since the latter has a different structure.</p>

AlphaBetaGamma96: <p>Hi <a class="mention" href="/u/wasabi">@wasabi</a>,</p>
<p>Perhaps using the <code>strict=False</code> argument might be a workaround for this error?</p>
'MinBackward1' returned nan values in its 0th output

sfr000: <p>After one batch, it is triggered <code>RuntimeError: Function 'MinBackward1' returned nan values in its 0th output.</code>. I checked the loss and what is in input to the function in the forward pass and everything is not nan.<br>
Here the forward function:</p>
<pre><code class="lang-auto">def forward(self, input): 
        input_vec = input.flatten(start_dim=1)

        t_vec = input_vec @ self.A

        input = (t_vec - t_vec.min()) * input_vec
        
        input = input.view(-1, self.n_ticks, self.n_classes)
        
        self.input = input / (input.sum(dim=-1, keepdim=True) + 1e-8)

        return self.input
</code></pre>
<p>I checked the loss itself and what used in <code>input = (t_vec - t_vec.min()) * input_vec</code> where there is this issue and <code>self.A</code>. Nothing is assuming nan values.</p>
<p>Can somebody help with this issue?<br>
Thanks,<br>
S</p>

soulitzer: <p>Do you have a short runnable snippet that can reproduce this?</p>

AlphaBetaGamma96: <p>Following on from what <a class="mention" href="/u/soulitzer">@soulitzer</a> has already said (share a minimal reproducible script!), the error is occurring when you’re computing the gradient as can be seen from <code>MinBackward1 in its 0th output</code> error message. So you won’t see the error in the forward pass, only its backward pass.</p>
GPU training error!

NoWay2Guess: <p>Hello everyone, I have this run-time error when I try to train the model on gpu. Can someone point out where I did wrong? Thanks!</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/e/fe516ec3dbf933f6990d28cc583ef022e270d0fe.png" data-download-href="https://discuss.pytorch.org/uploads/default/fe516ec3dbf933f6990d28cc583ef022e270d0fe" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/e/fe516ec3dbf933f6990d28cc583ef022e270d0fe_2_690x209.png" alt="image" data-base62-sha1="AhNKgnR2hJbgvEw2KuB026rsFfE" width="690" height="209" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/e/fe516ec3dbf933f6990d28cc583ef022e270d0fe_2_690x209.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/e/fe516ec3dbf933f6990d28cc583ef022e270d0fe_2_1035x313.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/f/e/fe516ec3dbf933f6990d28cc583ef022e270d0fe_2_1380x418.png 2x" data-dominant-color="3B3B3B"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">2333×707 74.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://colab.research.google.com/drive/1NJ1l17ulJ9gGREIndst6FRWL9zwhtuRL?usp=sharing">
  <header class="source">
      <img src="https://ssl.gstatic.com/colaboratory-static/common/ac0e6bae38aa2b5a52a4f6e10b43c43d/img/favicon.ico" class="site-icon" width="128" height="128">

      <a href="https://colab.research.google.com/drive/1NJ1l17ulJ9gGREIndst6FRWL9zwhtuRL?usp=sharing" target="_blank" rel="noopener nofollow ugc">colab.research.google.com</a>
  </header>

  <article class="onebox-body">
    <img width="260" height="260" src="https://colab.research.google.com/img/colab_favicon_256px.png" class="thumbnail onebox-avatar">

<h3><a href="https://colab.research.google.com/drive/1NJ1l17ulJ9gGREIndst6FRWL9zwhtuRL?usp=sharing" target="_blank" rel="noopener nofollow ugc">Google Colaboratory</a></h3>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


ptrblck: <p>Your notebook doesn’t show any errors so did you already solve the issue?</p>

NoWay2Guess: <p>I think the code is currently in cpu mode, can you switch to GPU mode? Thanks! <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
Out of tree OpenCL backend now easy to install

artyom-beilis: <p>I started working on <a href="https://github.com/artyom-beilis/pytorch_dlprim/tree/true_out_of_tree_support" rel="noopener nofollow ugc">OpenCL out of tree backend</a> for pytoch about 1/2 a year ago. As a backbone I used <a href="https://github.com/artyom-beilis/dlprimitives" rel="noopener nofollow ugc">dlprimitives</a> for computations - many efficient deep learning operations, including efficient GEMM, efficient convolution algorithms using integrated gemm with image access, winograd convolution for 3x3 and depthwise separable convolution.</p>
<p>It gives good performance: around 60% of CUDA/CUDNN performance for resnet18 for training and around 80% for inference. It tested on various nVidia, AMD and Intel embedded GPUs.</p>
<p>Now recently many improvements for out-of-tree backend support were added to pytorch nightly and now I don’t need to use custom pytorch version and just use pytorch that is installed using pip.</p>
<p>Using nightly pytorch build the <a href="https://github.com/artyom-beilis/pytorch_dlprim/blob/true_out_of_tree_support/README.md#build" rel="noopener nofollow ugc">installation procedure is very simple</a> you only need OpenCL API installed and optional (but recommended) SQLite library development files installed - for kernel caching.</p>
<p>Of course the project is still in early stages and there are likely many operators not implemented yet and probably some bugs running around <code>:-)</code></p>

artyom-beilis: <p>I also checked against torch 1.13 and it is compatible as well but instead of using device name like <code>ocl</code> you need to use <code>privateuseone</code> and it works.</p>
<p>In nightly version you can call</p>
<pre><code>torch.utils.rename_privateuse1_backend('ocl')
</code></pre>
<p>And use ‘ocl:0’ as more readable device name</p>
Torch.cat numerical error

conv8d: <p>On concatenating a <code>torch.long</code> tensor with a <code>torch.float</code> I observe some numerical errors.</p>
<pre><code class="lang-auto">a = torch.tensor([-123456789]) # dtype is torch.int64
b = torch.ones(1,) # dtype is torch.float32
c = torch.cat([a, b]) # dtype is torch.float32
print(c[0].item()) # results in -123456792
</code></pre>

ptrblck: <p>That’s expected due to the limited numerical precision.<br>
Take a look at <a href="https://en.wikipedia.org/wiki/Single-precision_floating-point_format">Single-precision floating-point format - Wikipedia</a> and the precision limits.<br>
Your value lies between <code>[-2**27, -2**26]</code> and thus rounds to multiples of <code>8</code> in <code>float32</code>.</p>

conv8d: <p>I see, thanks. Is there a way to show a warning in such cases?</p>

ptrblck: <p>Rounding is everywhere, so a general warning won’t be useful. I thus assume you are interested in a warning if an integer value is transformed into a floating point one and cannot be represented?<br>
If so, feel free to create a feature request on GitHub so that the code owners can discuss it.</p>
Peculiar Inference speed test

gpas: <p>Hello,</p>
<p>I had to run some inference speed tests since I was getting weird results. I observed that for batch size 1 the model’s inference was 4 times higher than with batch size of 10.<br>
Below is the code for the inference test.</p>
<pre><code class="lang-auto">pose = HRNet(32,17)
pose.load_state_dict(torch.load('pose_hrnet_w32_256x192.pth'))

# pose = TRTModule()
# pose.load_state_dict(torch.load('int8_hrnet.pth'))
pose.cuda().eval()
device = torch.device("cuda")
dummy_input = torch.randn(1, 3,256,192, dtype=torch.float).to(device)
repetitions=1000
total_time = 0
with torch.no_grad():
    for rep in range(repetitions):
        starter, ender = torch.cuda.Event(enable_timing=True),   torch.cuda.Event(enable_timing=True)
        starter.record()
        _ = pose(dummy_input)
        ender.record()
        torch.cuda.synchronize()
        curr_time = starter.elapsed_time(ender)/1000
        total_time += curr_time
Throughput =   repetitions/total_time
print('Final Throughput:',Throughput)
</code></pre>
<p>In this case the fps was 27, while in the case of batch 10 for 100 reps was 110.<br>
Keep in mind that the same test with a different method (VitPose) yields the same result regardless of the batch size.<br>
Is this something that is possible , i.e. higher batch size to lead to lower latency?</p>
Force single core per process

Joe_Harrison: <p>I’m trying to parallelise my experiments on a CPU cluster. Each experiment uses comparatively tiny symbolic expressions (max 16 operators). Yet all my CPU cores are used when I run a single experiment using SGD or Adam—resulting in no gains when parallelising. The BFGS and LM implementations from Scipy for example, only use a single CPU core. How can I force Pytorch to use only a single CPU core per process?</p>
Behaviour of torch.nn.Linear is different on GPU (cuda) and CPU

wdhd: <p>Hello!</p>
<p>I have noticed some strange behaviour with <code>torch.nn.Linear</code> layers when running on GPU (cuda) vs CPU when applied to tensors with zero elements but with non trivial shape. Here is a minimal example.</p>
<p>On CPU:</p>
<p><code>t = torch.zeros(10, 0)</code><br>
<code>lin = torch.nn.Linear(0, 13) # This is basically a bias</code></p>
<p><code>lin(t).shape = (10,13)</code> as expected.</p>
<p>On GPU:</p>
<p><code>t = torch.zeros(10, 0).to("cuda")</code><br>
<code>lin = torch.nn.Linear(0, 13).to("cuda") # This is basically a bias</code></p>
<p><code>lin(t).shape = (13)</code> and a warning is raised:</p>
<p><code>UserWarning: An output with one or more elements was resized since it had shape [n, p], which does not match the required output shape [p]. This behavior is deprecated, and in a future PyTorch release outputs will not be resized unless they have zero elements. You can explicitly reuse an out tensor t by resizing it, inplace, to zero elements with t.resize_(0). (Triggered internally at  ../aten/src/ATen/native/Resize.cpp:24.)</code></p>
<p>I think this might be happening because of cuda optimisations? But I would at least expect the behaviour to be consistent on GPU and CPU.</p>
<p>Any thoughts?</p>
<p>Thank you!</p>
How to reimplement the code without reuse function?

Moon_Lee: <p>Hello all, I have a code script as</p>
<pre><code class="lang-auto">class SPPBottleneck(nn.Module):
    """Spatial pyramid pooling layer used in YOLOv3-SPP"""

    def __init__(
        self, in_channels, out_channels, kernel_sizes=(5, 9, 13), activation="silu"
    ):
        super().__init__()
        hidden_channels = in_channels // 2
        self.conv1 = BaseConv(in_channels, hidden_channels, 1, stride=1, act=activation)
        self.m = nn.ModuleList(
            [
                nn.MaxPool2d(kernel_size=ks, stride=1, padding=ks // 2)
                for ks in kernel_sizes
            ]
        )
        conv2_channels = hidden_channels * (len(kernel_sizes) + 1)
        self.conv2 = BaseConv(conv2_channels, out_channels, 1, stride=1, act=activation)

    def forward(self, x):
        x = self.conv1(x)
        x = torch.cat([x] + [m(x) for m in self.m], dim=1)
        x = self.conv2(x)
        return 
</code></pre>
<p>I want to reimplement the block <code>x = torch.cat([x] + [m(x) for m in self.m], dim=1)</code> to ignore the warning [<a href="https://quic.github.io/aimet-pages/releases/1.17.0/api_docs/torch_model_guidelines.html#model-validator-utility" class="inline-onebox" rel="noopener nofollow ugc">PyTorch Model Guidelines — AI Model Efficiency Toolkit Documentation: ver tf-torch-cpu_1.17.0</a>](https://“Model with reused modules”). Could you please help me to do it?</p>
Loading the Trainable activation function

Siva: <p>Hi all,<br>
I am using an adaptive activation function to train a neural network defined in a class module. I defined the activation function in a (parent) class and called the parent class (activation function with trainable parameters) in a child class (where the neural layers are defined). I could optimize the activation function and reduce the loss function to the desired value. I save and load the model by using the state_dict method.</p>
<p>Activation function with trainable parameters:<br>
{Here, ‘n’ is a constant scaling factor, and ‘a’ is a trainable parameter}<br>
class My_SiLU(nn.Module)				# Swish Function<br>
def <strong>init</strong>(self, n, a):						<br>
super().<strong>init</strong>()<br>
self.n = n<br>
self.a = a<br>
def forward(self, x):<br>
output = nn.SiLU()<br>
return output(self.a<em>self.n</em>x)</p>
<p>Neural network:<br>
class NN(nn.Module):<br>
def <strong>init</strong>(self):<br>
super().<strong>init</strong>()<br>
self.main = nn.Sequential(<br>
nn.Linear(input, neurons),</p>
<pre><code>		        My_SiLU(n, a),
			nn.Linear(neurons,neurons),

			My_SiLU(n, a),
			nn.Linear(neurons,neurons),

			nn.Linear(neurons,1),
		)

	def forward(self, x):
		output = self.main(x)
		return output
</code></pre>
<p>model = NN()<br>
ac_func=My_SiLU(n, a)<br>
After some epochs and optimization, I would save the network by using model.state_dict() and the function by ac_func.state_dict().</p>
<p>In another program, when I try to load both state.dict() files, it throws an error that<br>
“in load_state_dict<br>
raise RuntimeError(‘Error(s) in loading state_dict for {}:\n\t{}’.format(<br>
RuntimeError: Error(s) in loading state_dict for NN:<br>
Unexpected key(s) in state_dict: “main.1.a”, “main.3.a”,”</p>
<p>Could anyone suggest me to fix this?<br>
Thanks in advance!</p>

ptrblck: <p>Your code is not formatted properly so quite hard to read.<br>
However, based on the error message it seems your model definition in the new script differs from the old one and the <code>state_dict</code> has additional trainable parameters called <code>main.1.a</code> and <code>main.3.a</code>, which seem to come from the <code>My_SiLU</code> modules in the <code>nn.Sequential</code> container.</p>
Pytorch-lightning: pre-fetch 1 batch disable

orazheng: <p>Hello,</p>
<p>I am facing a problem of saving current batch output as the input for next batch since the dataloader always load one batch ahead before current training gets output.</p>
<p><a href="https://pytorch-lightning.readthedocs.io/en/stable/guides/data.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch-lightning.readthedocs.io/en/stable/guides/data.html</a></p>
<p>“When iterable datasets are used, Lightning will pre-fetch 1 batch (in addition to the current batch) so it can detect when the training will stop and run validation if necessary.”</p>
<p>I wonder is there any way I can disable this pre-fetch? or is there any way I can save current output first and read the output as the input for the next batch?</p>
<p>Thanks</p>

ptrblck: <p>The <code>DataLoader</code> allows you to set the <code>prefetch_factor</code> and disable it if needed. I don’t know how Lightning handles it.</p>
StyleGan3. The program hangs on Setting up PyTorch plugin “filtered_lrelu_plugin”

ll2pakll: <p>Hello. I use StyleGan3 from this repository <a href="https://github.com/NVlabs/stylegan3" class="inline-onebox" rel="noopener nofollow ugc">GitHub - NVlabs/stylegan3: Official PyTorch implementation of StyleGAN3</a>. I had everything working fine, but at some point, I interrupted the program without letting it finish. And now I’m facing the fact that the program hangs on <code>Setting up PyTorch plugin "filtered_lrelu_plugin"... </code> and nothing else happens. This happens both when trying to generate new images with <code>gen_images.py</code> and when trying to open the visualizer. In this case, visualizer opens but hangs with Loading and then the name of the network file. First I reinstalled torch and all related packages, then I replaced all the stylegan3 files with the ones downloaded from the repository. This didn’t help. So I completely deleted the virtual environment and re-created it by downloading and installing all the required modules. But the behavior has not changed. I have no idea what to do.<br>
Windows 10</p>

ptrblck: <p>Already tracked <a href="https://github.com/NVlabs/stylegan3/issues/203">here</a>.</p>