akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Unable to resume job with 64 nodes, errors appeared during loading sharded optimizer state dict

B_Huang: <p>Hi,</p>
<p>I am following this test code to save and load optimizer states using load_sharded_optimizer_state_dict: <a href="https://github.com/pytorch/pytorch/blob/09b3517297d888714750217edf6c1f76a6340d85/test/distributed/checkpoint/test_fsdp_optim_state.py" class="inline-onebox" rel="noopener nofollow ugc">pytorch/test_fsdp_optim_state.py at 09b3517297d888714750217edf6c1f76a6340d85 · pytorch/pytorch · GitHub</a></p>
<p>The resuming went well on 1 node and 8 nodes jobs. But when I tested it on 64 nodes with large model size, two errors appeared on two nodes<br>
Here is first error</p>
<pre><code class="lang-auto">optim_state = dist_cp_op.load_sharded_optimizer_state_dict(
    File "/opt/conda/lib/python3.9/site-packages/torch/distributed/checkpoint/optimizer.py", line 282, in load_sharded_optimizer_state_dict
      state_dict[key] = _shard_tensor(
    File "/opt/conda/lib/python3.9/site-packages/torch/distributed/_shard/api.py", line 70, in _shard_tensor
      st = sharding_spec.shard(tensor, src_rank=src_rank, process_group=process_group)
    File "/opt/conda/lib/python3.9/site-packages/torch/distributed/_shard/sharding_spec/chunk_sharding_spec.py", line 181, in shard
      dist.scatter(
    File "/opt/conda/lib/python3.9/site-packages/torch/distributed/distributed_c10d.py", line 1451, in wrapper
      return func(*args, **kwargs)
    File "/opt/conda/lib/python3.9/site-packages/torch/distributed/distributed_c10d.py", line 2774, in scatter
      _check_tensor_list(scatter_list, "scatter_list")
    File "/opt/conda/lib/python3.9/site-packages/torch/distributed/distributed_c10d.py", line 599, in _check_tensor_list
      raise RuntimeError(
  RuntimeError: Invalid function argument. Expected parameter `scatter_list` to be of type List[torch.Tensor].
</code></pre>
<p>And second error</p>
<pre><code class="lang-auto">File "/src/shopqa/generative/workflows/seq2seq/train/snapshot.py", line 363, in load_model_optim_final
      optim_state = dist_cp_op.load_sharded_optimizer_state_dict(
    File "/opt/conda/lib/python3.9/site-packages/torch/distributed/checkpoint/optimizer.py", line 282, in load_sharded_optimizer_state_dict
      state_dict[key] = _shard_tensor(
    File "/opt/conda/lib/python3.9/site-packages/torch/distributed/_shard/api.py", line 70, in _shard_tensor
      st = sharding_spec.shard(tensor, src_rank=src_rank, process_group=process_group)
    File "/opt/conda/lib/python3.9/site-packages/torch/distributed/_shard/sharding_spec/chunk_sharding_spec.py", line 172, in shard
      assert local_tensor is not None
  AssertionError
</code></pre>
<p>My pytorch version: 2.0.1<br>
I am wondering is there anything missing when I used load_sharded_optimizer_state_dict in large-scale training?</p>
Absolute error stagnates in image-to-image regression task

jd91: <p>Hi all,</p>
<p>This is my first post on what has been an immensely helpful forum. Any suggestions are welcome.</p>
<p>I am currently working on a side project to help out a PhD student at my university. The task is image-to-image regression on simple, small (28x72) spectrograms. The goal is to train a network to be able to accurately predict corresponding output spectrograms from a given input. The task is supervised; I have access to roughly 90k input/output pairs. My current dataset uses around 18k of these.</p>
<p>I have designed several custom networks to accomplish this task. Fully convolutional autoencoders, vanilla fully-connected networks, etc. GANs/transformers are overkill, given the simplicity of the images.</p>
<p>I have observed the best results so far using a combination of cosine loss, SSIM loss, euclidean distance, and absolute error (implemented as L1 loss -reduction ‘sum’). All of my networks achieve “good” results, in that loss descends smoothly. Predicted spectra are visually almost identical to ground truths, however, the issue is with the magnitude of predicted values. All of the parameters defined by the shape of the spectra are excellent (thanks to cosine/SSIM similarity), but the parameters related to intensity/magnitude are way off (up to 10x the target value).</p>
<p>There are no errors in my code: I have learnt what I need on that front by simply trawling through the PyTorch discussions. Both Cosine and SSIM metrics reach an acceptable 96+% within a few epochs, and the rest of the training is a slow process of reducing <em>absolute error</em>. <strong>My question is about strategies for reducing this error for predicted values</strong>. Would weighting the euclidean distance/L1 dynamically throughout training potentially help? I have experimented with waiting until similarity reaches a threshold value before adding these extra terms into the loss function, also with varying degrees of weight decay.</p>
<p>For a sample in my test set, the best absolute error is around 0.25, ranging to &gt;4 for very difficult outliers <strong>for all experiments/networks</strong>, which is unacceptable, given that the target values are exceedingly small, ranging from 0 (the majority of the image) to a small quantity of intermediate values (0.001 to 0.01) with only one or two defined peaks ( reaching &lt;1.5 in extreme cases).</p>
<p>TL/DR: Image regression generates predictions that are visually identical to ground truths; what are some strategies/techniques for further reducing the magnitude of these predicted values?</p>
How to change the module from order dict to Module dict

Xsurgeon: <p>I want to know how to change the model modules from storing them in Orderdict to Moduledict.<br>
I want to know how to convert the model if it is already in Orderdict.</p>
<p>Thanks</p>

Xsurgeon: <p>I tried to do that</p>
<blockquote>
<p>for k,val in model.state_dict().items():<br>
torch.nn.ModuleDict(OrderedDict({key:val}))</p>
</blockquote>
<p>error → torch.FloatTensor is not a Module subclass</p>

ptrblck: <p>The <code>state_dict</code> will return the parameters and buffers, so you could try to use an <code>nn.ParameterDict</code>.<br>
<code>nn.ModuleDict</code> is for <code>nn.Module</code>s and works fine:</p>
<pre><code class="lang-python"># plain dict
d = {'a': nn.Linear(1, 1),
     'b': nn.Linear(2, 2)}
m = nn.ModuleDict(d)
print(m)

# OrderedDict
d = {'a': nn.Linear(3, 3),
     'b': nn.Linear(4, 4)}
o = OrderedDict(d)
print(o)
m = nn.ModuleDict(o)
print(m)
</code></pre>

Zihan_LI: <p>It works perfectly, thank u!</p>
Grad can be implicitly created only for scalar outputs in call to torch.autograd.backward

siddhsql: <p>Hello,</p>
<p>I am new to PyTorch and deep learning. I have <a href="https://gist.github.com/siddhsql/3e12570f2352b706ec147fb57431feb9" rel="noopener nofollow ugc">this</a> piece of code I have written to build a text classifier based on what I read in the book NLP with Transformers.</p>
<p>When I run this code (and <code>train</code> method is called which is the last line in the code snippet) I get this error:</p>
<pre><code class="lang-auto">Exception has occurred: RuntimeError       (note: full exception trace is shown but execution is paused at: _run_module_as_main)
grad can be implicitly created only for scalar outputs
  File "/llm/.env/lib/python3.10/site-packages/torch/autograd/__init__.py", line 88, in _make_grads
    raise RuntimeError("grad can be implicitly created only for scalar outputs")
  File "/llm/.env/lib/python3.10/site-packages/torch/autograd/__init__.py", line 193, in backward
    grad_tensors_ = _make_grads(tensors, grad_tensors_, is_grads_batched=False)
  File "/llm/.env/lib/python3.10/site-packages/torch/_tensor.py", line 487, in backward
    torch.autograd.backward(
  File "/llm/.env/lib/python3.10/site-packages/transformers/trainer.py", line 2753, in training_step
    loss.backward()
  File "/llm/.env/lib/python3.10/site-packages/transformers/trainer.py", line 1940, in _inner_training_loop
    tr_loss_step = self.training_step(model, inputs)
  File "/llm/.env/lib/python3.10/site-packages/transformers/trainer.py", line 1664, in train
    return inner_training_loop(
  File "/llm/transformers-ch03/classifier.py", line 106, in &lt;module&gt;
    trainer.train()
  File "/usr/local/Cellar/python@3.10/3.10.2/Frameworks/Python.framework/Versions/3.10/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/usr/local/Cellar/python@3.10/3.10.2/Frameworks/Python.framework/Versions/3.10/lib/python3.10/runpy.py", line 196, in _run_module_as_main (Current frame)
    return _run_code(code, main_globals, None,
RuntimeError: grad can be implicitly created only for scalar outputs
</code></pre>
<p>and I am not sure what is causing it or how to debug and fix it. Can anyone please help me here? I am just following the code samples in the book.</p>
<p>S.</p>

ptrblck: <p>Based on the error message it seems you are trying to call <code>backward</code> on a tensor containing multiple values so you would either have to reduce it or pass the gradient explicitly:</p>
<pre><code class="lang-python">x = torch.randn(1, 10, requires_grad=True)
lin = nn.Linear(10, 10)

out = lin(x)

# fails since out contains multiple values and expects a gradient input
out.backward()
# RuntimeError: grad can be implicitly created only for scalar outputs

# works
out.mean().backward()
</code></pre>

siddhsql: <p>thanks for the response. i fixed it.</p>
CNN model outputs NaN only for the final batch

eltonc: <p>Hi all,</p>
<p>I’ve been working on training a CNN using PyTorch and I’ve come across an interesting issue. When I was training my model, the model would output NaN but only for the final batch in the epoch (when the remaining samples does not match the batch size). Upon looking into my model, I realized it was the CNN sequential model that gave the final NaN value.</p>
<p>This happened when I was using a batch size of 64. When I ensured that the total training samples was divisible by 64, I did not receive a NaN value. Furthermore, when I changed the batch size to 32 and 128, the problem was also solved. After trying a few more batch sizes, I found that only batch sizes of 63 and 64 caused this problem.</p>
<p>I am using the following CNN model below:</p>
<pre><code class="lang-auto">self.CNN = nn.Sequential(

          nn.Conv2d(19, 32, kernel_size = 3, padding = 1),
          nn.ReLU(),
          nn.Conv2d(32, 64, kernel_size = 3, stride = 1, padding = 1),
          nn.ReLU(),
          nn.MaxPool2d(2,2),
      
          nn.Conv2d(64, 128, kernel_size = 3, stride = 1, padding = 1),
          nn.ReLU(),
          nn.Conv2d(128, 128, kernel_size = 3, stride = 1, padding = 1),
          nn.ReLU(),
          nn.MaxPool2d(2,2),
          nn.Flatten()
      )
</code></pre>
<p>So far, I’ve just decided to not use a batch size of 64, and my model is working fine. However, I am pretty curious as to why this issue is occurring. Wondering if you all have any ideas?</p>

ptrblck: <aside class="quote no-group" data-username="eltonc" data-post="1" data-topic="181243">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/eltonc/48/60979_2.png" class="avatar"> eltonc:</div>
<blockquote>
<p>After trying a few more batch sizes, I found that only batch sizes of 63 and 64 caused this problem.</p>
</blockquote>
</aside>
<p>Are you able to directly reproduce it using any random input?</p>
Why the unsqueeze and the expand

Helectus: <p>Can someone help me understand why are these are used in this example?<br>
2 tensors of shapes [32, 8, 72, 64].</p>
<pre><code class="lang-auto">B, H, L_K, E = K.shape
_, _, L_Q, _ = Q.shape

# calculate the sampled Q_K
K_expand = K.unsqueeze(-3).expand(B, H, L_Q, L_K, E)
index_sample = torch.randint(L_K, (L_Q, sample_k)) # real U = U_part(factor*ln(L_k))*L_q
K_sample = K_expand[:, :, torch.arange(L_Q).unsqueeze(1), index_sample, :]
Q_K_sample = torch.matmul(Q.unsqueeze(-2), K_sample.transpose(-2, -1)).squeeze()
# find the Top_k query with sparisty measurement
M = Q_K_sample.max(-1)[0] - torch.div(Q_K_sample.sum(-1), L_K)
</code></pre>
<p>The code is part of the implementation off this expression:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/2/729f1f1bcbb72b8ddb4b3b7e23a220885b967b77.png" alt="image" data-base62-sha1="glZlzmVfjg8ZRbkpQcxwcUTkkKP" width="518" height="103"></p>
Batch sampler for a multitask dataset

Yeo_Wei_jie: <p>I have a number of datasets which I have created as separate dataset classes and trying to perform multi-task training where each batch is sampled from each dataset inversely proportional to the size of the dataset to balanced out.<br>
But for some reason I am facing an error when doing so.<br>
i received a “TypeError: object of type ‘int’ has no len()” when accessing this particular code in</p>
<pre><code class="lang-auto">class BalancedSingleDatasetBatchSampler(torch.utils.data.Sampler):
    def __init__(self, datasets, batch_size,drop_last=False):
        self.datasets = datasets
        self.batch_size = batch_size
        total_dataset_len = sum(len(dataset) for dataset in datasets) &lt;------- error 
        self.dataset_probabilities = []
        for dataset in datasets:
            self.dataset_probabilities.append(len(dataset) / total_dataset_len)
        # self.dataset_probabilities = [len(dataset) / sum(len(dataset) for dataset in datasets) for dataset in datasets]
        self.drop_last =drop_last

    def __iter__(self):
        dataset_indices = list(range(len(self.datasets)))  # indices for each dataset
        while True:
            # Choose a dataset to draw a batch from, with probability proportional to the dataset's size
            dataset_index = np.random.choice(dataset_indices, p=self.dataset_probabilities)
            dataset = self.datasets[dataset_index]
            sample_indices = list(range(len(dataset)))  # indices for samples in this dataset
            np.random.shuffle(sample_indices)  # shuffle the indices if you want
            for i in range(0, len(dataset), self.batch_size):
                yield sample_indices[i:i+self.batch_size]

    def __len__(self):
        return sum(len(dataset) for dataset in self.datasets) // self.batch_size
</code></pre>
<p>given that the sampler is used in</p>
<pre><code class="lang-auto">train_datasets  = [TextDataset(self.tokenizer, self.hparams.data_dir, self.hparams.max_seq_length,task_name=task,split = 'train') for task in self.hparams.tasks]
multi_train_dataset = ConcatDataset(train_datasets)
weighted_sampler = BalancedSingleDatasetBatchSampler(train_datasets,self.hparams.train_batch_size)
dataloader = DataLoader(multi_train_dataset, 
                                    batch_sampler = weighted_sampler,
                                    collate_fn = TaskCollator(self.tokenizer),
                                    # batch_size = self.hparams.train_batch_size,
                                    num_workers = 8)
</code></pre>
<p>it does not make sense that the dataset is a int, I have checked the type outside of batchsampler and it is not an int, but strangely when checked inside the sampler code, it iteratively produces ints after the dataset class.</p>
Use .state_dict() and .parameters() after wrapping module in dict

Zihan_LI: <p>When defining my own model, I am doing things like</p>
<pre><code class="lang-auto">class MyNet(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc = {'fc1' : nn.Linear(10, 20)}
model = MyNet()
params = list(model.parameters())
print(len(params))
&gt;&gt; 0
state_dict = model.state_dict()
print(state_dict)
&gt;&gt; OrderedDict()
</code></pre>
<p>To be specific, I organize the internal sub modules with dict. I then find that several functionalities can not work properly with this design, including <code>.state_dict()</code>, <code>.parameters()</code> as shown above. I guess that after wrapping modules within dict, pytorch can no longer locate them.</p>
<p>I know one of ways to solve this is to overwrite those functions, where we unfold the dict and extract either <code>.parameters()</code> and <code>.state_dict()</code> from the sub modules. I wonder is there any neat thing that can be done to resolve this issue.</p>

ptrblck: <p>This is expected as PyTorch does not properly register plain Python <code>list</code>s and <code>dict</code>s.<br>
Use <code>nn.ModuleDict</code> and <code>nn.ModuleList</code> instead and the parameters will be registered and show up in <code>model.parameters()</code> and the <code>state_dict</code>.</p>
Trying to compile out of the box yolov5 into Tensorrt Model: What am I missing?

chrisuz: <p>I am using this code to create a Tensorrt model from a loaded yolov5 model. I use these lines:</p>
<pre><code class="lang-auto"># Model
model = torch.hub.load('ultralytics/yolov5', 'yolov5s', pretrained=True).eval()
inputs =[torch_tensorrt.Input([1, 3, 300, 300])]
enabled_precisions = [torch.half]

trt_model = torch_tensorrt.compile(model, inputs=inputs, enabled_precisions=enabled_precisions)
</code></pre>
<p>and get this error:</p>
<pre><code class="lang-auto">    raise NotSupportedError(ctx_range, _vararg_kwarg_err)
torch.jit.frontend.NotSupportedError: Compiled functions can't take variable number of arguments or use keyword-only arguments with defaults:
  File "/home/zeloada/.conda/envs/ze-tiling/lib/python3.10/warnings.py", line 477
    def __exit__(self, *exc_info):
                       ~~~~~~~~~ &lt;--- HERE
        if not self._entered:
            raise RuntimeError("Cannot exit %r without entering first" % self)
'__torch__.warnings.catch_warnings' is being compiled since it was called from 'SPPF.forward'
  File "/home/zeloada/.cache/torch/hub/ultralytics_yolov5_master/models/common.py", line 229
    def forward(self, x):
        x = self.cv1(x)
        with warnings.catch_warnings():
             ~~~~~~~~~~~~~~~~~~~~~~~ &lt;--- HERE
            warnings.simplefilter('ignore')  # suppress torch 1.9.0 max_pool2d() warning
            y1 = self.m(x)
</code></pre>
<p>new to tensorrt generally and not sure how to debug. any help would be greatly appreciated.</p>

ptrblck: <p>CC <a class="mention" href="/u/narendasan">@narendasan</a> in case you want to take a look at it.</p>