akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Neural network stuck in 4 out of 5 folds during cross-validation with scaled dataset

dahustle: <p>Im training a neural network of two parts, end-to-end. The first part consist in a Message passing Graph neural net that makes embeddings of the nodes, and the second part is a MLP that predicts similarity for a pair of nodes. Im training this by K-fold cross-validation with k=5, with random state fixed. The model was performing very well in a little dataset. When i scale the model with a 6x dataset, the model doesn’t train in 4 of the 5 folds.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/5/a5b5a568705a1d027cd4bf89b6e48b4b9c5ff167.png" data-download-href="https://discuss.pytorch.org/uploads/default/a5b5a568705a1d027cd4bf89b6e48b4b9c5ff167" title="scale1+selu+kaimingtb"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/5/a5b5a568705a1d027cd4bf89b6e48b4b9c5ff167_2_690x169.png" alt="scale1+selu+kaimingtb" data-base62-sha1="nDVXnde3zMODkEZJFfCPtu0VSuj" width="690" height="169" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/5/a5b5a568705a1d027cd4bf89b6e48b4b9c5ff167_2_690x169.png, https://discuss.pytorch.org/uploads/default/optimized/3X/a/5/a5b5a568705a1d027cd4bf89b6e48b4b9c5ff167_2_1035x253.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/a/5/a5b5a568705a1d027cd4bf89b6e48b4b9c5ff167.png 2x" data-dominant-color="3A3938"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">scale1+selu+kaimingtb</span><span class="informations">1295×319 84.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Here we can see the Training loss (left) and val loss (right) throughout the epochs. Every line is a fold. The differences of the magnitudes is because im training the model with 2 different loss functions. The thing here is that in all cases the first fold train properly an then the others don’t evolve at all.</p>
<p>Thing i try: Changin the activation from ReLU to SELU to avoid dying ReLU. Initialize all the weights with He (Kaiming). --&gt;Nothing changes.</p>
<p>Could you give me a clue where to attack the problem?</p>
Setting weights

curiousEngRes: <p>Hi!<br>
In PyTorch is there a way to specifically freeze only certain weights in a linear layer during the training process? Basically, I want to assign and maintain some weights in a linear layer to be zero. I am not able to figure out as to how to do this. Any help/input will be greatly appreciated.</p>
<p>Thanks!</p>

ptrblck: <p><a href="https://discuss.pytorch.org/t/how-do-i-freeze-the-specific-weights-in-a-layer/104722">This topic</a> might be helpful.</p>

curiousEngRes: <p>Thank you! I am going through it.<br>
I also had a question that is it possible to prune a network first using Pytorch pruning functionality and then train the pruned network?</p>
CosineAnnealingWarmRestarts improper setup

Mahran-xo: <p>using CosineAnnealingWarmRestarts in this setup causes improper behaviour , what could possibly be wrong with the training loop?</p>
<pre><code class="lang-auto">def train_fn(loader, model, optimizer, loss_fn, scaler):
    lr_scheduler = torch.optim.lr_scheduler.CosineAnnealingWarmRestarts(
        optimizer, T_0=1, T_mult=2, eta_min=5e-5,
    )
    loop = tqdm(loader)

    for batch_idx, (data, targets) in enumerate(loop):
        data = data.to(device=DEVICE)
        targets = targets.float().unsqueeze(1).to(device=DEVICE)

        # forward
        with torch.cuda.amp.autocast():
            predictions = model(data)
            loss = loss_fn(predictions, targets)

        # backward
        optimizer.zero_grad()
        scaler.scale(loss).backward()
        scaler.step(optimizer)
        lr_scheduler.step()
        scaler.update()

        # update tqdm loop
        loop.set_postfix(loss=loss.item())
</code></pre>
Unable to find a valid cuDNN algorithm

hedinghong: <p>File “/home/hong/.conda/envs/Ner2Ner/lib/python3.7/site-packages/torch/nn/modules/conv.py”, line 790, in forward<br>
output_padding, self.groups, self.dilation)<br>
RuntimeError: Unable to find a valid cuDNN algorithm to run convolution</p>
<p>i’m running this repo : <a href="https://github.com/TaoHuang2018/Neighbor2Neighbor" class="inline-onebox" rel="noopener nofollow ugc">GitHub - TaoHuang2018/Neighbor2Neighbor: Neighbor2Neighbor: Self-Supervised Denoising from Single Noisy Images</a><br>
on ubuntu22.04 pytorch1.5 with cuda: torch==1.5.0+cu101 torchvision==0.6.0+cu101<br>
but  i get the above error<br>
could any one give some advice to fix this?</p>

ptrblck: <p>Could you update to the latest stable or nightly PyTorch release and check if you would still run into the error, please?</p>
Dependency issue with old torch version

AriZe_BoLT: <p>Hi everyone i’m new to the community. We have been working on this github project which is based on NSGA-net, for model compression. (refer : <a href="https://github.com/mikelzc1990/nsganetv2" class="inline-onebox" rel="noopener nofollow ugc">GitHub - mikelzc1990/nsganetv2: [ECCV2020] NSGANetV2: Evolutionary Multi-Objective Surrogate-Assisted Neural Architecture Search</a>). I have tried to run the code following the instructions as given. But i seem to not able to get the correct versions. It uses an old torch version - PyTorch 1.5.1. These two libraries are also having problems: * [OnceForAll] 0.0.4,<br>
, [timm] - 0.1.30. I tried using the version 1.13.0  and cuda10.2 as given by some people to whom i have asked before but this still seems to not work. When i use another version there seems to be updates regarding the functions and libraries used by the project, it is either replaced or removed completely in the new torch versions. So it throws various errors.<br>
Could anyone please give some suggestions on how to work through this problem<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/8/5848c179762a615d480d15a3c640e4785808161c.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/5848c179762a615d480d15a3c640e4785808161c" title="torch_issue2"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/8/5848c179762a615d480d15a3c640e4785808161c_2_690x309.jpeg" alt="torch_issue2" data-base62-sha1="cAZTxOZXNfzwGciAHvX5wdeDi9u" width="690" height="309" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/8/5848c179762a615d480d15a3c640e4785808161c_2_690x309.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/5/8/5848c179762a615d480d15a3c640e4785808161c_2_1035x463.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/8/5848c179762a615d480d15a3c640e4785808161c.jpeg 2x" data-dominant-color="737272"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">torch_issue2</span><span class="informations">1080×484 46.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>The install instruction fails since <code>torch==1.5.1</code> wasn’t built with CUDA 11.7, which was released later.<br>
Could you share the errors you are seeing when updating to the latest stable or nightly PyTorch release?</p>
Pytorch equivalent to tf.signal.frame?

djams: <p>Does Pytorch have an equivalent implementation to Tensorflow’s <a href="https://www.tensorflow.org/api_docs/python/tf/signal/frame" class="inline-onebox" rel="noopener nofollow ugc">tf.signal.frame  |  TensorFlow Core v2.5.0</a> ? I’ve been searching everywhere and cannot seem to find anything, even in torchaudio. I know that librosa has an equivalent function, but it requires that the inputs be converted to numpy arrays first</p>

ptrblck: <p>Based on the description I guess that <a href="https://pytorch.org/docs/stable/tensors.html#torch.Tensor.unfold"><code>tensor.unfold</code></a> would perform the same operation (with an additional <code>F.pad</code> before if needed).</p>

djams: <p>You are correct, thank you! For completeness, this method appears to produce the same results as tf.signal.frame (albeit not tested extensively)</p>
<pre><code class="lang-auto">def frame(signal, frame_length, frame_step, pad_end=False, pad_value=0, axis=-1):
    """
    equivalent of tf.signal.frame
    """
    signal_length = signal.shape[axis]
    if pad_end:
        frames_overlap = frame_length - frame_step
        rest_samples = np.abs(signal_length - frames_overlap) % np.abs(frame_length - frames_overlap)
        pad_size = int(frame_length - rest_samples)
        if pad_size != 0:
            pad_axis = [0] * signal.ndim
            pad_axis[axis] = pad_size
            signal = F.pad(signal, pad_axis, "constant", pad_value)
    frames=signal.unfold(axis, frame_length, frame_step)
    return frames
</code></pre>
<p>Reference:<br>
<a href="https://superkogito.github.io/blog/SignalFraming.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://superkogito.github.io/blog/SignalFraming.html</a></p>

sadahry: <p>This one may be correct.</p>
<pre><code class="lang-auto">def frame(signal, frame_length, frame_step, pad_end=False, pad_value=0, axis=-1):
    """
    equivalent of tf.signal.frame
    """
    pad_size = 0
    signal_length = signal.shape[axis]
    if pad_end:
        frames_overlap = frame_length - frame_step
        rest_samples = np.abs(signal_length - frames_overlap) % np.abs(frame_step)
        if rest_samples != 0:
            pad_size = int(frame_length - rest_samples)
            pad_axis = [0] * signal.ndim
            pad_axis[axis] = pad_size
            signal = F.pad(signal, pad_axis, "constant", pad_value)
    frames=signal.unfold(axis, frame_length, frame_step)
    return frames
</code></pre>