akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Make sure output of network is normalized to min=0 max=1

PlzHelp: <p>Hi i am implementing a Version of DCGAN that works on 1d spectral data. And i have run into a problem where as i need each feature map in the output of the network to be normalized in the range min=0 max=1. I have tried a number of different solutions to this problem, but what i really want to do is just normalize the output of the last layer(Tanh) to be in that range.</p>
<p>What i want is something like this:<br>
min_val = X.min(-1)[0].min(-1)[0]</p>
<p>max_val = X.max(-1)[0].max(-1)[0]</p>
<p>X = (X-min_val[:,None, None])/(max_val[:,None,None]-min_val[:,None,None])</p>
<p>Is it possible to make a function that does this to the output of Tanh, and simultaneously tracks gradients, and is this necessary/desirable?</p>
<p>to illustrate what i want:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/2/42196122e8f4b8c58e60a8ba3c67fb677deea9e5.png" data-download-href="https://discuss.pytorch.org/uploads/default/42196122e8f4b8c58e60a8ba3c67fb677deea9e5" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/2/42196122e8f4b8c58e60a8ba3c67fb677deea9e5.png" alt="image" data-base62-sha1="9qJT4q4QhMyXKeG1qH2PsOYIgx7" width="533" height="500" data-dominant-color="FBFBFB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">694×651 1.59 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

googlebot: <p>This may diverge, because optimal span before normalization is indeterminable. And without span invariance, you just need to replace Tanh with Sigmoid or Hardtanh.</p>
Is the following a correct way to implement linear warmup?

yxchng: <pre><code class="lang-auto">for epoch in range(args.start_epoch, args.epochs + args.warmup_epochs):
        if args.distributed:
            train_sampler.set_epoch(epoch)

        if args.warmup_epochs:
            lr = args.warmup_lr + (args.lr - args.warmup_lr) * (epoch / args.warmup_epochs)
            if epoch &lt;= args.warmup_epochs:
                for param_group in optimizer.param_groups:
                    param_group['lr'] = lr

        
        # train for one epoch
        train(train_loader, model, criterion, optimizer, epoch, scaler, args)

        # evaluate on validation set
        acc1 = validate(val_loader, model, criterion, args)

        # remember best acc@1 and save checkpoint
        is_best = acc1 &gt; best_acc1
        best_acc1 = max(acc1, best_acc1)

        if not args.multiprocessing_distributed or (args.multiprocessing_distributed
                and args.rank % ngpus_per_node == 0):
            save_checkpoint({
                'epoch': epoch - args.warmup_epochs,
                'arch': args.arch,
                'state_dict': model.state_dict(),
                'best_acc1': best_acc1,
                'optimizer' : optimizer.state_dict(),
            }, is_best)

        if epoch &gt;= args.warmup_epochs:
            scheduler.step()
</code></pre>
<p>Most of the implementations I found couple warmup with scheduler. I am trying to implement one that decouples them. Is the above implementation correct? Or is there a better way to implement?</p>

googlebot: <p>looks good, but perhaps you’d need to also save scheduler.state_dict() to correctly resume training (though scheduler construction with last_epoch=epoch should be enough for most schedulers, I think)</p>
Sigmoid vs softmax activation for medical image segmentation with dice loss?

c18: <p>For segmentation tasks with multiple classes, especially in the context of medical images where there might be class imbalance, is it preferable to use sigmoid or softmax as the final activation? I believe softmax would assign each particular pixel (voxel) to a single class whereas sigmoid could assign a single pixel (voxel) to multiple classes. Is this correct? Also, if we use dice loss as the loss function, are both softmax and sigmoid compatible or is one preferred over the other?</p>

ptrblck: <aside class="quote no-group" data-username="c18" data-post="1" data-topic="113101">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/a183cd/48.png" class="avatar"> c18:</div>
<blockquote>
<p>I believe softmax would assign each particular pixel (voxel) to a single class whereas sigmoid could assign a single pixel (voxel) to multiple classes. Is this correct?</p>
</blockquote>
</aside>
<p>Yes, that would be the case and the choice would depend on the use case you are working on (i.e. does your target contain a single class per pixel or multiple ones)?</p>
<aside class="quote no-group" data-username="c18" data-post="1" data-topic="113101">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/a183cd/48.png" class="avatar"> c18:</div>
<blockquote>
<p>Also, if we use dice loss as the loss function, are both softmax and sigmoid compatible or is one preferred over the other?</p>
</blockquote>
</aside>
<p>I believe <code>softmax</code> is used in the dice loss implementations I’m aware of and I don’t know, if a sigmoid activation would also work.</p>

fepegar: <p>I would say it is indeed possible to use sigmoid and Dice but, as <a class="mention" href="/u/ptrblck">@ptrblck</a> said, the choice of sigmoid vs softmax depends on whether the instances are allowed to belong to multiple classes.</p>
Fast Dropout Training

matheo_r: <p>Hi,</p>
<p>Does anyone know why fast dropout training (<a href="http://proceedings.mlr.press/v28/wang13a.pdf" rel="noopener nofollow ugc">http://proceedings.mlr.press/v28/wang13a.pdf</a>) is not implemented in PyTorch ?</p>
<p>Thanks,<br>
Matheo</p>