akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
CrossValidation

Chamroukhi: <p>Hello,<br>
i want to use cross validation for reduce overfitting, can you help me with tutoriels or example of code ? Thanks</p>

ptrblck: <p><a href="https://github.com/skorch-dev/skorch">skorch</a> is a scikit-learn compatible PyTorch library, which would provide the interface to apply the cross-validation methods from scikit-learn.<br>
Besides that I think other users have already posted some approaches in this discussion board, e.g. <a class="mention" href="/u/alejandrodebus">@alejandrodebus</a> in <a href="https://discuss.pytorch.org/t/what-is-the-best-way-to-apply-k-fold-cross-validation-in-cnn/15035/8">this post</a>.</p>
Array of neural net objects

KeL21: <p>Is it possible to create an array of neural network objects? I’m sorry if this has been posted somewhere, but I can’t seem to find it.</p>
<p>Here’s how I define my nn.Module:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/7/c73911ab8ddc315dae2ba2d3f1577bc9dcc9b890.png" data-download-href="https://discuss.pytorch.org/uploads/default/c73911ab8ddc315dae2ba2d3f1577bc9dcc9b890" title="Screen Shot 2021-06-08 at 11.15.28 AM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/7/c73911ab8ddc315dae2ba2d3f1577bc9dcc9b890_2_690x456.png" alt="Screen Shot 2021-06-08 at 11.15.28 AM" data-base62-sha1="sqphL4vc4LkJrYmQEEh2PG4SxB6" width="690" height="456" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/7/c73911ab8ddc315dae2ba2d3f1577bc9dcc9b890_2_690x456.png, https://discuss.pytorch.org/uploads/default/original/3X/c/7/c73911ab8ddc315dae2ba2d3f1577bc9dcc9b890.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/c/7/c73911ab8ddc315dae2ba2d3f1577bc9dcc9b890.png 2x" data-dominant-color="EEEEEF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2021-06-08 at 11.15.28 AM</span><span class="informations">856×566 35.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>If I want to now define multiple instances of this network, e.g.<br>
net[0]=Net1[10]<br>
net[1]=Net1[10]<br>
I get the error, ’ ‘Net’ object does not support item assignment’</p>

Sandipan_Majhi: <p>Net is a class. Probably it should be net[0] = Net1(10). Check that it should be first brackets. Net1[10] would automatically cause it to assume that Net1 is a dict or a list or tuple etc.</p>

ptrblck: <p>Additionally to what <a class="mention" href="/u/sandipan_majhi">@Sandipan_Majhi</a> said: in case you would like to wrap these models in a list, which would properly register these models, use e.g. <code>nn.ModuleList</code> instead of a Python <code>list</code> (same for <code>nn.ModuleDict</code> vs. <code>dict</code>).</p>
Object tracking

Vadimcg: <p>Could i find functional for object tracking in Pytorch ? Or i need use something else additional ?</p>

ptrblck: <p>It should be possible and you can find a few blog posts sharing their ideas such as <a href="https://towardsdatascience.com/object-detection-and-tracking-in-pytorch-b3cf1a696a98">this one</a>.</p>
Best way to directly modify network weights with gradient

pytorch_user: <p>Hello PyTorch Community,</p>
<p>I am trying to efficiently implement the following, but I am finding it difficult. I would greatly appreciate your help.</p>
<p>I am simply trying to perturb the weights of a neural network, perform a forward pass, and then calculate the gradients with respect to the original weights. In math, I have $f_{w}$ (a standard <code>nn.Module</code> with parameters $w$) and I would like to calculate the gradient with respect to $w$ of $f_{w + e}(x)$, where $e$ is some fixed vector (that does not require gradient) and $x$ is some fixed data.</p>
<p>I feel like this should be very easy to do efficiently, but I am finding it surprisingly difficult. I may be overlooking a simple solution, so I am asking here.</p>
<p>Thank you very much for your assistance.</p>

pytorch_user: <p>Hello, has anyone tried to do this before? It seems like it should be somewhat common.</p>

soulitzer: <p>Sounds like you want to register a parametrization -<br>
<a href="https://pytorch.org/docs/master/generated/torch.nn.utils.parametrize.register_parametrization.html?highlight=parametrization#torch.nn.utils.parametrize.register_parametrization" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/master/generated/torch.nn.utils.parametrize.register_parametrization.html?highlight=parametrization#torch.nn.utils.parametrize.register_parametrization</a></p>
<p>This requires PyTorch version &gt;= 1.9 though.</p>

pytorch_user: <p>Hi soulitzer, thanks for your reply.</p>
<p>This looks absolutely amazing! I have not been this excited for a while. I will try it out and report back here with my experience.</p>
Is there an encode_png method for cuda tensors?

Pablo1: <p>Hello,</p>
<p>Is there a way to get the size of a cuda tensor as if it were encoded as PNG?</p>
<p>I’ve tried this way:</p>
<pre><code class="lang-auto">inv_trans = transforms.Compose([ transforms.Normalize(mean = [ 0., 0., 0. ],
                                                     std = [ 1/0.229, 1/0.224, 1/0.225 ]),
                                transforms.Normalize(mean = [ -0.485, -0.456, -0.406 ],
                                                     std = [ 1., 1., 1. ]),
                                lambda x: x*255
                               ])
def get_size_as_png(input_tensor):
  k = inv_trans(input_tensor)
  a = torchvision.io.encode_png(k.type(torch.uint8)) 
  return sys.getsizeof(a.storage())
</code></pre>
<p>But when I have tensors in cuda I get the following error:</p>
<pre><code class="lang-auto">RuntimeError: Input tensor should be on CPU
</code></pre>
<p>Many thanks in advance</p>

eqy: <p>You can call <code>cpu_tensor = mytensor.cpu()</code> to get a copy of the data in host memory.</p>

Pablo1: <p>Thanks for your response,</p>
<p>I’m looking for the size of the tensor as if it were a PNG image, not the size of the tensor</p>
Kaldi Voice Activity Detection (VAD)

Kla: <p>Hi everyone, is it possible to use Kaldi Voice Activity Detection (VAD) in Pytorch?</p>

ptrblck: <p>I’m not deeply familiar with Kaldi, but how would you like to use it and what have you tried so far?<br>
Are you stuck at a specific point?</p>

Kla: <p>I’m actually working on my thesis with audio data and I want to filter out from every audio non speech frames, I mean sequences were people do not speak. I have read that kaldi VAD works good in this case. Or is there any other option in torchaudio how to do it?</p>

ptrblck: <p>Unfortunately, I don’t know how Kaldi detects the speech and if it’s a filtering algorithm or some kind of machine learning model. If Kaldi would work, you could stick to it and preprocess the data in this way. I’m unsure, if you would like to reimplement Kaldi’s algorithm in <code>torchaudio</code> or how they should be combined.</p>

vincentqb: <p>torchaudio has an implementation of VAD based on sox, see <a href="https://pytorch.org/audio/stable/functional.html#vad" rel="noopener nofollow ugc">here</a>, and another implemented as an example <a href="https://github.com/pytorch/audio/blob/master/examples/interactive_asr/vad.py#L38" rel="noopener nofollow ugc">here</a>. Let us know how your experience goes <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

Kla: <p>I have implemented it this way:</p>
<p>waveform, sample_rate = torchaudio.load(file_path)<br>
waveform = torchaudio.functional.vad(waveform, sample_rate)</p>
<p>and it seems to work but befor VAD it took only 10 - 15 Minutes to train an epoch, and now it needs almost 10 hours per epoch. Have I done something wrong?</p>

Alexuan: <p>Hi!<br>
This phenomenon might be reasonable when the VAD takes too much time.<br>
It might be feasible to exert VAD on all samples before your training instead of having VAD in DataLoader.</p>

vincentqb: <p><a class="mention" href="/u/kla">@Kla</a> – have you tried <code>apply_effects_tensor</code> with VAD effect? see <a href="https://pytorch.org/audio/stable/sox_effects.html#torchaudio.sox_effects.apply_effects_tensor" rel="noopener nofollow ugc">here</a>.</p>
How to tell c++ optimizer step to not allow tensor value to be below zero

Omroth: <p>Hi all - I have an auto grad tensor and I do not with negative values to be allowed - how can I get SGD to not send it below zero?</p>
<p>Thanks.</p>

soulitzer: <p>If you want to restrict a parameter to hold only positive values:<br>
<a href="https://pytorch.org/tutorials/intermediate/parametrizations.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/intermediate/parametrizations.html</a></p>
<p>It might look something like:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.nn.utils.parametrize as parametrize

class Positive(nn.Module):
    def forward(self, X):
        return torch.clamp_min(X, 0)

layer = nn.Linear(3, 3)
parametrize.register_parametrization(layer, "weight", Positive())
</code></pre>
<p>This is only available in versions &gt;= 1.9.0 though.</p>
Loss.backward with some intermediate layer frozen

manhon: <p>Hi,</p>
<p>I would like to ask if I could do this.</p>
<p>input → layer1,2,3 → layer4.5.6 (CNN and RNN) → predicted output.<br>
then i have a loss function with L1 loss on predicted output and target output.</p>
<p>i would like to freeze layer4,5.6 weight, such that the loss will update the parameters in layer1,2,3 only.</p>
<p>I tried by setting:<br>
layer4,5,6.eval()<br>
then call forwards of layer4,5,6.</p>
<p>but got the error when call loss backward:<br>
RuntimeError: cudnn RNN backward can only be called in training mode</p>
<p>many thanks for advice.</p>

soulitzer: <p><code>.eval()</code> does not freeze your layers. It puts your layers in “evaluation mode” (in contrast to training mode) which should only be done if you are testing/validating your model. To freeze layers while training, you should set requires_grad of layers 4,5,6 to false.</p>
<blockquote>
<p>To freeze parts of your model, simply apply <code>.requires_grad_(False)</code> to the parameters that you don’t want updated. And as described above, since computations that use these parameters as inputs would not be recorded in the forward pass, they won’t have their <code>.grad</code> fields updated in the backward pass because they won’t be part of the backward graph in the first place, as desired.</p>
</blockquote>
<blockquote>
<p>Because this is such a common pattern, <code>requires_grad</code> can also be set at the module level with <code>nn.Module.requires_grad_()</code>. When applied to a module, <code>.requires_grad_()</code> takes effect on all of the module’s parameters (which have <code>requires_grad=True</code> by default).</p>
</blockquote>
<p>See <a href="https://pytorch.org/docs/1.9.0/notes/autograd.html#locally-disabling-gradient-computation" class="inline-onebox" rel="noopener nofollow ugc">Autograd mechanics — PyTorch 1.9.0 documentation</a></p>
Why does some operators need to be float 32

amsword: <p>In mixed precision calculation, some operators can be fp16, while some can not. The question is that why those operator can not be fp16, e.g. layer norm. One rule could be that, do the model training by turning it to fp16. If the performance is not as good, then it should be fp32. But this  test could be very expensive. Is there a way to figure out whether an operator should be fp32 or fp16 by some rules?</p>
Dataloader returns data whenever any data is ready

amsword: <p>The order of the data is determined by the sampler. Is there a way to make the order determined by when the data is ready? That is, each worker processes the data independently, and the data loader returns the data which are ready rather than waiting for some data (indexed by the sampler)</p>
<p>The reason: in my application, I find different data needs different amount of computational time. The training becomes slow (2~4 times slower) lagged by the data which requires more time.</p>

eqy: <p>Can you work around this by more aggressively prefetching? (e.g., increase <code>prefetch_factor</code> when creating your dataloader)</p>

amsword: <p>Thanks for your reply. Before, i had tried to increase this and encountered insufficient shared memory issues.</p>

eqy: <p>In many cases I find it can be easier to improve the data loading process rather than to change dataloader behavior itself; changing the order of samples based on loading time might also introduce some unwanted bias in the training. Can you share some more details about the data loading computation so that potential bottlenecks can be mitigated?</p>

amsword: <p>thanks for the reply. Eventually, i find some images are too large and the problem is solved by resizing the image and resaving them to a lower-quality version. Perhaps, the disk I/O is one issue or needs to be improved.</p>
UNET Multiclass Segmentation from Binary Segmentation

hanoody: <p>Hi,</p>
<p>Any tips on how I would modify the following UNET binary segmentation architecture for multi-class segmentation with 6 classes? I am new to PyTorch and I am lost with regards to modifying this network for a multi-class usage.</p>
<p>Mainly, I do not understand how the two architectures would differ.</p>
<p>Thank you!</p>
<pre><code class="lang-auto">class DoubleConv(nn.Module):
    def __init__(self, in_channels, out_channels):
        super(DoubleConv, self).__init__()
        self.conv = nn.Sequential(
            nn.Conv2d(in_channels, out_channels, 3, 1, 1, bias=False), 
            nn.BatchNorm2d(out_channels),
            nn.ReLU(inplace=True),
            nn.Conv2d(out_channels, out_channels, 3, 1, 1, bias=False),
            nn.BatchNorm2d(out_channels),
            nn.ReLU(inplace=True),
        )

    def forward(self, x):
        return self.conv(x)

class UNET(nn.Module):
    def __init__(
            self, in_channels=3, num_classes=5, out_channels=1, features=[64, 128, 256, 512],
    ):
        super(UNET, self).__init__()
        self.ups = nn.ModuleList()
        self.downs = nn.ModuleList()
        self.pool = nn.MaxPool2d(kernel_size=2, stride=2)

        # Down part of UNET
        for feature in features:
            self.downs.append(DoubleConv(in_channels, feature))
            in_channels = feature

        # Up part of UNET
        for feature in reversed(features):
            self.ups.append(
                nn.ConvTranspose2d(
                    feature*2, feature, kernel_size=2, stride=2,
                )
            )
            self.ups.append(DoubleConv(feature*2, feature))

        self.bottleneck = DoubleConv(features[-1], features[-1]*2)
        self.final_conv = nn.Conv2d(features[0], out_channels, kernel_size=1)

    def forward(self, x):
        skip_connections = []

        for down in self.downs:
            x = down(x)
            skip_connections.append(x)
            x = self.pool(x)

        x = self.bottleneck(x)
        skip_connections = skip_connections[::-1]

        for idx in range(0, len(self.ups), 2):
            x = self.ups[idx](x)
            skip_connection = skip_connections[idx//2]

            if x.shape != skip_connection.shape:
                x = TF.resize(x, size=skip_connection.shape[2:])

            concat_skip = torch.cat((skip_connection, x), dim=1)
            x = self.ups[idx+1](concat_skip)

        return self.final_conv(x)
</code></pre>

eqy: <p>Looking at a de-facto reference implementation, it looks like all that needs to be changed is the number of output channels in the last convolution: <a href="https://github.com/milesial/Pytorch-UNet/blob/master/unet/unet_model.py" rel="noopener nofollow ugc">Pytorch-UNet/unet_model.py at master · milesial/Pytorch-UNet (github.com)</a></p>

hanoody: <p>Thank you very much!</p>
<p>I guess applications with 1 or 3 classes are really convenient because the model output will already have a valid image shape.</p>