akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Nested nn.ParameterDict

hamedB: <p>Hi<br>
Is it possible to have nested nn.ParameterDict like nested nn.ModuleDict?If not, is there any way to implemented it?</p>
<p>Thank you.</p>

ptrblck: <p>No, I don’t think that’s possible as you would get this error:</p>
<pre><code class="lang-python">TypeError: cannot assign 'torch.nn.modules.container.ParameterDict' object to parameter 'b' (torch.nn.Parameter or None required)
</code></pre>
<p>for:</p>
<pre><code class="lang-python">p1 = nn.ParameterDict({'a': nn.Parameter(torch.randn(1)),
                       'b': nn.ParameterDict({'ba': nn.Parameter(torch.randn(1))})})
</code></pre>
Cannot load a pre-trained model

Omar_Samir: <p>I am a beginner on this. I want to use a pre-trained model that I found somewhere. I do not know the exact architecture of the model, I only know that it is based on resnet18. When I load the file, I get the following dictionary. Any help on how to make a PyTorch model out of this?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/9/d97f6072e8dc79862d48b1898e90ef8f9ef18773.png" data-download-href="https://discuss.pytorch.org/uploads/default/d97f6072e8dc79862d48b1898e90ef8f9ef18773" title="Screen Shot 2022-03-18 at 9.32.43 AM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/9/d97f6072e8dc79862d48b1898e90ef8f9ef18773_2_690x282.png" alt="Screen Shot 2022-03-18 at 9.32.43 AM" data-base62-sha1="v24vTPdWRX1srNuoOQRkL7d9w6n" width="690" height="282" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/9/d97f6072e8dc79862d48b1898e90ef8f9ef18773_2_690x282.png, https://discuss.pytorch.org/uploads/default/optimized/3X/d/9/d97f6072e8dc79862d48b1898e90ef8f9ef18773_2_1035x423.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/d/9/d97f6072e8dc79862d48b1898e90ef8f9ef18773_2_1380x564.png 2x" data-dominant-color="F2F2F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2022-03-18 at 9.32.43 AM</span><span class="informations">2146×880 143 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

Mukesh1729: <p>I think you need the model architecture in this case to load the weights. You can try other models from torch. hub() which can be loaded without the need of the architecture depending on your use case</p>
How to calculate gradient of y=(-x.exp()).exp()

jamesp: <p>Currently, if x is too big( like 100), it’ll result in NaN, while it should be 0.</p>

jamesp: <p>I defined a custom function to fix this:</p>
<pre><code class="lang-auto">class Expmexp(Function):
    """
    y=e**(-e**(x)), to avoid NaN backward when x is big.
    """

    @staticmethod
    def forward(ctx, input):
        x = input.clamp(max=80)
        ctx.save_for_backward(x)
        return (-x.exp()).exp_()

    @staticmethod
    def backward(ctx, grad_output):
        """
        f' = -e**x e**(-e**x)
        """
        grad_input = None
        if ctx.needs_input_grad[0]:
            input, = ctx.saved_tensors
            ex = input.exp().neg_()
            grad_input = ex.mul_(ex.exp()).mul_(grad_output)
        return grad_input

</code></pre>
How to define a default constructor on a custom module

iiSeymour: <p>I’m starting to use the C++ frontend and want to create a custom <code>Module</code> but I am not sure how to define the default constructor so my <code>Model</code> can use it.</p>
<p>Take a simple example of the <code>Convolution</code> below that just wraps <code>Conv1d</code>, <code>SiLU</code> and sets the padding…</p>
<pre><code class="lang-auto">struct Convolution : Module {

    Convolution(int size, int outsize, int k, int stride)  {
        conv = register_module("conv", Conv1d(Conv1dOptions(size, outsize, k).stride(stride).padding(k/2)));
        activation = register_module("activation", SiLU());
    }

    torch::Tensor forward(torch::Tensor x) {
        return activation(conv(x));
    }
    Conv1d conv{nullptr};
    SiLU activation{nullptr};

};

struct Model : Module {
    Model(int size, int outsize) {
        conv1 = Convolution(1, 4, 5, 1);
        conv2 = Convolution(4, 16, 5, 1);
        conv3 = Convolution(16, outsize, 19, 5);
    }
    Convolution conv1{nullptr}, conv2{nullptr}, conv3{nullptr};
};
</code></pre>
<p>This leads to the following error from the compiler…</p>
<pre><code class="lang-auto">error: no matching constructor for initialization of 'Convolution'
    Convolution conv1{nullptr}, conv2{nullptr}, conv3{nullptr};
                     ^~~~~~~~~
note: candidate constructor (the implicit copy constructor) not viable: no known conversion from 'nullptr_t' to 'const Convolution' for 1st argument
struct Convolution : Module {
       ^</code></pre>

ptrblck: <p>I’m not sure but wouldn’t the c’tor in <code>Model</code> need to use <code>register_module</code> for each <code>Convolution</code> object as well?</p>

iiSeymour: <p>Yes but <code>register_module</code> takes a <code>ModuleHolder</code> not a <code>Module</code> so I think the following is correct <em>(and this also solves the original question I had)</em>.</p>
<pre><code class="lang-auto">struct ConvolutionImpl : Module {

    ConvolutionImpl(int size = 1, int outsize = 1, int k = 1, int stride = 1)  {
        conv = register_module("conv", Conv1d(Conv1dOptions(size, outsize, k).stride(stride).padding(k/2)));
        activation = register_module("activation", SiLU());
    }

    torch::Tensor forward(torch::Tensor x) {
        return activation(conv(x));
    }

    Conv1d conv{nullptr};
    SiLU activation{nullptr};

};

typedef ModuleHolder&lt;ConvolutionImpl&gt; Convolution;

struct Model : Module {
    Model(int size, int outsize) {
        conv1 = register_module("conv1", Convolution(1, 4, 5, 1));
        conv2 = register_module("conv2", Convolution(1, 16, 5, 1));
        conv3 = register_module("conv3", Convolution(16, outsize, 19, 5));
    }
    Convolution conv1{nullptr}, conv2{nullptr}, conv3{nullptr};
};
</code></pre>
Suggestions for Training strategy on continuous video stream

ekmungi: <p>I have videos that are each about 30-40 mins long. With the first 5-10 mins (at 60fps, can be down-sampled to 5fps) are one type of activity that would be categorized by label-1 and the rest of the video as label-2. I started off by using CNN-LSTM to do this prediction (Resnet-50 + LSTM + FC-classifier).</p>
<p>For training my initial approach was to treat this as an activity classification task. So, I split my videos into smaller segments with each segment having a label.</p>
<p>video1.mp4 (5 mins) → label-1<br>
Split into 30 seconds → <br>
video1_0001.mp4 → label-1<br>
.<br>
.<br>
.</p>
<p>video1_0010.mp4 → label-1</p>
<p>But, with this strategy even after 100 epochs the network does not train. I can at the most fit about 40 frames on the 2-GPUs, but a 30 second segment of video <span class="mention">@5fps</span> has about 150 frames. Any further subsampling seems to not capture the essence of the video segment.</p>
<p>I also tried training without shuffling and with single thread, so a single stream is loaded continuously. But perhaps it’s not the right strategy.</p>
<p>I wanted to request some help on how to tackle the problem. I would really appreciate some insights into a training strategy for this problem.</p>
<ul>
<li>Is using CNN-LSTM the right strategy here?</li>
</ul>
<p>=== Update ===</p>
<p>After reading a few other posts on similar topics,</p>
<ul>
<li><a href="https://ai.stackexchange.com/questions/17661/how-to-process-data-in-a-data-stream-for-a-lstm" rel="noopener nofollow ugc">data stream for a LSTM</a></li>
<li><a href="https://ai.stackexchange.com/questions/21740/action-recognition-using-video-stream-data" rel="noopener nofollow ugc">action recognition on video stream</a></li>
<li><a href="https://discuss.pytorch.org/t/video-frame-classificaion-with-a-few-long-videos/119001">Classification for long videos</a></li>
</ul>
<p>I feel that to get the network to see a larger part of the sequence, I will have to use more GPUs or resize images. However, since the pretrained Resnet accepts 224x224, I will need more GPUs. But I am curious; is there another strategy? Because the question could also be about what is the ideal segment length that would enable the network to learn.</p>
<p>From my perception, a 30 second video sampled <span class="mention">@5fps</span> at the bare minimum captures the context. From observation so far, going below this number hasn’t allowed the network to learn.</p>
'RuntimeError: falseINTERNAL ASSERT FAILED' when using torch.linalg.lstsq

IwakuraRein: <p>I tried to implement torch.linalg.lstsq at the end of my CNN. Under default settings it usually outputs Nan so I switched the driver to ‘gelsy’. ‘gelsy’ supports rank-deficient matrics but can only run on CPU according to the <a href="https://pytorch.org/docs/1.11/generated/torch.linalg.lstsq.html" rel="noopener nofollow ugc">documentation</a>.</p>
<p>My codes are:</p>
<pre><code class="lang-python">    def _lstsq(self, A, B):
        # A is the feature map extracted by CNN and B is the reference image
        N, C_A, H, W, = A.size()
        C_B = B.size(1)

        A = A.permute([0,2,3,1]).reshape(N, H*W, C_A)
        ones = torch.ones_like(A[:, :, 0:1])
        A = torch.cat([ones, A], dim=2)
        B = B.permute([0,2,3,1]).reshape(N, H*W, C_B)

        params = torch.linalg.lstsq(A, B, driver='gelsy').solution

        out = torch.nn.functional.relu(torch.bmm(A, params))
        out = out.reshape(N, H, W, C_B)
        out = out.permute([0, 3, 1, 2])
        return out

    # ......

    def lstsq_layer(self, A, B):
        A = A.cpu()
        B = A.cpu()
        N,C_A,H,W = A.size()

        # normalize input A to [-1,1]
        A = A.reshape(N, C_A, -1)
        max_a = torch.max(A, dim=2, keepdim=True).values
        min_a = torch.min(A, dim=2, keepdim=True).values
        A = (2*A2-max_a-min_a)/(max_a-min_a)
        A = A.reshape(N, C_A, height, width)

        # calling torch.linalg.lstsq
        output = self._lstsq(A, B)
        output.to(device)
        return output
</code></pre>
<p>It caused <code>RuntimeError: falseINTERNAL ASSERT FAILED at "/opt/conda/conda-bld/pytorch_1646755903507/work/aten/src/ATen/native/LinearAlgebraUtils.h":328, please report a bug to PyTorch. torch.linalg.lstsq: Argument 4 has illegal value. Most certainly there is a bug in the implementation calling the backend library.</code></p>
<p>I then tried ‘gelsd’ and ‘gelss’; I also updated mkl with conda; the error was still.</p>
<p>Using ‘gels’ can make the codes get running but always ended in outputting Nans.</p>

ptrblck: <p>Could you try to install the latest nightly binary and rerun your code? If you are still hitting this error could you create an issue on <a href="https://github.com/pytorch/pytorch/issues">GitHub</a>?</p>

IwakuraRein: <p>Problem solved. It was because the normalization: <code>A = (2*A2-max_a-min_a)/(max_a-min_a)</code>. I forgot to add a small value to the denominator henceforth division by zero occurred. <img src="https://discuss.pytorch.org/images/emoji/apple/rofl.png?v=12" title=":rofl:" class="emoji" alt=":rofl:" loading="lazy" width="20" height="20"></p>
Change dataset member variable once per epoch

Manuel_Lopez: <p>I would like to change a member variable of the dataset class every time the dataloader iterates trough my data again (Once per epoch). Since the Sampler gets called once per epoch when calling <code>iter()</code> I though of the following:</p>
<pre><code class="lang-auto">class MySampler(object): 
    def __init__(self, dataset):
        self.dataset = dataset
        self.sampler = RandomSampler(self.dataset)
        self.first_epoch = True
        self.rng = default_rng()

    def __iter__(self):
        if not self.first_epoch:
            self.dataset.other_data_indices = np.concatenate([self.rng.choice(len(100), size=len(100), replace=False))
        self.first_epoch = False
        return self.sampler
</code></pre>
<p>However, it doesn’t work because probably the dataset passed in the creation of the Sampler class is not shared with the dataset that my dataloader is using. Does anyone know how could I change <code>self.dataset.other_data_indices</code> from my Dataset or from any class/function passed to the dataloader once per epoch? (I can’t change the Dataloader itself because is a custom version of a private repo)</p>

ptrblck: <p>I don’t know how your private <code>DataLoader</code> works, but <a href="https://discuss.pytorch.org/t/changing-transformation-applied-to-data-during-training/15671/13">this post</a> might be helpful.</p>

Manuel_Lopez: <p>Thank you, I think that would work. Is there a way to do this for all the workers if they are persistent?</p>

ptrblck: <p>I’m not aware of a clean way to change the copies in each worker process.</p>

jabbarshah: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>  Sorry to text you here. Could you please answer to the below problem?<br>
<a href="https://discuss.pytorch.org/t/attributeerror-module-torchvision-transforms-has-no-attribute-scale/146687">AttributeError: module ‘torchvision.transforms’ has no attribute ‘Scale’</a></p>

paganpasta: <p>How I understand is that you want 2 different sets of data depending on the epoch.<br>
The RandomSampler is initialised on the entire dataset so regardless of the flag or the attribute you will have all training samples in a training epoch.</p>
<p>One way to have custom choice of samples is to implement your own BatchSampler where in the <strong>iter</strong> call you can choose the indices you wish to use depending on the current epoch.</p>
How to remove a prediction head from the model?

JimW: <p>I am working on a ViT (Vision Transformer) related project and some low level definition is deep inside timm library, which I can not change.   The low level library definition involves a linear classification prediction head, which is not a part of my network.</p>
<p>Every thing was fine until I switched to DDP parallel implementation. Pytorch complained about some parameters which didn’t contribute to the loss, and it instructed me to use “find_unused_parameters=True”.  In fact, it is a common scenario and it worked again if I added this “find_unused_parameters=True” to the training routine.   However, I am only allowed to change the model definition in our code base,  but I cannot modify anything related to training …</p>
<p>So I guess the only thing I can do right now, is to “remove” the linear head from the model.<br>
Although I cannot dig into the low level definition of ViT,  but I can output this tensor like this:</p>
<pre><code class="lang-auto">encoder_output,   linear_head_output =  ViT(input)
</code></pre>
<p>Is it possible to remove this linear prediction head based on this linear_head_output tensor?</p>

tom: <p>The easiest could be to just assign a <code>nn.Identity()</code> module on the prediction head.<br>
This lets the model call into the prediction head as desired, but just returns the input.</p>
<p>Best regards</p>
<p>Thomas</p>

JimW: <p><a class="mention" href="/u/tom">@tom</a> Thanks for your suggestion.  My current work around is like this:<br>
<code> encoder_output = encoder_output + 0*linear_head_output</code><br>
It won’t affect the calculation of loss based on encoder_ouput, while making linear_head_output “used”. But it is obviously not an elegent way to fix this, and I am not sure if it also affect runtime.</p>
<p>Could you please provide some exemplary instruction how to make use of nn.Identity() in this case?  I got a feeling that it would be a better fix than mine, but I just don’t know to implement that.</p>

tom: <p>Basically it would just be something like (I don’t know what exactly the submodule is that is causing you trouble, so I’'m calling it <code>linear_head</code> here.)</p>
<pre><code class="lang-auto">        model.linear_head = torch.nn.Identity()
</code></pre>
<p>I have an <a href="https://github.com/MathInf/toroidal/blob/bff09f725627e4629d464008dc7c5f9d6322ebad/test/test_bert.py#L87" rel="noopener nofollow ugc">example of this</a> (albeit with my own BERT implementation) in the tests of my (educational + WIP) <a href="https://github.com/MathInf/toroidal/" rel="noopener nofollow ugc">Toroidal transformer library</a>.</p>
<p>Best regards</p>
<p>Thomas</p>
If I call the model twice before backward(), will that have a negative effect on training?

Miquel_Espinosa: <p>Hi.</p>
<p>I have been working in a vanilla siamese network in which, given an input image:</p>
<ul>
<li>I generate two different transformations for that same image</li>
<li>Forward pass both images through the network</li>
<li>Invert the transformations</li>
<li>Get the mean of both outputs and compute the loss.</li>
</ul>
<p>The code looks like this:</p>
<pre><code class="lang-auto">   output_1 = model(img_1)[0]
   output_2 = model(img_2)[0]

   # Undo transformations
   output_reset_1 = F_vision.affine(output_1,**inverse_params_1)
   output_reset_2 = F_vision.affine(output_2,**inverse_params_2)

   # Average both outputs
   final_output = (output_reset_1 + output_reset_2)/2

   # Compute loss with respect to ground truth heatmaps
   loss = criterion(final_output, heatmap_self.float())
</code></pre>
<p>I plot the loss function obtained below. As it can be seen, it soon flattens out and it stays there.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/e/d/ed034342d0ce2111e9bef1df3cbdae6184a194b5.png" alt="image" data-base62-sha1="xOI9lZ9uSdpuprLhy6BlOBWbTOl" width="593" height="394"></p>
<p>I would highly appreciate any feedback or insights on what the problem may be.</p>
<p>NOTE: The network without this vanilla ‘Siamese learning’ yields a much lower error with the same num of epochs</p>
<p>Thanks!!</p>

eqy: <p>I don’t see any obvious problems with this setup, but it is difficult to identify the source of the problem without more context on the training task and the precise differences with the single-image approach.</p>
Tutorials on detecting black screen from live video frames

vamshi: <p>Hi,<br>
I am looking for any tutorials/guidance on detecting black screens from live video frames. Any help on this is appreciated.</p>
<p>Thanks</p>

ptrblck: <p>I could imaging that taking the sum of all pixels of the current frame and using a simple threshold might work already assuming that “black screens” have (mostly) zero pixels while “non-black screens” contain some color information.</p>

vamshi: <p>Many thanks <a class="mention" href="/u/ptrblck">@ptrblck</a> for your response. lets say if we want to detect anomaly screen on web UI testing, would you suggest auto-encoder approach or classification approach for this problem statement?</p>

ptrblck: <p>I don’t know what “anomaly screen on web UI” would mean and how it would “look” so cannot recommend a specific algorithm to detect it.<br>
E.g. if the “anomaly” looks like a white noise screen, you should be able to use the entropy as a feature and applying a simple threshold might already work quite well. On the other hand, if the “anomaly” would change e.g. the text of a web UI, the use case could be much more complicated.</p>
CUDA driver error: a PTX JIT compilation failed

tangolin: <p>I have no idea why I ran into this error, here is the code snippet:</p>
<pre><code class="lang-auto">def message(self, x_j: Tensor, alpha_j: Tensor, alpha_i: OptTensor, alpha_e,index: Tensor, ptr: OptTensor, size_i: Optional[int]) -&gt; Tensor:

        alpha = alpha_j if alpha_i is None else alpha_j + alpha_i
        alpha += alpha_e # add edge features...
        alpha = F.leaky_relu(alpha, self.negative_slope)
        print('here')
        alpha = softmax(alpha, index, ptr, size_i)
        print('pass')
        self._alpha = alpha
        alpha = F.dropout(alpha, p=self.dropout, training=self.training)
</code></pre>
<p>The softmax function came from <code>from torch_geometric.utils import softmax</code>. The error pops at the softmax line, interesting it can always execute one time before there is an error. How can I fix this?</p>
<p>Here is the softmax code in torch_geometric</p>
<pre><code class="lang-auto">@torch.jit.script
def softmax(src: Tensor, index: Optional[Tensor] = None,
            ptr: Optional[Tensor] = None, num_nodes: Optional[int] = None,
            dim: int = 0) -&gt; Tensor:
    r""
    Computes a sparsely evaluated softmax.
    Given a value tensor :attr:`src`, this function first groups the values
    along the first dimension based on the indices specified in :attr:`index`,
    and then proceeds to compute the softmax individually for each group.

    Args:
        src (Tensor): The source tensor.
        index (LongTensor, optional): The indices of elements for applying the
            softmax. (default: :obj:`None`)
        ptr (LongTensor, optional): If given, computes the softmax based on
            sorted inputs in CSR representation. (default: :obj:`None`)
        num_nodes (int, optional): The number of nodes, *i.e.*
            :obj:`max_val + 1` of :attr:`index`. (default: :obj:`None`)
        dim (int, optional): The dimension in which to normalize.
            (default: :obj:`0`)

    :rtype: :class:`Tensor`
   ""
    if ptr is not None:
        dim = dim + src.dim() if dim &lt; 0 else dim
        size = ([1] * dim) + [-1]
        ptr = ptr.view(size)
        src_max = gather_csr(segment_csr(src, ptr, reduce='max'), ptr)
        out = (src - src_max).exp()
        out_sum = gather_csr(segment_csr(out, ptr, reduce='sum'), ptr)
    elif index is not None:
        N = maybe_num_nodes(index, num_nodes)
        src_max = scatter(src, index, dim, dim_size=N, reduce='max')
        src_max = src_max.index_select(dim, index)
        out = (src - src_max).exp()
        out_sum = scatter(out, index, dim, dim_size=N, reduce='sum')
        out_sum = out_sum.index_select(dim, index)
    else:
        raise NotImplementedError

    return out / (out_sum + 1e-16)
</code></pre>
Support for Deformable Convolution V2 in Pytorch

rammy_vadlamudi: <p>Hi,</p>
<p>I am using the latest version of torch and I couldn’t find any implementation of Deform Conv V2 , I could only see DeformConv V1 in torchvision.</p>
<p>Do you plan to provide a native support for this any time soon ?</p>
<p>I tried to add extensions from github :</p>
<ol>
<li><a href="https://github.com/xi11xi19/CenterNet2TorchScript" rel="noopener nofollow ugc">xi11xi19/CenterNet2TorchScript: centernet pytorch model to torch script model (github.com)</a></li>
<li><a href="https://github.com/chengdazhi/Deformable-Convolution-V2-PyTorch" rel="noopener nofollow ugc">chengdazhi/Deformable-Convolution-V2-PyTorch: Deformable ConvNets V2 (DCNv2) in PyTorch (github.com)</a></li>
</ol>
<p>They seem old and not compatible with latest version of Pytorch.</p>
<p>Is there any resource that I could use on latest version of pytorch ?</p>

developer0hye: <p>Refer to <a href="https://github.com/developer0hye/Simple-PyTorch-Deformable-Convolution-v2" rel="noopener nofollow ugc">Simple-PyTorch-Deformable-Convolution-v2</a></p>

thancaocuong: <p>you can refer to this repo<br>
<a href="https://github.com/jinfagang/DCNv2_latest" rel="noopener nofollow ugc">https://github.com/jinfagang/DCNv2_latest</a></p>
Custom FasterRCNN

Bruno_Oliveira: <p>Hey everyone.</p>
<p>I’m trying to modify fasterrcnn_mobilenet_v3_large_320_fpn to train it on a custom 5 classes dataset. The problem is I’m having issues using this modified model.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/5/a5885448738dd8edf4ed27222c28cebd1268169d.png" data-download-href="https://discuss.pytorch.org/uploads/default/a5885448738dd8edf4ed27222c28cebd1268169d" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/5/a5885448738dd8edf4ed27222c28cebd1268169d_2_690x323.png" alt="image" data-base62-sha1="nCmRJpLVGPXY7qqCqGYJUsgGua9" width="690" height="323" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/5/a5885448738dd8edf4ed27222c28cebd1268169d_2_690x323.png, https://discuss.pytorch.org/uploads/default/optimized/3X/a/5/a5885448738dd8edf4ed27222c28cebd1268169d_2_1035x484.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/a/5/a5885448738dd8edf4ed27222c28cebd1268169d_2_1380x646.png 2x" data-dominant-color="323058"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1721×808 121 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>This raises the error: TypeError: forward() takes 2 positional arguments but 3 were given</p>
<p>I have read that this is not a sequential model, so maybe that’s why this approach is not working, but I’m still wondering how should I approach this issue. I’m just interested in getting the pretrained model, adapting it to 5 classes and training it. What would be the easiest way to do so? Could you provide me some insights and/or reading material?</p>
<p>Thanks in advance.</p>

ptrblck: <p>I think the easier approach would be to specify the <code>num_classes</code> during the model instantiation and to use a pretrained backbone if needed:</p>
<pre><code class="lang-python">model = torchvision.models.detection.fasterrcnn_mobilenet_v3_large_320_fpn(
    pretrained=False, num_classes=5, pretrained_backbone=True)
</code></pre>

Bruno_Oliveira: <p>That is true, it solved for me. Thanks for helping out, as always.</p>
Setting LSTM Parameters

iiSeymour: <p>I have 4 <code>torch::Tensor</code> in C++ that I want to set as the <code>{weight,bias}_{i,h}h</code> parameters on a LSTM… what is the correct method to do this?</p>

iiSeymour: <p>Something like?</p>
<pre><code class="lang-auto"> auto lstm_params = lstm-&gt;named_parameters();

 lstm_params.find("weight_ih_l0")-&gt;data() = w_ih;
 lstm_params.find("weight_hh_l0")-&gt;data() = w_hh;
 lstm_params.find("bias_ih_l0")-&gt;data() = b_ih;
 lstm_params.find("bias_hh_l0")-&gt;data() = b_hh;
</code></pre>
Converting Detectron2 Predicted Boxes to COCO Format

Lu_Bin_Liu: <p>I’m wondering if there’s a pre-existing script that takes in a list of predictions (and their corresponding image file names) from a Detectron2 model and converts them to COCO format. The predictions are a list of instance objects:</p>
<pre><code>[
{'instances': Instances(num_instances=1, image_height=h, image_width=w, fields=[pred_boxes: Boxes(tensor([[xmin, ymin, xmax, ymax]])), scores: tensor([score]), pred_classes: tensor([class])])}, 
...]</code></pre>
How to implement directed labeled hypergraphs

Aeryan: <p>Hi,</p>
<p>I’m a beginner at Pytorch and was wondering the best method/framework/library to use for my problem. I need to apply Graph Neural Networks (GNNs) to relational data, represented in first-order logic. The GNN architecture I need to implement is given in the following <a href="https://arxiv.org/pdf/2109.10129.pdf" rel="noopener nofollow ugc"><strong>paper</strong></a> in <em>Algorithm 1</em>. The only change with respect to the method proposed in the paper is that I need to add a global readout (i.e., compute a graph embedding) to each GNN layer, in what is known as a <em>Aggregate-combine-Readout GNN</em> or <em>ACR-GNN</em>.</p>
<p>I will represent the input data as a <strong>directed labeled hypergraph</strong>. This hypergraph will also allow for (hyper)edges with a single node and more than one hyperedge (although with different labels) between the same group of nodes.</p>
<p>I have seen that <strong>Pytorch Geometric</strong> can be used to work with graphs, GNNs, etc. However, I don’t know if it suits my specific representation as directed labeled hypergraphs. The closest thing I have seen in Pytorch is the class <a href="https://pytorch-geometric.readthedocs.io/en/latest/modules/nn.html#torch_geometric.nn.conv.HypergraphConv" rel="noopener nofollow ugc"><strong>torch_geometric.nn.conv.HypergraphConv</strong></a>, which implements a hypergraph convolution operation, but <strong>I don’t know if it works for directed edges</strong> or assumes every edge is undirected.</p>
<p>Regarding the edge labels, each label represents a different “type” of edge, so I need to perform the message-passing computations separately for each edge type and then aggregate the results (messages) for each node for each edge type, as shown in the paper mentioned above.</p>
<p>Do you think I can rely on the functionality provided in Pytorch Geometric or I will need to implement this specific case of GNN by hand?</p>
<p>Thank you</p>
A case of exploding RAM while using pinned memory (probably)

ayhyap: <p>Just wanted to document an issue I encountered that I couldn’t find online, and how I resolved it.</p>
<p>tl;dr if you are collecting the labels (or any tensor) from batches while iterating through a dataset using a dataloader with pinned memory enabled, use a copy of it with clone() instead of the original.</p>
<p>In other words…</p>
<pre><code class="lang-auto">pred_list = []
label_list = []
for batch in train_loader:
    '''
    training code here
    '''
    pred_list.append(preds.cpu().detach())
    # label_list.append(batch['labels']) # bad
    label_list.append(batch['labels'].clone()) # better
</code></pre>
<p>What I think is happening: the label tensors that come directly from the batch occupy pinned memory and cannot be moved to virtual memory, leading to RAM usage increasing as they are collected into the label list.</p>
<p>In my case, RAM usage was climbing over 20GB before the fix (and causing thrashing), and stayed at 5GB with the fix.</p>
Backpropagation based on aggregated output of all batches

aminpdi: <p>Hi everyone,</p>
<p>I have one question that I want to make sure about it.</p>
<p>My loss function is defined on the aggregated output of all batches. It means I don’t have a loss function to evaluate the output of each batch.  I aggregate the output of all batches to generate the final output and I have a cross-entropy on the final output.</p>
<p>Does backpropagation work properly by loss.backward and optimizer.step?</p>
<p>Thank you</p>

tom: <p>This would only work if you had the memory to keep the computational graph of all batches, which is highly unlikely.</p>
<p>Best regards</p>
<p>Thomas</p>

rfmac: <p>Hi, <a class="mention" href="/u/aminpdi">@aminpdi</a>!</p>
<p>Answering you question, yes, backpropagation work properly for this scenario you described.<br>
But I also agree with <a class="mention" href="/u/tom">@tom</a>, it could require a considerable amount of memory.</p>
<p>To reduce the amount of required memory, you could use the gradient accumulation technique.<br>
The good news is that it’s quite simple to implement it in Pytorch, you just need to call <code>loss.backward()</code> as many times you want (in your case after all the batches that you want to accumulate), then when you want to update the model weights, just call <code>optimizer.step()</code>. If you’re going to repeat this process, don’t forget to use <code>model.zero_grad()</code> after the step function <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"></p>
<p>Best Regards,<br>
Rafael Macedo.</p>

aminpdi: <p>Sorry, I think my question was not clear enough. let’s say I have a minibatch with size B * C * H * W. Then my output size is B * Cp * Hp * Wp. the output of Regular cross-entropy is B*1. Then we get for example average of errors to have the final error for backpropagation.</p>
<p>In my case, On the OUTPUT batch B * Cp * Hp * Wp, I want to first do an averaging on all images in the current mini-batch (OUTPUT.mean(dim=0)), then  OUTPUT will be Cp * Hp * Wp. Finally, I apply my cross-entropy loss to the final output.</p>
<p>The reason is, I don’t have supervision on each image of the mini-batch. in my case Cp=2,Hp=1,Wp=1.</p>
Forcing tensors to be on the same GPU

avalon1511: <p>Is there a way to use only one GPU out of multiple available GPUs when training a model?</p>
<p>I have two GPUs and I havent used <code>nn.DataParallel</code> because of memory overhead issues. However it seems like the code automatically distributes the tensors between the two GPUs leading to this error:</p>
<pre><code class="lang-auto">codes = codes + self.latent_avg.repeat(codes.shape[0], 1, 1) 

RuntimeError: Expected all tensors to be on the same device, but found at 
least two devices, cuda:1 and cuda:0!
</code></pre>
<p>As I understand, all my tensors are not on the same GPU -  is there a way to make this happen?</p>

tom: <p>There is the <code>CUDA_VISIBLE_DEVICES</code> envirionment variable or you could use <code>cuda:0</code> as the device to get the first GPU.</p>
<p>Best regards</p>
<p>Thomas</p>
Apply different custom functions to different subsets

lima: <p>I have a dataset class function:</p>
<pre><code class="lang-auto">class MYDataset(Dataset):
    def __init__(self, path):
        df = read_csv(path, header=None, delimiter=r"\s+")
        df = df.iloc[:, 1:-1].values
        self.X = df[:, 1:]
        self.y = df[:, 0]
        self.y = self.y.reshape((len(self.y), 1))

    def __len__(self):
        return len(self.X)

    def __getitem__(self, idx):
        return [self.X[idx], self.y[idx]]
    
    def get_splits(self, n_test=0.3):
        testing_size = round(n_test * len(self.X))
        train_size = len(self.X) - testing_size
        val_size = round(testing_size /2)
        test_size = testing_size - val_size
        return random_split(self, [train_size, val_size, test_size])
    
def prepare_data(path):
    dataset = MYDataset(path)
    train, val, test = dataset.get_splits()
    train_dl = DataLoader(train, batch_size=64, shuffle=True)
    val_dl = DataLoader(val, batch_size=len(val), shuffle=False)
    test_dl = DataLoader(test, batch_size=len(test), shuffle=False)
    return train_dl, val_dl, test_dl
</code></pre>
<p>and I have 2 other custom functions:</p>
<pre><code class="lang-auto">  def augment_data(array):
     ...
     return(gen_data)
</code></pre>
<pre><code class="lang-auto">  def BINNED(dataframe):
     ...
     return(data_ohe, 0,1)
</code></pre>
<p>How can I apply the function <code>BINNED</code> on train, val, and test subsets and apply the function <code>augment_data</code> <strong>only</strong> on ‘train’ subset?</p>

rvarm1: <p>cc <a class="mention" href="/u/vitalyfedyunin">@VitalyFedyunin</a> re: Data loader question</p>

ptrblck: <p>You could create three different datasets for training, validation, and testing using the corresponding transformations. Afterwards, you could wrap each of them into a <code>Subset</code> and pass the desired indices to them so that no samples will be repeated.<br>
This approach would need more memory if you are preloading the data and depending on your use case you might want to enable lazy loading.</p>

lima: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>  Thank you for the suggestion. I’m new to pytorch. Do you have an example I can look at?</p>

ptrblck: <p>Something like this should work:</p>
<pre><code class="lang-python">train_dataset = MyDataset(transform=train_transform)
val_dataset = MyDataset(transform=val_transform)
test_dataset = MyDataset(transform=test_transform)

# you can use e.g. train_test_split from sklearn here to create the indices
train_idx, val_idx, test_idx = create_indices() 

train_dataset = Subset(train_dataset, train_idx)
val_dataset = Subset(val_dataset, val_idx)
test_dataset = Subset(test_dataset, test_idx)

train_loader = DataLoader(train_dataset)
val_loader = DataLoader(val_dataset)
test_loader = DataLoader(test_dataset)
</code></pre>
Applying the gradient of a task onto another task for identifying task groupings for multitask learning

kek: <p>Hello,</p>
<p>I am a complete newbie on pytorch. I am trying to implement “Efficiently Identifying Task Groupings for Multi-Task Learning” paper for a bioinformatics task. As far as I understand, it requires applying the gradient of one task onto others and calculating an inter task affinity score based on applied gradient from this other task.</p>
<p>My model consists of a hard parameter sharing backbone that splits to 30-40 individual sub modules for each task.</p>
<p>Can I apply the gradient of one of the tasks onto another? And if so how?</p>
<p>Thank you in advance</p>
<p>My model is like this:</p>
<p>class NN(nn.Module):<br>
def <strong>init</strong>(self, input_size, num_classes):<br>
super(NN, self).<strong>init</strong>()<br>
self.num_classes = num_classes<br>
self.shared_network = nn.Sequential(<br>
nn.Dropout(0.5),<br>
nn.Linear(input_size, 256),<br>
nn.BatchNorm1d(num_features=256),<br>
nn.ReLU(),<br>
nn.Dropout(0.7)<br>
)<br>
self.sub_networks = nn.ModuleList()<br>
for _ in range(self.num_classes):<br>
self.sub_networks.append(nn.Sequential(<br>
nn.Linear(256,32),<br>
nn.Linear(32,1),<br>
nn.Sigmoid()<br>
)<br>
)</p>
<pre><code>def forward(self, x):
    outputs = []
    representation = self.shared_network(x)
    for i in self.sub_networks:
        outputs.append(i(representation))
    return torch.cat(outputs,axis=1)</code></pre>
F.cross entropy vs torch.nn.Cross_Entropy_Loss

cherry: <p>What is the difference between pytorch <code>F.cross entropy</code> vs <code>torch.nn.Cross_Entropy_Loss</code>?<br>
As far as I understand <code>torch.nn.Cross_Entropy_Loss</code> is calling <code>F.cross entropy</code></p>

albanD: <p>Hi,</p>
<p>There isn’t much difference for losses.<br>
The main difference between the <code>nn.functional.xxx</code> and the <code>nn.Xxx</code> is that one has a state and one does not.<br>
This means that for a linear layer for example, if you use the <code>functional</code> version, you will need to handle the weights yourself (including passing them to the optimizer or moving them to the gpu) while the <code>nn.Xxx</code> version will do all of that for you with <code>.parameters()</code> or <code>.to(device)</code>.</p>
<p>For loss functions, as no parameters are needed (in general), you won’t find much difference. Except for example, if you use cross entropy with some weighting between your classes, using the <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.CrossEntropyLoss"><code>nn.CrossEntropyLoss()</code></a> module, you will give your weights only once while creating the module and then use it. If you were using <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.functional.cross_entropy">the functional version</a>, you will need to pass the weights every single time you will use it.</p>

cherry: <p>Super clear, thanks so much.</p>

Srinjoy_Mukherjee: <p>Can you tell me what do you mean by passing the wieights manually?<br>
I mean if i use lossFn=nn.CrossEntropyLoss()<br>
then the code i use is:</p>
<blockquote>
<p>optimiser = optim.SGD(net.parameters(), lr = 0.01, momentum=0.9)<br>
loss = lossFn(output,actual)<br>
optimiser.zero_grad()<br>
loss.backward()<br>
optimiser.step()</p>
</blockquote>
<p>What differently do i have to do with the functionall one?<br>
Thanks in advance</p>

albanD: <p>You are not using a weighted cross entropy. So no difference as mentionned above.</p>
Missing libcaffe2.so

nurettinabaci: <p>Hello, I built PyTorch with make install to use caffe2 with c++ but <code>libcaffe2.so</code> file is missing. <code>sudo ldconfig -p | grep "libcaffe2*"</code> is returning nothing so I couldn’t run <a href="https://github.com/PacktPublishing/Caffe2-Quick-Start-Guide/blob/master/Chapter01/ch1.cpp" rel="nofollow noopener">that simple code</a>. (I think it didn’t build properly.) Also in cmakelist.txt <code>option(BUILD_SHARED_LIBS "Build libcaffe2.so" ON)</code><br>
What can I do?</p>
<p>Last lines of the make command is:</p>
<pre><code class="lang-auto">[100%] Linking CXX shared library ../../lib/libtorch_python.so
[100%] Built target torch_python
Install the project...
-- Install configuration: "Release"
-- Set runtime path of "/usr/local/bin/protoc" to "$ORIGIN"
-- Set runtime path of "/usr/local/lib/libc10.so" to "$ORIGIN"
-- Set runtime path of "/usr/local/lib/libtorch.so" to "$ORIGIN:/usr/lib/openmpi/lib:/home/ubuntu/anaconda2/lib"
-- Set runtime path of "/usr/local/lib/python2.7/site-packages/caffe2/python/caffe2_pybind11_state.so" to "$ORIGIN:/home/ubuntu/anaconda2/lib"
-- Set runtime path of "/usr/local/lib/libtorch_python.so" to "$ORIGIN:/home/ubuntu/anaconda2/lib:/usr/lib/openmpi/lib"
-- Set runtime path of "/usr/local/lib/libshm.so" to "$ORIGIN:/home/ubuntu/anaconda2/lib"
-- Set runtime path of "/usr/local/bin/torch_shm_manager" to "$ORIGIN/../lib:/home/ubuntu/anaconda2/lib"
-- Set runtime path of "/usr/local/lib/libcaffe2_detectron_ops.so" to "$ORIGIN:/home/ubuntu/anaconda2/lib"
-- Set runtime path of "/usr/local/lib/libcaffe2_observers.so" to "$ORIGIN:/home/ubuntu/anaconda2/lib"
(base) ubuntu@ubuntu-vm:~/Desktop/pytorch/build$
</code></pre>

wyh5625: <p>I have also faced this problem. Did you find the solution?</p>
Debug libtorch while developing C++ extension for PyTorch

anubane: <p>Kindly read the detailed issue reproduction steps from <a href="https://github.com/microsoft/vscode-cpptools/issues/9033#issue-1171017658" rel="noopener nofollow ugc">Mixed mode Python C++ debug issue with PyTorch and libtorch</a>.</p>
<p>This is currently my problem: I am unable to <em>look inside</em> the ScriptModule object received at the C++ side from the Python side. Is there some specific configuration that I need to use while building the project? The libtorch documentation is way too limited for this.</p>
Interpolation at given indexes

abhisek: <p>Some indexes of a <code>k</code> dimensional tensor is stored at tensor <strong>T</strong> (<code>n x k</code>) with corresponding vector values stored in tensor <strong>V</strong> (<code>n x p</code>). I want to interpolate values at indexes given by another tensor <strong>I</strong> (<code>m x k</code>). How this can be achieved?</p>
Change ResNet50's number of output filters

CarlosHernandezP: <p>Hello,</p>
<p>I am trying to use the torchvision’s pretrained ResNet50 model to extract features. The key point is that I want to change the number of output filters.</p>
<p>If we take a look at the <code>torchvision.models.resnet50()</code> we can see that the last part of the network has the following layers:</p>
<pre><code class="lang-auto">    (2): Bottleneck(
      (conv1): Conv2d(2048, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv3): Conv2d(512, 2048, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn3): BatchNorm2d(2048, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace=True)
    )
  )
  (avgpool): AdaptiveAvgPool2d(output_size=(1, 1))
  (fc): Linear(in_features=2048, out_features=1000, bias=True)
</code></pre>
<p>I want to get rid of the FC layer and change the number output filters from 2048 to 512 features thus I apply the following code:</p>
<pre><code class="lang-auto">self.encoder.layer4[2].conv3 = nn.Conv2d(512,512, kernel_size=(1,1), stride=(1,1), bias=False)
self.encoder.layer4[2].bn3   = nn.BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
self.encoder.fc = nn.Identity()
</code></pre>
<p>It might be worth noting that the resnet50 model has 4 main blocks, named “layerX” and 4 is the last one.</p>
<p>I thought that by having these changes, the output of the network would be a tensor of <code>[bs, 512]</code> but I obtain the following error message:</p>
<pre><code class="lang-auto">RuntimeError: The size of tensor a (512) must match the size of tensor b (2048) at non-singleton dimension 1
</code></pre>
<p>Reading <a href="https://discuss.pytorch.org/t/training-resnet18-output-of-resnet18-images/88674/2">this previous post</a>, my approach seems correct but something is clearly missing. I tried initializing the avgpool layer (although it  has no parameters of its own) but the error remains.</p>
<p>Any suggestions on what to do are highly appreciated.</p>

ptrblck: <p>The error is raised in the skip connection:</p>
<pre><code class="lang-python">    out += identity

RuntimeError: The size of tensor a (512) must match the size of tensor b (2048) at non-singleton dimension 1
</code></pre>
<p>as you are manipulating the output layer(s) of the block while the input still uses its original channel dimension.</p>

CarlosHernandezP: <p>Oh, of course, it is a Residual Network after all. It seems that I will have to build the network by myself then.</p>
<p>Option B is to just change the last FC to:</p>
<pre><code class="lang-auto">self.encoder.fc = nn.Linear(2048, 512)
</code></pre>
<p>Thank you very much for your help.</p>
Understanding all the different ways in which one can change the backward computational graph for a model

puranjay_mishra: <p>Hi everyone!<br>
I am trying to change the backwards computational graph for a custom model in which use a regular python function to change the weights of the model before performing the forward and backward prop.</p>
<p>I am trying to change the backward computation for the gradient computation of this function. I am familiar with backward hooks and tried doing this with them. That did not work (I printed out the new model with hooks using torchviz to confirm this).<br>
I would like to know if there are any other ways to change the backward computation graph for a model and even better, to achieve my goal.</p>
<p>Thank you in advance for any help!</p>
Pytorch CUDA build for compute capability 3.0

Shahzad_Akram: <p>I’m pretty new to Pytorch. After some googling, I found that current builds support compute capability &gt;=3.5. I have a laptop with a weak GPU with compute capability 3.0. It supports up to CUDA 11.4 and works fine with Blender. I tried all available PyTorch builds for different CUDA versions but nothing is working. Should I build my own for 3.0? Would it work?</p>
Is it possible to iterate through all model parameters AND see which module type they belong to?

Sam_Lerman: <p>I want to:</p>
<ol>
<li>Iterate through model parameters. Easily can be done: <code>model.parameters()</code>
</li>
</ol>
<p>And:</p>
<ol start="2">
<li>See what module type they belong to, e.g., <code>type(param.MODULE) == nn.Conv2d</code>
</li>
</ol>
<p>However, I’m not sure how to get the param’s module… does <code>model.named_parameters()</code> return something I can use here to do this?</p>

ptrblck: <p>You could use the returned name to get the module. E.g this code would print the registered modules at the highest hierarchy level:</p>
<pre><code class="lang-python">model = torchvision.models.resnet50()
for name, param in model.named_parameters():
    print(name)
    print(getattr(model, name.split('.')[0])) # print "highest" module
</code></pre>

Sam_Lerman: <p>I’m not sure this would print something like <code>nn.BatchNorm2d</code>, but it suits my purposes, thanks.</p>
Even though I have cuda toolkit installed, torch.cuda.is_available() is False

chandan_hariharan: <p>Previously, the program was working fine and i was getting torch.cuda.is_available() true. This was when i had windows 11 insider program. But, later i encountered issues with windows 11 insider program and therefore i reinstalled stable version of windows 11. But, now when i run torch.cuda.is_available() i get false even though i have cuda toolkit 11.3.</p>
<p>I downloaded it from <a href="https://pytorch.org/" rel="noopener nofollow ugc">https://pytorch.org/</a>.<br>
I have a RTX 3050 GPU.</p>
<p>Kindly help me to resolve this issue.</p>

ptrblck: <p>Based on your description I would guess that your NVIDIA driver installation wasn’t successful after installing the “stable” Windows 11 version.</p>