akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Save folder of images

jasg: <p>full_dataset = datasets.ImageFolder(os.path.join(root, ‘Imagenes_categoria’), transform=train_transform)<br>
ttv=0.7 <span class="hashtag">#tasa</span> de entrenamiento y test<br>
size_t = int(ttv * len(full_dataset))<br>
size_v = len(full_dataset) - size_t<br>
train_data, test_data= torch.utils.data.random_split(full_dataset, [size_t, size_v])</p>
<p>how can I save an image folder after the split, the train_data folder on my disk, and then apply validations to the epochs?</p>
Using ConcatDataset

su-yeon.kim: <p><a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.ConcatDataset" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/data.html#torch.utils.data.ConcatDataset</a></p>
<p>I’m using ConcatDataset class which is justified in pytorch document.<br>
And when after the ConcatDataset ‘TypeError: object of type ‘ABCMeta’ has no len()’ is keep occuring.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/3/b3afdbbaae49ec7e50535c68f016bdf99179740d.png" data-download-href="https://discuss.pytorch.org/uploads/default/b3afdbbaae49ec7e50535c68f016bdf99179740d" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/3/b3afdbbaae49ec7e50535c68f016bdf99179740d_2_690x203.png" alt="image" data-base62-sha1="pDAeZVHQHhaULLSJMXHSBOG3D7T" width="690" height="203" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/3/b3afdbbaae49ec7e50535c68f016bdf99179740d_2_690x203.png, https://discuss.pytorch.org/uploads/default/original/3X/b/3/b3afdbbaae49ec7e50535c68f016bdf99179740d.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/b/3/b3afdbbaae49ec7e50535c68f016bdf99179740d.png 2x" data-dominant-color="0C0C0C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">870×256 37.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Below is the code which I implemented. I am trying to concat two datasets and train.</p>
<pre><code class="lang-auto">def prepare_data(args, ckpt_dir=None, with_input_orig=False, split=None):

    train_sets = []

    train_preprocessor_kwargs = {}
    pdb.set_trace()
    if 'cityscapes-with-depth' in args.dataset:
        Dataset = Cityscapes
        dataset_kwargs = {
            'n_classes': 19,
            'disparity_instead_of_depth': False
        }
        valid_set = 'valid'
        train_sets.append(Dataset)

    if 'sunrgbd' in args.dataset:
        Dataset = SUNRGBD
        dataset_kwargs = {}
        valid_set = 'test'
        train_sets.append(Dataset)

    if len(train_sets) != len(args.dataset):
        raise Exception('Something went wrong. Please check your dataset names are valid')
    
    if len(train_sets) == 0:
        raise Exception('Dataset {} is not supported'.format(args.dataset))

    if len(args.dataset) != 1:
        Datasets = ConcatDataset(train_sets)
</code></pre>
<p>What is the problem?</p>

nivek: <p>The initialization of <code>ConcatDataset</code> requires that each input dataset has <code>len()</code>. Can you check if <code>len()</code> can be applied to each of element within <code>train_sets</code>?</p>

ejguan: <p>You are passing class rather than instance to the list. You need to create Dataset instance from <code>Cityscapes</code>, <code>SUNRGBD</code>.</p>
Is this an error?

NagaYu: <pre><code class="lang-auto">model.train()
for epoch in range(num_epoch):
    loss_hist.reset()
    
    for i, (images, targets, ImageIDs) in enumerate(train_loader):

        images = list(image.to(device) for image in images)
        targets = [{k: v.to(device) for k, v in t.items()} for t in targets]

        loss_dict = model(images, targets)
        
        losses = sum(loss for loss in loss_dict.values())
        loss_value = losses.item()

        loss_hist.send(loss_value)

        optimizer.zero_grad()
        losses.backward()
        optimizer.step()

        if itr % 500 == 0:
            print(f"Iteration #{itr} loss: {loss_value}")

        itr += 1
    
    if lr_scheduler is not None:
        lr_scheduler.step()

    print(f"Epoch #{epoch} loss: {loss_hist.value}")   
</code></pre>
<p>Output exceeds the size limit. Open the full output data in a text editor</p>
<p>22/06/22 15:00:54 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation.</p>

ptrblck: <p>This warning:</p>
<pre><code class="lang-python">22/06/22 15:00:54 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation.
</code></pre>
<p>seems to be raised by Spark and is thus unrelated to PyTorch or are you concerned about another issue in your code?</p>

NagaYu: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="154756">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>seems to be raised by Spark and is thus unrelated to PyTorch or are you concerned about another issue in your code?</p>
</blockquote>
</aside>
<p>I’m worried that the code is learning properly.</p>

ptrblck: <p>I cannot see any obvious issues in the posted code.</p>

NagaYu: <p>Thank you. I was worried that my learning wouldn’t end at all.</p>
Select subset of Conv2d layer filters in the forward method

SofiaCP: <p>Hi,</p>
<p>Let’s assume I have a set of data instances of shape <code>[1,64,8,8]</code> that can belong to one of two groups (A or B), and a <code>Conv2d</code> layer with the arguments <code>(in_channels=64, out_channels=128, kernel_size=3)</code>.</p>
<p>Can I use a subset of the 128 filters of the <code>Conv2d</code>, depending on the group of the instance? For example, can I divide the <code>Conv2d</code> into two parts (64 filters + 64 filters) and use the first half if the instance belongs to group A and the second part if the instance belong to group B?</p>
<p>My idea would be to define a strandard <code>Conv2d</code> in the <code>init</code> method and then somehow select the subset of filters that I want to use in the forward. During the backward pass I am expecting that only the filters that were used for that instance get updated.</p>
<p>Thanks,<br>
Sofia</p>
3D signed distance field computation?

Peter_Mitrano: <p>I’m looking for a fast, batched, GPU-based implementation of computing a signed distance field given an occupancy grid in 3D. Does anyone know of any? If not, what algorithm would you suggest for implementing one?</p>

Henry_Chibueze: <p>When you say signed distance, do you mean distance from a point in a region to the boundaries of said region?<br>
Or do you mean distance between the boundaries of two separate regions?</p>

Peter_Mitrano: <p>I’m not really thinking of regions and boundaries, so I’m not sure how to answer. The input is a binary voxelgrid with values of 1 indicating occupied points and 0 everywhere else. I want each output cell to contain the distance to the nearest occupied point.</p>
<p>EX: <a href="https://prideout.net/blog/distance_fields/" rel="noopener nofollow ugc">https://prideout.net/blog/distance_fields/</a></p>
What does it mean to mark unused parameters as ready in DDP forward pass

Jianxin_Wang: <p>I read the paper <a href="https://arxiv.org/pdf/2006.15704.pdf" rel="noopener nofollow ugc">https://arxiv.org/pdf/2006.15704.pdf</a> about DDP. In the paper, there is a pseudo-code about the algorithm<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/8/08e603297af605fd614f8ec0c436e45dd049af88.png" data-download-href="https://discuss.pytorch.org/uploads/default/08e603297af605fd614f8ec0c436e45dd049af88" title="Screen Shot 2022-06-18 at 10.09.21 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/8/08e603297af605fd614f8ec0c436e45dd049af88_2_475x500.png" alt="Screen Shot 2022-06-18 at 10.09.21 PM" data-base62-sha1="1gICng91q78POBzUdKWbPVjEkSk" width="475" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/8/08e603297af605fd614f8ec0c436e45dd049af88_2_475x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/0/8/08e603297af605fd614f8ec0c436e45dd049af88_2_712x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/0/8/08e603297af605fd614f8ec0c436e45dd049af88_2_950x1000.png 2x" data-dominant-color="EAEAEA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2022-06-18 at 10.09.21 PM</span><span class="informations">1070×1124 90.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I wonder why ‘traverse autograd graph from out and mark unused<br>
parameters as ready’ in the forward pass. What does ‘unused parameters ready’ mean?</p>
<p>Thanks</p>

H-Huang: <p>Please see section 3.2.3 third paragraph of the paper. Explanation is bolded:</p>
<p>"<br>
Second, it is possible that one training iteration only involves a sub-graph in the model and the sub-graph can be different from iteration to iteration, meaning that some gradients might be skipped in some iterations. However, as gradient-to-bucket mapping is determined at the construction time, those absent gradients would leave some buckets never seeing the final autograd hook and failing to mark the bucket as ready. <strong>As a result, the backward pass could hang</strong>. Fig. 3 (b) shows an example, where the parameter corresponding to gradient g3 is skipped in one iteration, leading to the absent of the ready signal for g3. <strong>To address this problem, DDP traverses the autograd graph from the output</strong><br>
<strong>tensors of the forward pass to find all participating parameters. The readiness of those participating tensors is a sufficient signal to conclude the completion of the backward pass. Therefore, DDP can avoid waiting for the rest of the parameter gradients by proactively marking them ready at the end of the forward pass.</strong> Note that, this change does not prevent us from developing non-intrusive APIs, because application directly invokes the forward function on DDP and hence DDP can easily insert this step in its member function.<br>
"</p>
Parallel Multi-Task Training

Mayank_Jha1: <p>I’m trying to parallelize multi-task training in PyTorch Lightning(have different threads compute losses for each task). More precisely:</p>
<ul>
<li>Parallelizing the forward function of components that can be executed in parallel</li>
<li>Parallelizing the computation of multiple losses</li>
</ul>
<p>I want to know if PyTorch currently supports it or has anyone tried/implemented this before?</p>

H-Huang: <p>I’m not very familiar with lightning but in the PyTorch distributed package we don’t currently have a framework that is exactly for this paradigm. Distributed Data Parallel will parallelize the forward function and each loss is computed locally on each node, but this is for a replicated model and not a multi-task model as I believe you are asking for. A way to accomplish this on your own would be to use <a href="https://pytorch.org/docs/master/rpc.html?highlight=rpc" class="inline-onebox" rel="noopener nofollow ugc">Distributed RPC Framework — PyTorch master documentation</a> and handle the parallelization and tensor communication on your own via remote procedure calls</p>
PyTorch model converted to ONNX generates scatterNDupdate layers

Ashish_Papanai: <p>Hi,<br>
I converted a PyTorch model to ONNX to run inferences on it using OpenVINO on NCS2 (VPU). The converted model comprises scatterNDupdate layers which are not supported by the VPU. Is there any specific reason for the creation of those layers? and a way to avoid them?</p>
<p>From the PyTorch ONNX logs, I’ve found the code segment which is creating those layers, It will be of great help if I can be suggested a workaround to get rid of those.</p>
<pre><code class="lang-auto">           while 0 in out_length and now_step &lt; nsteps:
                tmp_result = C[now_step, :, :]

                out_res[now_step] = tmp_result
                tmp_result = tmp_result.topk(1)[1].squeeze(dim=1)
                for j in range(b):
                    if out_length[j] == 0 and tmp_result[j] == 0: #Check Here
                        out_length[j] = now_step + 1
                now_step += 1
            for j in range(0, b):
                if int(out_length[j]) == 0:
                    out_length[j] = nsteps #Check Here
            start = 0
            output = torch.zeros(int(out_length.sum()), self.nclass).type_as(input.data)
            for i in range(0, b):
                cur_length = int(out_length[i])
                output[start: start + cur_length] = out_res[0: cur_length, I, :] #Check Here
                start += cur_length

            return output, out_length
</code></pre>
Poor fitting of time series using LSTM

Fandier: <p>Guys, I am using LSTM for time series prediction. Where the input is 8 sets of time series and the output is a given time coefficient, but there have been no good training results. I have tried using Adjusted Loss, Minibatch and network structures etc. but have not had good results. I hope you can help me to see what the problem is.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/a/4/a4c9b846068e0ba58f76d126c033a3fdbff24d93.jpeg" alt="Test_compare" data-base62-sha1="nvMufnsvRS9jCGv0W3gImBjddE7" width="640" height="480"><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/0/f/0fcb3c4328de339b897a3350dca964c1f44a4122.jpeg" alt="Loss" data-base62-sha1="2fIAHezwCb6cT7gHEpVachp3cLU" width="640" height="480"></p>
Affine_grid / grid_sample for rotation in non-square inputs

mangoxb: <p>In case of non-square inputs, which theta is valid?<br>
I want to rotate around a center (x, y)<br>
then theta is<br>
[ [cosθ, -sinθ, x * sinθ + y * (1 - cosθ)],<br>
[ [sinθ, cosθ, x * (1-cosθ) - y * sinθ)] ]</p>
<p>But in case of height and width of input is not equal, it doesn’t works.<br>
what is valid theta ?</p>
Question about DataLoader and pin_memory_device

_Hyperion: <p>When looking at the source code on github, there is an option to specify on which device pin_memory is used. I need to specify which gpu my dataloader runs but when I try to add the pin_memory_device argument, I get</p>
<p>TypeError: DataLoader.<strong>init</strong>() got an unexpected keyword argument ‘pin_memory_device’</p>
<p>Any idea why I get this error?<br>
Thanks!</p>

nivek: <p>The parameter was added to the master branch on Apr 20, 2022 and will be part of release 1.12 (which is coming out in ~2 weeks). If you would like to use that parameter, please install the nightly version of PyTorch.</p>

_Hyperion: <p>Now I see why I get this error, thanks !</p>
By default, profiler.emit_nvtx is enabled or not?

songyuc: <p>Hi, guys,<br>
I am learning <code>torch</code> today, and find a function of <code>torch.autograd.profiler.emit_nvtx</code>, so, by default, <code>profiler.emit_nvtx</code> is enabled or not?</p>
<p>Your answer or guide will be appreciated!</p>
Difference between ReflectionPadding2d and padding_mode='reflect'

kandicst: <p>Hello, this is my first post here, so don’t be too harsh on me <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=9" title=":smile:" class="emoji" alt=":smile:"></p>
<p>I was just wondering if there is any difference between using nn.ReflectionPadding2d before Conv2d layer and just setting padding_mode of Conv2d to <em>‘reflect’</em>?</p>

ptrblck: <p>It should yield the same results as seen here:</p>
<pre><code class="lang-python">x = torch.randn(2, 3, 24, 24)
conv = nn.Conv2d(3, 6, 3)
pad = nn.ReflectionPad2d(1)

out = pad(x)
out = conv(out)

conv_pad = nn.Conv2d(3, 6, 3, 1, 1, padding_mode='reflect')
conv_pad.load_state_dict(conv.state_dict())

out_pad = conv_pad(x)

print((out - out_pad).abs().max())
&gt; tensor(0., grad_fn=&lt;MaxBackward1&gt;)
</code></pre>

glennres: <p>Hello, your posts are always so helpful! I’ve been using pytorch a lot over the past year and I find myself coming across your posts all the time. I totally want to thank you for all your helpful posts. Meta, give this man a raise! <img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=12" title=":grinning:" class="emoji" alt=":grinning:" loading="lazy" width="20" height="20"></p>
The details of PyTorch Linux binaries *.wkl

igreen: <p>What is the difference between cu90/torch-0.3.1-cp27-cp27mu-linux_x86_64.whl and<br>
cu90/torch-0.3.1-cp27-cp27m-linux_x86_64.whl. In details, what does ‘u’ in ‘cp27mu’ mean? And cp27 means ‘Python 2.7.x’?</p>

ptrblck: <p>As far as I know, the <code>cp27m(u)</code> defines the ABI-tag (application binary interface).<br>
While <code>cp27m</code> defines the usage of narrow-unicode, <code>cp27mu</code> uses wide-unicode.</p>
<p>The <code>cp27</code> is the python tag and defines the python version.</p>
<p>You can find more information on this in <a href="https://www.python.org/dev/peps/pep-0513/">PEP513</a> or the <a href="https://github.com/pypa/manylinux">pypa/manylinux repo</a>.</p>

abrahamowos: <p>So which one do you recommend that I use?<br>
I want to install PyTorch 0.3</p>

ptrblck: <aside class="quote no-group" data-username="abrahamowos" data-post="3" data-topic="17485">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/abrahamowos/48/50720_2.png" class="avatar"> abrahamowos:</div>
<blockquote>
<p>I want to install PyTorch 0.3</p>
</blockquote>
</aside>
<p>I would not recommend to use these ancient wheels, as neither Python2.x is supported nor would you be able to execute your workload on any new GPUs.</p>

abrahamowos: <p>Thank you. I’ll have to write the entire code base to use py37</p>
Is there any loss function that can measure the similarity between two sequences of different lengths

B.W_Zhang: <p>Is there any loss function that can measure the similarity between two sequences of different lengths?</p>
<p>For example, say <code>hidden1</code> (B, T1, F) and <code>hidden2</code> (B, T2, F) are encoded from <code>input1</code> and <code>input2</code>, respectively, by a transformer encoder.</p>
<p>Assume that <code>input1</code> and <code>input2</code> are two speakers saying the same sentence. Due to their different speaking speed, etc., T1 != T2, but since the contents are the same, the encoded <code>hidden1</code> and <code>hidden2</code> should thus be (somehow) similar.</p>
<p>I’m wondering whether there exists a loss in PyTorch that can measure such similarities (if the lengths are the same there’re many choices such as MSE, but when the lengths are different?)</p>
<p>Thanks in advance!</p>
Replace "for loop iteration" by Torch matrix multiplication

Hyeonuk_Woo: <p>Hi, I’m not good at using trick for matrix multiplication.</p>
<p>I want to get rid of “for loop iteration” by using Pytorch functions in my code. But the formula is complicated and I can’t find a clue. Can the “for loop iteration” in the below replaced with the Torch operation?</p>
<pre><code class="lang-auto">import torch ,sys
B=10
L=20
H=5
mat_A=torch.randn(B,L,L,H)
mat_B=torch.randn(L,B,B,H)
tmp_B=torch.zeros_like(mat_B)
for x in range(L):
   for y in range(B):
       for z in range(B):
           tmp_B[:,y,z,:]+=mat_B[x,y,z,:]*mat_A[z,x,:,:]
</code></pre>
<p>I’d like to know if there are any tips on how to solve this kind of  problems…</p>

KFrank: <p>Hi Hyeonuk!</p>
<aside class="quote no-group quote-modified" data-username="Hyeonuk_Woo" data-post="1" data-topic="154702" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hyeonuk_woo/48/39179_2.png" class="avatar"> Hyeonuk_Woo:</div>
<blockquote>
<p>Can the “for loop iteration” in the below replaced with the Torch operation?</p>
<pre><code class="lang-auto">for x in range(L):
   for y in range(B):
       for z in range(B):
           tmp_B[:,y,z,:]+=mat_B[x,y,z,:]*mat_A[z,x,:,:]
</code></pre>
<p>I’d like to know if there are any tips on how to solve this kind of  problems…</p>
</blockquote>
</aside>
<p>Yes, my tip is to use pytorch’s “Swiss-army knife” of tensor multiplication,<br>
<a href="https://pytorch.org/docs/1.11/generated/torch.einsum.html" rel="noopener nofollow ugc">einsum()</a>.</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.11.0'
&gt;&gt;&gt;
&gt;&gt;&gt; _ = torch.manual_seed (2022)
&gt;&gt;&gt;
&gt;&gt;&gt; B=10
&gt;&gt;&gt; L=20
&gt;&gt;&gt; H=5
&gt;&gt;&gt;
&gt;&gt;&gt; mat_A=torch.randn(B,L,L,H)
&gt;&gt;&gt; mat_B=torch.randn(L,B,B,H)
&gt;&gt;&gt; tmp_B=torch.zeros_like(mat_B)
&gt;&gt;&gt; for x in range(L):
...    for y in range(B):
...        for z in range(B):
...            tmp_B[:,y,z,:]+=mat_B[x,y,z,:]*mat_A[z,x,:,:]
...
&gt;&gt;&gt; C = torch.einsum ('xyzh,zxlh -&gt; lyzh', mat_B, mat_A)
&gt;&gt;&gt;
&gt;&gt;&gt; torch.allclose (C, tmp_B, atol = 1.e-5)
True
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

Hyeonuk_Woo: <p>Amazing, I have to practice for using einsum! Thanks</p>
Is my multi-label classification model overfitting?

se7en.dreams: <p>Hi, I am training a multi-label classifier. Looking at the loss and F1 curves, is my model overfitting?</p>
<p>I am using weighted BCE loss, along with dropout layers. What can I do to improve validation F1 and reduce validation loss, i.e., further prevent overfitting? Should I try <code>MultiLabelMarginLoss</code> for multi-label classification problem?</p>
<p>Appreciate any help on this.</p>
<p>(red is training, blue is validation)</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/8/5/85ce5bc97b4c9863c2c4f3bbc51ae4a32ed2c398.png" alt="loss_values" data-base62-sha1="j5HCJnIEhdDs8uzpPZfJ3ZKPJoI" width="360" height="249"></p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/c/acc893d11b53e49ced968669bf0fb4486c2f278c.png" alt="f1_scores" data-base62-sha1="oEvRjYrCa5gl56phMEE7kjsGMSE" width="360" height="249"></p>

azhanmohammed: <aside class="quote no-group" data-username="se7en.dreams" data-post="1" data-topic="154644">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/a4c791/48.png" class="avatar"> se7en.dreams:</div>
<blockquote>
<p>Looking at the loss and F1 curves, is my model overfitting?</p>
</blockquote>
</aside>
<p>The model is definitely over-fitting, the validation curve should follow the trend of training curve, but the loss in this case increases, while the F1 Score is almost 0.<br>
Can you please add the final layer of the model and the training script, along with the loss function being used to train the model. That might help in understanding how to reduce the validation loss and increase F1 score.</p>

se7en.dreams: <p>Thanks for your response. My multi-label classification model is as follows:</p>
<pre><code class="lang-auto">classifier = nn.Sequential(
    nn.Conv1d(128, 128, 1),
    nn.ReLU(),
    nn.BatchNorm1d(128),
    nn.Dropout(p=0.2),
    nn.Conv1d(128, 128, 1),
    nn.ReLU(),
    nn.BatchNorm1d(128),
    nn.Dropout(p=0.2),
    nn.Conv1d(128, 1, 1)
)
</code></pre>
<p>I’m making use of <code>BCEWithLogitsLoss</code> and I describe how I’m using it in this post: <a href="https://discuss.pytorch.org/t/loss-for-multi-label-classifier/154149" class="inline-onebox">Loss for Multi-label Classifier</a></p>
<p>Please let me know how I can improve it.</p>
How to get torch 0.3.0 in google colab?

princeofpython: <p>I need to use 0.3.0 version of torch in google colab. But when I try to install with pip, it says it isn’t a supported wheel in the platform. I need to replicate results of an old repository which uses the same version and it’s difficult to change the code to newer version. Can you please help? Also, torch 0.3.0 requires CUDA 9.0, so I need to install that too.</p>
<p>Link of the repo : <a href="https://github.com/alirezazareian/gbnet" class="inline-onebox" rel="noopener nofollow ugc">GitHub - alirezazareian/gbnet: Bridging Knowledge Graphs to Generate Scene Graphs, ECCV 2020</a></p>

abrahamowos: <p>Hey, did you later find a solution?</p>
Processing coordinates

OuisYasser: <p>Hello community,</p>
<p>Please, I am thinking about applying some idea but my input is considered to be X.Y coordinates and I am needed to extract a vector representing this input X Y</p>
<p>Mainly, I do have a trajectory on 2D space but I don’t have an idea about how to exploit it.</p>
<p>If anyone has an idea or have seen some papers on this problem please help me out</p>
<p>Thank you</p>

Andrei_Cristea: <p>Hello, Yasser! Could you be a little more specific about what exactly you’re trying to calculate. In fact, it would be best if you could give a small example with actual data, so that we can understand what you need.</p>

OuisYasser: <p>I do have a path that a pen followed in order to write some thing. Lets take the example of writing the letter A. So now my goal is to take the letter coordinates X and Y (let’s say a vector of size (2,200) ) and pass them into an ANN for example in order to have as an output one vector representing the probability of each word being written. (This is just an example to understand)<br>
My problem is that I dont know how to do it (for example represent the coordinates in another space) or which architecture I should use.</p>

Andrei_Cristea: <p>In the example given, at a very high level, a few ways I could imagine approaching this would be:</p>
<ul>
<li>Use the (X, Y) path to literally draw the picture as a data processing step, then train a standard image classification model (look up ResNet as an example) that can label the resulting image. This is slow since you’re doing some not-strictly-necessary preprocessing work, but I think it will get the job done.</li>
<li>If doing this would make the picture very muddled (e.g. say you have touchpad information, so the (X, Y) from different timesteps overlap, and the order really matters) perhaps you can think of a (X, Y, T) input into a series of Conv3d layers. You may have to experiment to find an architecture that works here, perhaps if you search around you can find someone who has done this already.</li>
<li>Feed the coordinate vector directly into one or more RNN layers (e.g. LSTM) with two input features, and train it to recognize the output you’re looking for.</li>
</ul>
<p>The right choice probably depends on the specifics of the problem.</p>
How to choose between clip_grad_norm and BatchNorm2d

laro: <ul>
<li>
<code>clip_grad_norm_</code> performs gradient clipping, in order to mitigate the problem of exploding gradients.</li>
<li>
<code>BatchNorm2d</code> applies Batch Normalization (for the same reason - mitigate the problem of exploding gradients)</li>
</ul>
<p>When we should choose <code>clip_grad_norm_</code> and when we should prefer <code>BatchNorm2d</code>?</p>

Andrei_Cristea: <p>I think in practice you will find that:</p>
<ul>
<li>
<p>BatchNorm2d is often used in ConvNet layers (e.g. in a standard ResNet block, as seen <a href="https://towardsdatascience.com/residual-blocks-building-blocks-of-resnet-fd90ca15d6ec" rel="noopener nofollow ugc">here</a> as BN):</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/a/fa03be9b3d3752a5bc1c306c4bd1b96623e3b171.png" alt="image" data-base62-sha1="zFJocgkCAlh9J8ccGsXcq12iyVH" width="135" height="321"></p>
</li>
<li>
<p>gradient clipping is often used in RNN models (such as LSTM) because the deep recurring structure can cause gradients to blow up</p>
</li>
</ul>
<p>I don’t think these are mutually exclusive. For example, in an OCR model built as a ConvNet stacked on top of a RNN, you may use BatchNorm2d in the ConvNet and gradient clipping for the sake of the RNN.</p>
<p>Hope this helps! (and curious if others have diverging opinions)</p>
Output of Conv1d to LSTM Input

Mert_Arda_Asar: <p>I am trying to use Conv1d and LSTM layers together. Output of conv1d layer is [8, 32, 10] which is form of Batch x Channel x Seq. Len. I can not give this output to LSTM layer directly. When I use permute function and replace sequence length with channel, training process works correctly. But is it the correct way to map conv output to sequence?</p>
How to do scalar operation on grad of a model

hzhangxyz: <p>Hello, I want to do some scalar operation on gradient of model, such as plus gradient of two point/multiplied a gradient with a float number. How to do it?</p>
<p>I am not optimization the model, so I do not use optimizer.</p>
<p>What I want to do is something like the following</p>
<pre><code class="lang-auto">model(x1).backward()
g1 = [i.grad for i in model.parameters()]
model(x2).backward()
g2 = ...
...
result = g1 * 4.542 + g2 * 5.3848 + ...
</code></pre>
<p>Is there any easy way to pack the gradient of a model as a whole to do scalar operation? Or i have to wrap it manually by myself?.</p>
<p>I want to keep the line</p>
<pre><code class="lang-auto">result = ...
</code></pre>
<p>simple, because this code is also used in other things beside this pytorch model.</p>
Use libtorch on ubuntu16.04 32 bist

BBniccccccc: <p>Hello,<br>
I would like to know if it is possible de use the lib libtorch for C++ on Ubuntu16.04 32 bits.</p>
<p>Thanks for answering <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

ptrblck: <p>I’m not 100% sure (as I haven’t tested it) but would assume that the pre-built <code>libtorch</code> binaries are not compatible with 32bit systems. You might try to build <code>libtorch</code> from source but I also don’t know if this would succeed.</p>

BBniccccccc: <p>Ok thank you, but looks like you can only build from source the python version with this <a href="https://github.com/pytorch/pytorch#from-source" class="inline-onebox" rel="noopener nofollow ugc">GitHub - pytorch/pytorch: Tensors and Dynamic neural networks in Python with strong GPU acceleration</a><br>
Am i looking in the wrong website ?</p>

ptrblck: <p>These instructions would build the full framework. If you want to build <code>libtorch</code> only, you could check <a href="https://github.com/pytorch/pytorch/blob/master/docs/libtorch.rst">these docs</a>.</p>
Trying to run CenterNet on AMD GPU

Praj_G: <p>I am trying the implement this paper <a href="https://github.com/xingyizhou/CenterNet" class="inline-onebox" rel="noopener nofollow ugc">GitHub - xingyizhou/CenterNet: Object detection, 3D detection, and pose estimation using center point detection:</a></p>
<p>In one of the steps to install it, I am getting an error -</p>
<pre><code class="lang-auto">
Traceback (most recent call last):

File "[build.py](https://build.py/)", line 21, in &lt;module&gt;

raise ValueError('CUDA is not available')

ValueError: CUDA is not available

Traceback (most recent call last):

File "build_double.py", line 21, in &lt;module&gt;

raise ValueError('CUDA is not available')

ValueError: CUDA is not available

</code></pre>
<p>I have installed the dependencies as stated.</p>
<p>I have an AMD GPU and I read that CUDA works only on NVIDIA GPUs.</p>
<p>Can I run this somehow and is there a way to solve this?</p>
<p>This is the code for build.py-</p>
<pre><code class="lang-auto">import os
import torch
from torch.utils.ffi import create_extension

torch.load(map_location=torch.device("cpu"))

sources = ['src/dcn_v2.c']
headers = ['src/dcn_v2.h']
defines = []
with_cuda = False

extra_objects = []
if torch.cuda.is_available():
    print('Including CUDA code.')
    sources += ['src/dcn_v2_cuda.c']
    headers += ['src/dcn_v2_cuda.h']
    defines += [('WITH_CUDA', None)]
    extra_objects += ['src/cuda/dcn_v2_im2col_cuda.cu.o']
    extra_objects += ['src/cuda/dcn_v2_psroi_pooling_cuda.cu.o']
    with_cuda = True
else:
    raise ValueError('CUDA is not available')

extra_compile_args = ['-fopenmp', '-std=c99']

this_file = os.path.dirname(os.path.realpath(__file__))
print(this_file)
sources = [os.path.join(this_file, fname) for fname in sources]
headers = [os.path.join(this_file, fname) for fname in headers]
extra_objects = [os.path.join(this_file, fname) for fname in extra_objects]

ffi = create_extension(
    '_ext.dcn_v2',
    headers=headers,
    sources=sources,
    define_macros=defines,
    relative_to=__file__,
    with_cuda=with_cuda,
    extra_objects=extra_objects,
    extra_compile_args=extra_compile_args
)

if __name__ == '__main__':
    ffi.build()
</code></pre>
Generate a dictionary composed of random strings with random lengths from random characters of the supported character set

Aarya: <p>Hi, I need to write a python script that:</p>
<ol>
<li>Get a random length for the string to generate</li>
<li>Generate a string of this length, using random characters from [the supported/valid character set]</li>
<li>Write the generated strings one per line in a text file (length &gt;=2 million) and save this file to disk.</li>
</ol>
<p>Following is the supported/valid character set:<br>
<strong><code>0123456789abcdefghijklmnopqrstuvwxyzäöüßABCDEFGHIJKLMNOPQRSTUVWXYZÄÖÜ!\"#$%&amp;'()*+,-./:;&lt;=&gt;?@[\\]_~€§£¥</code></strong></p>
<p>Any kind of help would be appreciated.</p>

ptrblck: <p>This question doesn’t seem to be PyTorch-specific, so you might get a faster and better answer e.g. on StackOverflow.<br>
In any case, something like this would work:</p>
<pre><code class="lang-python">chars = [c for c in '0123456789abcdefghijklmnopqrstuvwxyzäöüßABCDEFGHIJKLMNOPQRSTUVWXYZÄÖÜ!\"#$%&amp;\'()*+,-./:;&lt;=&gt;?@[\\]_~€§£¥']

length = random.randint(1, 10)
s = []
for _ in range(length):
    c = random.choice(chars)
    print(c)
    s.append(c)

s = "".join(s)
print(s)
# oPHwDR&amp;!N
</code></pre>
<p>Note that this code is slow and is showing the explicit steps.<br>
You could easily speed it up e.g. by indexing the valid characters and create the string directly instead of using the loop etc.</p>

Aarya: <p>Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a> . It works.</p>
How to change config file to start fine tuning?

Aarya: <p>Hello,</p>
<p>I am working on word detection and using MMOCR’s ‘<strong>DBNetpp</strong>’ detection model, please refer <a href="https://arxiv.org/abs/2202.10304" class="inline-onebox" rel="noopener nofollow ugc">[2202.10304] Real-Time Scene Text Detection with Differentiable Binarization and Adaptive Scale Fusion</a>.</p>
<p>I want to use this pre-trained model and fine tune on my dataset.<br>
Therefore, I need to update the configuration file for ‘<strong>DBNetpp</strong>’.<br>
here is the part of config file. I would like to know which parameters (specifically the learning rates) can be changed to have descent first results (decrease in loss).</p>
<pre><code class="lang-auto"># optimizer

optimizer = dict(type="SGD", lr=0.007, momentum=0.9, weight_decay=0.0001)

optimizer_config = dict(grad_clip=None)

# learning policy

lr_config = dict(policy="poly", power=0.9, min_lr=1e-7, by_epoch=False)
</code></pre>
<p>Any help would be appreciated.<br>
Thank you in advance.</p>
Compute the loss of a moving dataset

Zac_Boussaid: <p>Hi guys,<br>
I’m new to the world of machine learning, so it could be that my question is trivial or incorrectly posed.</p>
<p>I am using a moving dataset that I have forwarded to an STN network (Spatial Tranformation Network). To the STN I forward each image individually, then restack the whole images together in a tuple.<br>
My problem lies in the loss calculation. My target has a torch.tensor with this size [2,1,64,64]<br>
and my prediction that I want to implement has a torch.tensor [2,1,10,64,64], which means that the prediction and the target are not the same.<br>
Could someone explain an idea to me. The only idea I have is to return the last STN output meaning something like this [2,1,1,64,64] to my prediction and then squeezed to be [2,1,64,64] and then calculate the loss.</p>
<p>Thank you in advance</p>
RNN to predict label from sequence in pytorch

ba_torch: <p>Each training example is a sequence of how 8 input variables vary across 5 timesteps</p>
<p>I.e Input is <code>[ip0_0, ip1_0,...,ip4_0], [ip0_1, ip1_1,...,ip4_1]..., [ip0_1, ip1_1,...,ip4_1]</code></p>
<p>Each training example has a label 0 or 1</p>
<p>I want to create a RNN that predicts the label from the inputs.</p>
<p>I see two ways of doing it</p>
<p>See <code>RNNModelMultiforward</code> below.  High level idea is</p>
<ol>
<li>Have a single <code>torch.RNN()</code>
</li>
<li>Initialize hidden state to 0</li>
<li>Run the following 5 times<br>
<code>out, h = RNN([ip0_i,...,ip4_i], h)</code>, where <code>i = 0,...,4</code>
</li>
<li>Run a feedforward layer that predicts the label from the final hidden state <code>h</code>
</li>
</ol>
<p>Is this the right way to do it or should I use a <code>torch.RNN</code> with <code>num_layers = 5</code> and run it once to get the output: I.e <code>hn = RNN([[ip0_0,...,ip4_0],.....,[ip0_4,...,ip4_4]], h0)</code> (see <code>RNNModelMultilayer</code> below)</p>
<p><code>RNNModel multiforward</code></p>
<pre><code class="lang-auto"># Create RNN Model
class RNNModelMultiforward(nn.Module):
    def __init__(self, input_dim, hidden_dim, layer_dim, output_dim, num_epochs, act_fn='relu'):
        super(RNNModelMultiforward, self).__init__()
        
        # Number of hidden dimensions
        self.hidden_dim = hidden_dim
        
        # Number of hidden layers
        self.layer_dim = layer_dim

        #Number of times the RNN will be run (the input should be num_epochs X input_dim in size)
        self.num_epochs = num_epochs
        
        # RNN
        self.rnn = nn.RNN(input_dim, hidden_dim, layer_dim, batch_first=True, nonlinearity=act_fn)
        
        # Readout layer
        self.fc = nn.Linear(hidden_dim, output_dim)
    
    def forward(self, x):
        # Initialize hidden state with zeros
        h = Variable(torch.zeros(self.layer_dim, self.hidden_dim))
    
        for ts in range(0, self.num_epochs):
            out, h = self.rnn(x[ts], h)
        out = self.fc(h) 
        return out
</code></pre>
<p><code>RNNModel multilayer</code></p>
<pre><code class="lang-auto"># Create RNN Model
class RNNModelMultilayer(nn.Module):
    def __init__(self, input_dim, hidden_dim, layer_dim, output_dim, act_fn='relu'):
        super(RNNModelMultilayer, self).__init__()
        
        # Number of hidden dimensions
        self.hidden_dim = hidden_dim
        
        # Number of hidden layers
        self.layer_dim = layer_dim
        
        # RNN
        self.rnn = nn.RNN(input_dim, hidden_dim, layer_dim, batch_first=True, nonlinearity=act_fn)
        
        # Readout layer
        self.fc = nn.Linear(hidden_dim, output_dim)
    
    def forward(self, x):
        # Initialize hidden state with zeros
        h0 = Variable(torch.zeros(self.layer_dim, self.hidden_dim))
    
        out, hn = self.rnn(x, h0)
        out = self.fc(hn[4]) 
        return out
</code></pre>
Torch.load cause CPU memory increase when model eval

sdu_gz: <p>Hi all, I find a question that the CPU memory increase shapely  when I use torch.load() for model eval. I use a GPU so the map_location was set as “cuda : 0”. I wonder why the CPU memory cost so much. Thanks !</p>

Michal_Bogacz: <p>Try doing this:</p>
<h1><a name="load-all-tensors-onto-gpu-1-1" class="anchor" href="#load-all-tensors-onto-gpu-1-1"></a>Load all tensors onto GPU 1</h1>
<blockquote>
<blockquote>
<blockquote>
<p>torch.load(‘tensors.pt’, map_location=lambda storage, loc: storage.cuda(1))</p>
</blockquote>
</blockquote>
</blockquote>
Loss in multiple outputs network

muttistefano: <p>Hi all, i have a network composed by a RNN layer , followed by 3 separated fully connected portions that output each a single value:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/4/7/47fc718f7bc8e278993986c99dc0ad5afa00bbcb.png" alt="Untitled Diagram.drawio" data-base62-sha1="agOL0WLUCKeMjtNtb1fuS4PkPyP" width="261" height="321"></p>
<p>The net is defined by :</p>
<pre><code class="lang-auto">class RNN(nn.Module):
    def __init__(self, hidden_size, num_layers,fd_n):
        super(RNN, self).__init__()
        self.hidden_size = hidden_size
        self.num_layers = num_layers
        self.rnn = nn.RNN(input_size=510,hidden_size=hidden_size,num_layers=num_layers,batch_first=True,dropout=0.0)
        self.fcx   = nn.Linear(in_features=hidden_size,out_features=fd_n)
        self.fcx2  = nn.Linear(in_features=fd_n,out_features=1)
        self.fcy   = nn.Linear(in_features=hidden_size,out_features=fd_n)
        self.fcy2  = nn.Linear(in_features=fd_n,out_features=1)
        self.fcw   = nn.Linear(in_features=hidden_size,out_features=fd_n)
        self.fcw2  = nn.Linear(in_features=fd_n,out_features=1)

    def forward(self,x):
        h0 = torch.zeros(self.num_layers, x.size(0), self.hidden_size).to(device) 
        out_rnn, _ = self.rnn(x,h0)
        outx     = self.fcx(F.relu(out_rnn[:, -1, :]))
        outx     = self.fcx2(F.relu(outx))
        outy     = self.fcy(F.relu(out_rnn[:, -1, :]))
        outy     = self.fcy2(F.relu(outy))
        outw     = self.fcw(F.relu(out_rnn[:, -1, :]))
        outw     = self.fcw2(F.relu(outw))

        return outx,outy,outw
</code></pre>
<p>while the training part is:</p>
<pre><code class="lang-auto">        outputs_x, outputs_y, outputs_w  = model(inputs)
        loss_x = criterion_x(outputs_x, labels)
        loss_y = criterion_y(outputs_y, labels)
        loss_w = criterion_w(outputs_w, labels)

        loss = loss_x + loss_y + loss_w

        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
</code></pre>
<p>At the beginning I always get :</p>
<pre><code class="lang-auto">UserWarning: Using a target size (torch.Size([1520, 3])) that is different to the input size (torch.Size([1520, 1])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.
</code></pre>
<p>am I correctly backpropagating ?<br>
Thanks</p>

ptrblck: <p>I would not ignore the warning as broadcasting might be used which could yield invalid loss values.<br>
The warning is raised since the <code>criterion_x/y/w</code> calls are using a target with 3 features while each model output has 1 feature/logit only. Assuming the target contains the values for all 3 model outputs, you could <code>torch.cat((outputs_x, outputs_y, outputs_w), dim=1)</code> the outputs and calculate the loss afterwards.</p>

muttistefano: <p>Seems like the problem is solved.<br>
thanks <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
For best utilization of GPU memory, should checkpoint also be applied to the nested child modules?

songyuc: <p>Hi, guys,<br>
I want to use <code>checkpoint</code> to reduce the GPU memory usage, and I have known that, checkpoint can be applied like,</p>
<pre><code class="lang-python">def forward(self, x): 
 def _inner_forward(x): 
        identity = x 
        out = self.conv1(x) 
        out = self.norm1(out) 
        out = self.relu(out) 
        out = self.conv2(out) 
        out = self.norm2(out) 
        if self.downsample is not None: 
            identity = self.downsample(x) 
        out += identity 
        return out 
 
    if self.with_cp and x.requires_grad: 
        out = cp.checkpoint(_inner_forward, x) 
    else: 
        out = _inner_forward(x) 
    out = self.relu(out) 
    return out
</code></pre>
<p>So, if there is a big module has a nested custom module like,</p>
<pre><code class="lang-auto">class Net(nn.Module):
 def forward(self, x): 
  def _inner_forward(x): 
         identity = x 
         out = self.ConvBnRelu1(x) 
         out = self.ConvBnRelu2(x)  # instance of ConvBnRelu
         if self.downsample is not None: 
             identity = self.downsample(x) 
         out += identity 
         return out 
 
     if self.with_cp and x.requires_grad: 
         out = cp.checkpoint(_inner_forward, x) 
     else: 
         out = _inner_forward(x) 
     out = self.relu(out) 
     return out
</code></pre>
<p>For best utilization of GPU memory, should checkpoint also be applied to the nested child modules of <code>ConvBnRelu</code>?</p>
<p>Your answer and guide will be appreciated!</p>
Einsum is not fast?

coincheung: <p>I test it like this:</p>
<pre><code class="lang-auto">import torch
n_times = 10000
bs = 20000
i, j, k = 3000, 40000, 50000
a = torch.rand(2000, 3, 4).double()
b = torch.rand(2000, 4, 5).double()
c = torch.rand(2000, 5, 6).double()

t1 = time.time()
for _ in range(n_times):
    r1 = (a @ b) @ c
torch.cuda.synchronize()

t2 = time.time()
for _ in range(n_times):
    r2 = a @ (b @ c)
torch.cuda.synchronize()

t3 = time.time()
for _ in range(n_times):
    r3 = torch.einsum('abc,acd,ade-&gt;abe', a, b, c)
torch.cuda.synchronize()

t4 = time.time()

print((r1 - r2).abs().max())
print((r2 - r3).abs().max())


print((t2 - t1))
print((t3 - t2))
print((t4 - t3))

</code></pre>
<p>The time of the 3 are 6s, 1s, 2s.</p>
<p>It seems that torch.einsum does not choose the best computing method according to the input. Is this the expected behavior?</p>

ptrblck: <p>Note that the first timing is wrong as you are also accumulating the tensor creation due to the lack of synchronization.<br>
I get quite similar times for all workloads, but <code>einsum</code> might indeed add a small overhead:</p>
<pre><code class="lang-python">0.47559189796447754
0.4516446590423584
0.4858055114746094
</code></pre>
Mask in Attention: scalar vs torch.tensor vs torch.zeros

etc: <p>Hello all,</p>
<p>I implemented a Transformer model and I want to use masking in the score table.</p>
<p>I implemented the masking with 3 different ways that represent exactly the same masking and I have 3 different results.</p>
<p>The score table is of dimension (time_len * joint_num):<br>
<strong>Way 1:</strong><br>
t_mask = torch.ones(time_len * joint_num, time_len * joint_num)<br>
filtered_area = torch.zeros(joint_num, joint_num)</p>
<pre><code>    for i in range(time_len):
        row_begin = i * joint_num
        column_begin = row_begin
        row_num = joint_num
        column_num = row_num

        t_mask[row_begin: row_begin + row_num, column_begin: column_begin + column_num] *= filtered_area
</code></pre>
<p><strong>Way 2</strong><br>
t_mask = torch.ones(time_len * joint_num, time_len * joint_num)</p>
<pre><code>    for i in range(time_len):
        row_begin = i * joint_num
        column_begin = row_begin
        row_num = joint_num
        column_num = row_num

        t_mask[row_begin: row_begin + row_num, column_begin: column_begin + column_num] *= 0.0
</code></pre>
<p><strong>Way 3</strong><br>
t_mask = torch.ones(time_len * joint_num, time_len * joint_num)</p>
<pre><code>    for i in range(time_len):
        row_begin = i * joint_num
        column_begin = row_begin
        row_num = joint_num
        column_num = row_num

        t_mask[row_begin: row_begin + row_num, column_begin: column_begin + column_num] *= torch.tensor(0.0)
</code></pre>
<p><strong>The Accuracy of Way1 is better compared to the Accuracy of Way2 and 3</strong><br>
Why does it happen ?<br>
torch version : 1.7.0</p>

InnovArul: <p>As far as I could see, I do not find a particular differentiating factor in way-1 that would make it perform better than way-2 &amp; way-3.<br>
What’s the performance difference that you are observing?</p>

etc: <p>Me neither ! It does not makes sense! I use the same seed every time !<br>
The difference is around 2%.</p>

InnovArul: <p>ok. 2% might be a pretty big difference depending on the task.<br>
Do you think it could be due to the difference in weight initialization?</p>