akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Finding indexes of an array with different length

username: Adrian_Briceno_Aguil
cooked: <p>I have 3 tensors with:</p>
<pre><code class="lang-auto">x.shape = (2500,2) # in this case the each value in this array are unique(like a grid)
y.shape = (2500,1)

</code></pre>
<p>The last tensor is a subset of x, let’s call it z, and has a  shape (650,2), I would like to find the indexes where x = z. For me to use the indexes in tensor “y” and slice it to find the corresponding values.</p>
<p>I decided to do this problem as I did it with only one dimension I used <code>torch.where()</code>. However, I found different problems with this way of using it.</p>
<p>First of all, I can’t use a <code>torch.where(x==z)</code>, since it has different lengths.</p>
<p>Second of all, I believe this function isn’t pairwise. Meaning that it can’t check each of the pairs of numbers for each of them. (This might not be true).</p>
<p>I created a way that finds this indexes:</p>
<pre><code class="lang-auto">def findIndexes(domain,quadrants):
    noleaf = torch.tensor([])
    for j in range(0,quadrants[i].shape[0]):
      index = torch.where((domain[:,0] == quadrants[j,0]) &amp; (domain[:,1] == quadrants[j,1]))[0]
      noleaf = torch.cat((noleaf,index))
    
  return noleaf
</code></pre>
<p>I was wondering if this way could be too inefficient, in terms of memory. I wanted to pick your brains to see if there is a more sophisticated way to do it. If so how could I do it?</p>
Batch learning on large datasets

username: jagraves21
cooked: <p>I am trying to do batch learning on a large dataset that will not fit on the GPU.  I am not sure where to clear the gradients and compute the loss.  Is this the correct way to use a <code>DataLoader</code> and move the data to the GPU in pieces for batch learning?</p>
<pre><code>train_dataset = torch.utils.data.TensorDataset(X_train, y_train)
train_dataloader = torch.utils.data.DataLoader(train_dataset, batch_size, shuffle=True)

for epoch in range(n_epochs):
    for loc_X_train,loc_y_train in train_dataloader:
        # move data to the device
        loc_X_train, loc_y_train = loc_X_train.cuda(), loc_y_train.cuda()

        # clear any calculated gradients
        optimizer.zero_grad()

        # forward pass, compute outputs
        outputs = model.forward(loc_X_train)

        # compute loss
        loss = loss_function(outputs, loc_y_train)

        # backward pass, compute gradients
        loss.backward()

    # update learnable parameters
    optimizer.step()</code></pre>

username: ptrblck
cooked: <p>The <code>DataLoader</code> loop looks alright.<br>
You shouldn’t call <code>model.forward</code>, but the model directly via <code>outputs = model(loc_X_train)</code> so that registered hooks will be properly called.<br>
Also, usually you would update the parameters after the <code>backward</code> call in each iteration not once per epoch, so you might want to call <code>optimizer.step()</code> inside the <code>DataLoader</code> loop.</p>

username: jagraves21
cooked: <p>In batch learning, the weights are only updated once per epoch, so the <code>optimizer.step()</code> call needs to be at the end of the outer loop.  I’m starting to think that the <code>optimizer.zero_grad()</code> call should be just before the inner loop.</p>

username: ptrblck
cooked: <p>Yes, in that case you shouldn’t zero out the gradients in each iteration and note that each <code>backward()</code> call would accumulate the gradients. In batch training the gradients are usually calculated using the mean, so you might also need to scale the gradients before applying the <code>optimizer.step()</code> operation.</p>

username: jagraves21
cooked: <p>What’s the best way to do this?  If I put the following between the loss calculation and backward step, would it take care of it?</p>
<pre><code>if loss_function.reduction != "sum":
    loss *= len(outputs)</code></pre>

username: ptrblck
cooked: <p>No, this would increase the loss even further.<br>
Often you are scaling the loss via:</p>
<pre><code class="lang-python">loss = loss / accumulation_steps
</code></pre>
<p>in each iteration.</p>
FCN with patches creates boundary

username: Remote_Senser
cooked: <p>Hello,</p>
<p>I am trying to train a Unet model to do per pixel regression predictions on images. To do this, I separate my large image (1000x1000) to 200x200 pixel squares. Then use that to train an FCN model with a linear final layer. The loss function is MSE loss. In the prediction stage, I extract the same boxes but stitch it together and obtain a final output image. When I do that, the problem I am getting is that there is discontinuities between the boundary of boxes. (I can clearly see the boxes)</p>
<p>I’ve tried to deal with this by feeding 250x250 boxes to my FCN and calculating the loss for the 200x200 centre region. I do the same process for the prediction state. Extract 250x250 patches crop the 200x200 centre region and stitch the image back together. Please see some code below:</p>
<p>Loss Function:</p>
<pre><code class="lang-auto">criterion = nn.MSELoss()
optimizer = optim.Adam(self.model.parameters(), lr=LR)
for inputs, labels in train_loader:
    inputs, labels = inputs.to(device), labels.to(device)
    optimizer.zero_grad()
    output = model(inputs)
    output = output.squeeze()
     _, dimx, dimy = output.shape
     loss = criterion(output[:,25:dimx-25, 25:dimy-25], labels[:,25:dimx-25, 25:dimy-25])
     loss.backward()
     optimizer.step()
</code></pre>
<p>My code for predictions is as follows:</p>
<pre><code class="lang-auto">pred = np.zeros((height, width))
for i in range(25, height, 200):
    for j in range(25, width, 200):
        patch = img[:, i-25:i+225, j-25:j+225]
        patch = torch.from_numpy(patch)
        patch = patch.unsqueeze(dim=0).to(device)
        out = model(patch)
        out = out[0,0,25:225, 25:225]
        pred[i:i+200, j:j+200] = out.cpu().numpy()

</code></pre>
<p>I’m not sure if my problem makes complete sense. I can provide more clarification if necessary but I have been stuck on this for a while now.</p>

username: ptrblck
cooked: <p>The original <a href="https://arxiv.org/abs/1505.04597" rel="nofollow noopener">UNet paper</a> uses a similar strategy, i.e. a sliding window approach where only the center is predicted and mirrored pixel values on the border, if I’m not mistaken.</p>
<p>Did you see any improvement if you predicted the center of the sliding window compared to the complete patch?</p>

username: Remote_Senser
cooked: <p>I actually solved this problem. My fault for not showing my FCN model code. The problem was that I had batch normalization between the convolution layers. This created discontinuities between the patches when stitching the images back together.</p>

username: joemcglinchy
cooked: <p>Hi <a class="mention" href="/u/remote_senser">@Remote_Senser</a>, do you have your FCN model code available? We are looking at using FCN (UNet perhaps) for pixel-level continuous surface prediction, but not many examples out there as most FCN applications are for the classification problem as opposed to the regression problem.</p>
<p>If not, what is the general approach of modifying a FCN built for pixel-level classification, to be used for pixel-level regression?</p>
SparseTensorImpl vs TensorImpl

username: almeetb
cooked: <p>Hello,</p>
<p>Is it necessary to have a SparseTensorImpl for operations that work on sparse tensors and are all ops that are mean to work on Sparse tensors labelled with <code>sparse</code> in <code>RegistrationDeclarations.h</code></p>
<p>for e.g. does <code>_sparse_addmm</code> take in sparse tensors and return sparse tensors?</p>

username: glaringlee
cooked: <p>Hi, I think you need to  check the return type and input parameter type to figure out.<br>
eg. _sparse_addmm<br>
‘<a href="https://github.com/pytorch/pytorch/blob/dc7cd97402791945648372e0b08961b00a4f5d26/aten/src/ATen/native/sparse/SparseTensorMath.cpp#L930" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/blob/dc7cd97402791945648372e0b08961b00a4f5d26/aten/src/ATen/native/sparse/SparseTensorMath.cpp#L930</a>’<br>
The output of this function is a strided tensor.</p>
Transfer Learning: Batch Normalisation: To Freeze or not to Freeze

username: bing
cooked: <p>Hi Everyone,<br>
I am doing a multi-image classification problem using tranfer learning.<br>
Does batch normalisation layers gets frozen or not(Either of the last layer or the previous layers)?<br>
According to me, they should not be frozen of the layer, of which the weights and bias parameters I am unfreezing?</p>
Pytorch issue--why embeddings throwing me an index error in the middle of training?

username: Sumanta_Bhattachary1
cooked: <p>/pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [192,0,0], thread: [127,0,0] Assertion <code>srcIndex &lt; srcSelectDimSize</code> failed.<br>
0<br>
TITAN RTX<br>
Memory Usage:<br>
Allocated: 0.0 GB<br>
Cached:    0.0 GB<br>
Traceback (most recent call last):<br>
File “/users/sbhatta9/emnlp_code/train_ebrwt.py”, line 331, in <br>
train(args)<br>
File “/users/sbhatta9/emnlp_code/train_ebrwt.py”, line 317, in train<br>
valid_bleu = run_eval_bleu(epoch, (rebatch(b, field_names) for b in valid_iter), model_par, field_names)<br>
File “/users/sbhatta9/emnlp_code/train_ebrwt.py”, line 187, in run_eval_bleu<br>
hypo_scores = torch.stack([model(getattr(batch, field).cuda()) for field in field_names[1:1 + sample_count]]).view(sample_count, -1).t()<br>
File “/users/sbhatta9/emnlp_code/train_ebrwt.py”, line 187, in <br>
hypo_scores = torch.stack([model(getattr(batch, field).cuda()) for field in field_names[1:1 + sample_count]]).view(sample_count, -1).t()<br>
File “/users/sbhatta9/sumanta/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 722, in _call_impl<br>
result = self.forward(*input, **kwargs)<br>
File “/users/sbhatta9/emnlp_code/train_ebrwt.py”, line 71, in forward<br>
_, sample_hidden = self.bert_model(hypo_sample)<br>
File “/users/sbhatta9/sumanta/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 722, in _call_impl<br>
result = self.forward(*input, **kwargs)<br>
File “/users/sbhatta9/sumanta/lib/python3.7/site-packages/pytorch_transformers/modeling_bert.py”, line 845, in forward<br>
attention_mask=attention_mask, head_mask=head_mask)<br>
File “/users/sbhatta9/sumanta/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 722, in _call_impl<br>
result = self.forward(*input, **kwargs)<br>
File “/users/sbhatta9/sumanta/lib/python3.7/site-packages/pytorch_transformers/modeling_bert.py”, line 707, in forward<br>
embedding_output = self.embeddings(input_ids, position_ids=position_ids, token_type_ids=token_type_ids)<br>
File “/users/sbhatta9/sumanta/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 722, in _call_impl<br>
result = self.forward(*input, **kwargs)<br>
File “/users/sbhatta9/sumanta/lib/python3.7/site-packages/pytorch_transformers/modeling_bert.py”, line 255, in forward<br>
embeddings = words_embeddings + position_embeddings + token_type_embeddings<br>
Why this error means in the middle of the training?</p>

username: Abhilash_Srivastava
cooked: <p>Pasting the code here will give a better idea.<br>
Maybe the id for a train (or validation) example token is not found in the vocab. You may want to double-check the vocab creation.</p>
Is it possible to train a generative model on features other than text?

username: ogabrielluiz
cooked: <p>Sorry if the title isn’t very descriptive.</p>
<p>Say I have a model such as GPT or T5 and I make it generate some text, now I want to do something with this text, for example, count how many title case words are in it, and have a label that says 5.</p>
<p>Now I want to train the generative model on the loss between the number of title case words of the generated text and my label (5).</p>
<p>Is it possible for me to create a loss function that relies on functions (and probably data) outside of pytorch’s scope? Or is this example of mine a possible scenario?</p>
<p>Thanks!</p>

username: Abhilash_Srivastava
cooked: <p>The loss function should be differentiable.<br>
See if the loss function you come up with (looks like a count function in this case) is differentiable or not.</p>

username: ogabrielluiz
cooked: <p>But it necessarily has to be implemented using PyTorch functions, right?</p>
<p>Say I have functions like this:</p>
<pre><code class="lang-auto">def count_title_case(text):
    return len([word for word in text.split() if word.istitle()])

def loss(output, target):
    output_count = count_title_case(text)
    loss = torch.mean((output_count - target)**2)
    return loss

</code></pre>
<p>Would they work considering I’m using another library to extract information from the output?</p>

username: Abhilash_Srivastava
cooked: <p>It doesn’t matter if it’s using Pytorch functions or not. The functions you plan to use should work with tensors and should be differentiable.</p>
Gradient of a vector with respect to another vector

username: WowPy
cooked: <p>Hi,</p>
<p>I am trying to get the gradient of a vector (with length m and batch size N) with respect to another vector (with length m and batch size N). Hence, I am expecting the gradient matrix to be Nxmxm.</p>
<p>For example,</p>
<pre><code class="lang-auto">x = torch.rand(5, 3)
x.requires_grad_(True)
x.retain_grad()
u = x*x
dudx = grad(u, x, torch.ones(x.size()[0], 3, 3), create_graph=True, retain_graph=True)
</code></pre>
<p>What is the mistake I am making? Many thanks in advance</p>

username: albanD
cooked: <p>Hi,</p>
<p>First, You seem to assume that <code>N</code> is a batch size that should be considered specially? But the autograd does not know about that. It will compute the gradient considering N as any other dimension.</p>
<p>Also <code>autograd.grad()</code> does a vector jacobian product (when grad_outputs is provided) and so will return something with the same size as the input.<br>
You can use <code>autograd.functional.jacobian()</code> to get the full Jacobian but as mentionned above, you will get <code>NxmxNxm</code> as N is considered as any other dimension.</p>
<p>You can do a for loop around <code>jacobian</code> for the N dimension if you want to consider the batch samples one at a time.</p>

username: WowPy
cooked: <p>Thanks, <a class="mention" href="/u/alband">@albanD</a>. This is helpful.</p>
Checkoint Inside forward pass

username: gnadaf
cooked: <p>In DataParallel() with 2 GPUs.</p>
<p>If I save checkpoint inside forward pass, checkpoint saves the weights of both GPUs (these weights are the same). However, If I keep checkpoint outside of forward pass, checkpoint saves weights of default devices.</p>
<p>Which place is good to keep checkpoint, (inside forward pass or outside forward pass) ?</p>

username: albanD
cooked: <p>Hi,</p>
<p>All these weights should have the exact same value. So I would recommend saving them outside so that you only save a single copy for them.</p>

username: gnadaf
cooked: <aside class="quote no-group" data-username="albanD" data-post="2" data-topic="100280">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar"> albanD:</div>
<blockquote>
<p>you on</p>
</blockquote>
</aside>
<p>Yes, If I save checkpoint outside forward pass, checkpoint saves weights of single default device.</p>
<p>But, If I would like to save checkpoint inside forward, checkpoint saves weights of all GPUs (which are same) I believe it is issue in PyTorch.</p>
<p>If it is an issue, I would like to work on it and try to avoid duplicates weights inside the checkpoint.</p>

username: albanD
cooked: <p>Hi,</p>
<p>The DataParallel’s whole point is to run copies of your module on different GPU with different part of the input. So it is expected that the forward is called multiple times.<br>
So if you do something in the forward that has side effects (like saving a file), then it is expected that this will not behave the same way as without DataParallel.</p>
<p>You can guard that part of the code to only run if you are running on GPU 0 bu that would be quite fragile. You should not do checkpointing inside the forward in general.</p>

username: gnadaf
cooked: <p>I agree,<br>
Is there any particular location where we can keep checkpoint?<br>
I am wondering are there any use cases where we can keep checkpoint inside forward pass</p>

username: gnadaf
cooked: <p>I agree,<br>
Is there any particular location where we can keep checkpoint?<br>
I am wondering are there any use cases where we can keep checkpoint inside forward pass</p>

username: albanD
cooked: <p>From my experience we usually checkpoint in the main training loop either at each epoch or after a fixed number of batches.<br>
Doing checkpoint for each batch is most likely slowing down your training quite significantly.</p>
CPU and CUDA tensor to OpenCV Mat differ

username: Joe_Harrison
cooked: <p>When I run the model on cuda and on cpu in Libtorch I get different images while the tensor is almost the same (some numerical error?).  To see whether the tensor are the same I subtract and then sum them which results in a small deviation of around 10e-5.</p>
<p>The resulting images, however, differ a lot. When I construct the image as follows:</p>
<pre><code class="lang-auto">    img = img.to(torch::kUInt8);
    Mat imout(512,512,CV_8UC3);
    
     memcpy((void*)imout.data, img.data_ptr(), sizeof(torch::kUInt8)*img.numel());
</code></pre>
<p>The image is exactly the same as in Python. The cuda image, however, has 9 mini images as such:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/b/0/b058ac3ada8119afbeafdfdaa12bb14ad9376367.png" alt="multitrump" data-base62-sha1="pa20WXJm4vtnipYptICD9KhWGPB" width="299" height="299"></p>
<p>After some research I figured that the order of channels in memory may be different. I tried to construct my own OpenCV Mat by iterating over each element in order.</p>
<pre><code class="lang-auto">   Mat imout(512,512,CV_8UC3);
    for(int i = 0; i&lt;512; i++){
        for(int j = 0; j&lt;512; j++){
            for(int k = 0; k&lt;3; k++){
                imout.at&lt;uint8_t&gt;(i,j,k) = img[i][j][k].item&lt;uint8_t&gt;();
            }
        }
    }
</code></pre>
<p>This results in a colour channel shifted, 3d glasses like, image when I run the model on cuda.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/0/c04003964b9dca80c03a5ba67bfdfb844cc1c0e7.png" alt="cuda" data-base62-sha1="rqIP4mf9UySSRtwxXxFMH2dIO4T" width="299" height="299"></p>
<p>How can I correctly turn a CUDA tensor to a Mat</p>
Initlzie multiple instance of a network?

username: peepeepoopoo
cooked: <p>I’m wonder how one could be able to do something like this in pytroch:</p>
<p>for example I want to initalize 64 copies of nn.Conv.2d, I tried to call a for loop and store the network like this:</p>
<pre><code class="lang-auto">
self. conv_array = []
for i in range (64):
         self. conv_array.append(nn.Conv2d(1, 16, stride=2, kernel_size = 2))
</code></pre>
<p>This apparently doesn’t work after I print out the network after initialization as the layers wont show up.</p>

username: ptrblck
cooked: <p>Use <code>nn.ModuleList</code> instead of a Python <code>list</code> to register these modules properly.</p>

username: peepeepoopoo
cooked: <p>Thank you, thank works!</p>
Time Series with multiple entries/values for one date and feature

username: Username2
cooked: <p>Hey,</p>
<p>I’m trying to forecast the product demand.<br>
(Long-Term-Demand-Forecasting…)<br>
My raw data looks (really simplified) like this:</p>
<pre><code class="lang-auto">	Date	    Customer_id	article_id	indicator1	indicator2	order_size
0	2020-03-01	D021     	104     	True    	213.6   	10
1	2020-03-02	D034    	243     	True    	325.2   	15
2	2020-03-02	D034    	311     	False    	65.3    	43
3	2020-03-02	D054    	104     	False    	853.2    	5
4	2020-03-03	D021    	554     	False    	125.8   	67
5	2020-03-03	D093    	219     	True    	34.2    	11
</code></pre>
<p>I want to predict the order_size (maybe separately for every article).<br>
I already written something that combines Embedding for the categorical data with an LSTM.<br>
I really would like to use a RNN, but I don’t have a proper time series since i have multiple rows for one date, because there are multiple orders every day from different customers for different products.</p>
<p>I could just “force” it into one row by replacing the customer_id-column with a "no. of customer that placed an order that day’-column, but then I would lose important information about which customer orders which product, etc.</p>
<p>Any idea how I could manage this without losing information?</p>
Run the Pytoch with Double precision gets error!

username: Sumanta_Ghosh
cooked: <pre><code class="lang-auto">type or paste code hereclass CSVDataset(Dataset):
    def __init__(self, path):
        df  = read_csv(path,usecols=[5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,30], header=None)
        
        self.X = df.values[:, :-1]
        self.y = df.values[:, -1]
       

        self.X = self.X.astype('float32')
        self.y = self.y.astype('float32')
        

        self.y = self.y.reshape((len(self.y), 1))
        

    def __len__(self):
        return len(self.X)

    def __getitem__(self, idx):
        return [self.X[idx], self.y[idx]]

    def get_splits(self, n_test=0.20):                              # spliting of dataset 
        test_size = int(round(n_test * len(self.X)))
        train_size = int(len(self.X) - test_size)
        return random_split(self, [train_size, test_size])

def prepare_data(path):
    dataset = CSVDataset(path)
    train, test = dataset.get_splits()
    train_dl = DataLoader(train, batch_size=50, shuffle=False)
    test_dl = DataLoader(test, batch_size=500, shuffle=False)
    return train_dl, test_dl


</code></pre>
<p>Here I have assigned by datatype as “float32”. But when I try to change the dtype to double precision it shows some error.</p>
<p>Traceback (most recent call last):<br>
File “./amplitude_signANN.py”, line 267, in <br>
train_model(train_dl, model)<br>
File “./amplitude_signANN.py”, line 147, in train_model<br>
yhat = model(inputs)<br>
File “/home/sumanta/.local/lib/python2.7/site-packages/torch/nn/modules/module.py”, line 532, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “./amplitude_signANN.py”, line 87, in forward<br>
X = self.hidden1(X)<br>
File “/home/sumanta/.local/lib/python2.7/site-packages/torch/nn/modules/module.py”, line 532, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/home/sumanta/.local/lib/python2.7/site-packages/torch/nn/modules/linear.py”, line 87, in forward<br>
return F.linear(input, self.weight, self.bias)<br>
File “/home/sumanta/.local/lib/python2.7/site-packages/torch/nn/functional.py”, line 1370, in linear<br>
ret = torch.addmm(bias, input, weight.t())<br>
RuntimeError: Expected object of scalar type Float but got scalar type Double for argument <span class="hashtag">#2</span> ‘mat1’ in call to _th_addmm</p>
<p><strong>Can pytorch train the data with double precision ? If yes then how can that be fixed?</strong></p>

username: ptrblck
cooked: <p>You have to transform the input data as well as the model parameters to <code>float64</code>.<br>
The error points to a <code>dtype</code> mismatch so I guess your model parameters are still in <code>float32</code>.</p>
Creating big tensors and insert ones at specific dimensions

username: kontrabas
cooked: <p>Hello,</p>
<p>I’ve got problem to which I can’t find the proper answer.</p>
<p>I need to create very large tensors (with zeros) - my_tensor.shape = [256, 3, 500000] and assign to specific indexes “1”. I have a tensor with .shape = [256, 3] containing indexes at which should I assign “1” to every tensor inside my_tensor.</p>
<p>So I was doing it in the loop one by one and concatenate to create 256 sized batch (which is time consuming), and now I’m thinking if there is possibility to make it faster in that way:</p>
<pre><code class="lang-auto">my_tensor = torch.zeros([256, 3, 500000])
my_tensor[#here some magic indexes] = 1
</code></pre>
<p>Could you suggest me some efficient way to solve my problem?</p>
<p>Best regards</p>

username: kontrabas
cooked: <p>I think that I found a “bug”.<br>
To reproduce, try first this:</p>
<pre><code class="lang-auto">import torch
t1 = torch.zeros([3, 3, 10])
t2 = torch.tensor([[0,1,2],[3,4,5],[6,7,8]])
t1[range(t2.shape[0]), torch.tensor([range(t2.shape[1])] * t2.shape[0]).T, t2.T] = 1
Output:
tensor([[[1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
         [0., 1., 0., 0., 0., 0., 0., 0., 0., 0.],
         [0., 0., 1., 0., 0., 0., 0., 0., 0., 0.]],

        [[0., 0., 0., 1., 0., 0., 0., 0., 0., 0.],
         [0., 0., 0., 0., 1., 0., 0., 0., 0., 0.],
         [0., 0., 0., 0., 0., 1., 0., 0., 0., 0.]],

        [[0., 0., 0., 0., 0., 0., 1., 0., 0., 0.],
         [0., 0., 0., 0., 0., 0., 0., 1., 0., 0.],
         [0., 0., 0., 0., 0., 0., 0., 0., 1., 0.]]])
</code></pre>
<p>This works, and now with one small modification (type of the tensor modified):</p>
<pre><code class="lang-auto">import torch
t1 = torch.zeros([3, 3, 10])
t2 = torch.tensor([[0,1,2],[3,4,5],[6,7,8]], dtype=torch.uint8)
t1[range(t2.shape[0]), torch.tensor([range(t2.shape[1])] * t2.shape[0]).T, t2.T] = 1
Output:
IndexError: too many indices for tensor of dimension 3
</code></pre>
<p>Do you have some explanation for this?</p>
<p>Best regards</p>

username: ptrblck
cooked: <p><code>torch.uint8</code> was used as a bool index in past PyTorch versions and you should also get a warning:</p>
<pre><code class="lang-python">UserWarning: indexing with dtype torch.uint8 is now deprecated, please use a dtype torch.bool instead. 
</code></pre>
<p>To index a tensor use <code>LongTensor</code>s or a mask via <code>BoolTensor</code>s.</p>
How to normalize feature using running mean and running var meanwhile keep update them?

username: Shanyan_Guan
cooked: <p>In my mind, the eval mode will normalize feature maps with the recorded running mean and running var. But in eval mode, they cannot be updated.</p>
<p>In train mode, the running mean and running var can be updated, but the BN layer doesn’t use them to normalize feature.</p>
<p>How to update running mean and var and then use them to normalize the input features?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Shanyan_Guan" data-post="1" data-topic="100110">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/shanyan_guan/48/29049_2.png" class="avatar"> Shanyan_Guan:</div>
<blockquote>
<p>How to update running mean and var and then use them to normalize the input features?</p>
</blockquote>
</aside>
<p>Do you want to calculate the batch stats, update the running stats, and use the running stats to normalize the data?<br>
If so, I think the cleanest way would be to implement a custom batchnorm layer by changing the logic in the current implementation.</p>
Question about multiple optimizers for different parts of model

username: ChrisLiu2
cooked: <p>Hi, I have a situation in my neural network that I don’t know how to handle, can someone help me get this straight?</p>
<p>I’m implementing a new graph pooling layer, and this graph pooling layer will be inserted between graph conv layers. However, the learnable components in the pooling layer uses a completely different loss that is calculated at the end of each epoch. I think I need to define a separate optimizer for the pooling layer, but how do I specify the other optimizer so that it doesn’t try to change the weight of the pooling layer and only changes the weight in conv layers? Thanks.</p>

username: ptrblck
cooked: <p>You can pass subsets of parameters to an optimizer while creating it, which makes sure that the optimizer only updates these parameters.</p>
Are tensor method call duplications unnecessarily inefficient?

username: Victory
cooked: <p>I inherited someone’s code which has usages like the following all over the place:</p>
<pre><code class="lang-auto">        a = colors[idxs.long()]
        b = coords[idxs.long()]

        c = foo1(offsets.cuda())
        d = foo2(offsets.cuda())
</code></pre>
<p>In the first two lines, call to idxs.long() gets repeated. In the other two lines, call to “offsets.cuda()” gets repeated.</p>
<p>Is it more efficient to assign the results of those calls to some temporary variables and then use them instead of replicating the calls, or, does some kind of internal optimization take care of such cases anyway?</p>
<p>Thanks.</p>

username: ptrblck
cooked: <p>I would avoid repeating transformations and transforms the tensors once.<br>
Often you also don’t need the same tensor in different <code>dtype</code>s or devices, so I’m wondering if e.g. <code>offsets</code> is used anywhere later on the CPU.</p>
Single model vs multiple model shows different Accuracy

username: Woosung
cooked: <p>Hi Guys,</p>
<p>I’m currently executing several kinds of models at the same time as multiple processes.</p>
<p>and I’m using same Imagenet 1k dataset which inside of my data folder.</p>
<p>the problem is when I do validate the single(mobilenetv2) model and when I do validate with multiple(mobiletv2 + other dnns)models, the accuracy shows slightly different.</p>
<p>I think multiple dataloader looking at the same data folder might be the problem but not sure it is.</p>
<p>Do you know what makes this problem?</p>
<p>I use PyTorch 1.4.0 on NVIDIA Jetson Xavier<br>
I got the data sets and dataloader code from the below page<br>
one thing different is that i set num_worker as a 0.</p>
<aside class="onebox allowlistedgeneric">
  <header class="source">
      <a href="https://pytorch.org/tutorials/advanced/static_quantization_tutorial.html" target="_blank" rel="noopener nofollow ugc">pytorch.org</a>
  </header>
  <article class="onebox-body">
    <img src="" class="thumbnail" width="" height="">

<h3><a href="https://pytorch.org/tutorials/advanced/static_quantization_tutorial.html" target="_blank" rel="noopener nofollow ugc">(beta) Static Quantization with Eager Mode in PyTorch — PyTorch Tutorials...</a></h3>



  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>Thanks</p>

username: ptrblck
cooked: <p>How reproducible is this behavior, i.e. are you always seeing this issue using e.g. 10 different runs with different seeds?</p>
How to load multiple .mat files for training in Pytorch

username: Abhi
cooked: <p>I have 3D dataset(1000 files) in .mat format. and I wish to load for training. can anyone please suggest to me how to do this?</p>

username: ptrblck
cooked: <p>You could use <code>scipy</code> to load the <code>mat</code> files as shown <a href="https://discuss.pytorch.org/t/pytorch-customized-data-loader/30077/2">here</a>.</p>

username: Abhi
cooked: <p>Hi. <a class="mention" href="/u/ptrblck">@ptrblck</a> I created the data loader by using the referenced function. but my data is not divided into batches while training.(mat file contain: complex128)<br>
(one thing I don’t understand I am unable to define batch size greater than path length.)</p>
<p>‘’’<br>
import <a href="http://scipy.io" rel="noopener nofollow ugc">scipy.io</a> as io<br>
class MyDataset(Dataset):<br>
def <strong>init</strong>(self, mat_paths,col_name,transform):<br>
self.paths = mat_paths</p>
<pre><code>    self.matX = io.loadmat(self.paths)[col_name]

    self.transform = transform
def __getitem__(self, index):
  
    X = self.matX[index].astype(np.float32)
   
    X = torch.from_numpy(X)
    X = abs(X)

    return X

def __len__(self):
    return len(self.paths)
</code></pre>
<p>mat_paths = (r’C:\Users\Abhi\Downloads\dataset\outputs\images\train1.mat’)</p>
<p>trainDataset = MyDataset(mat_paths,‘train4’,transform=transform)</p>
<p>trainLoader = DataLoader(trainDataset, batch_size=32, shuffle=True, num_workers=0, pin_memory=True,drop_last=True)<br>
‘’’</p>

username: ptrblck
cooked: <p>I assume <code>self.matX</code> is a numpy array containing all samples in <code>dim0</code>.<br>
If that’s the case, you might need to return <code>len(self.matX)</code> or <code>self.matX.shape[0]</code> in the <code>__len__</code> function instead of the length of <code>self.paths</code>.</p>
<p>PS: you can post code snippets by wrapping them into three backticks ```, which makes your code easier to debug. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

username: Abhi
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> it works. thankyou.<br>
okay, I will wrap my code in three ‘’’ before posting.</p>
Add weight to a metrics in a pytorch code

username: marcomameli01
cooked: <p>Dear,<br>
I’m using the code from: <a href="https://github.com/qubvel/segmentation_models.pytorch" rel="noopener nofollow ugc">Semantic models PyTorch</a> but I need to create a custom loss with a weight assigned to each class.</p>
<p>I would like to understand how to do this.</p>
<p>Can anyone help me please?</p>

username: Naruto-Sasuke
cooked: <pre><code class="lang-python"> def SoftCrossEntropy(inputs, target, weight_id=None, reduction='sum'):
     """
     inputs are performed by log_softmax
     """
     log_likelihood = -inputs

     batch = inputs.shape[0]

     if weight_id is None:
         if reduction == 'average':
             loss = torch.sum(torch.mul(log_likelihood, target)) / batch
         else:
             loss = torch.sum(torch.mul(log_likelihood, target))
         return loss
     else:
         log_likelihood = log_likelihood * weight_id
         target = target * weight_id
         if reduction == 'average':
             loss = torch.sum(torch.mul(log_likelihood, target)) / batch
         else:
             loss = torch.sum(torch.mul(log_likelihood, target))
         return loss
</code></pre>
Loss decrease a lot at end of 1st Epoch. is this fine or mistake

username: 111392
cooked: <pre><code class="lang-auto">for epoch in range(EPOCH):

        for i, data in enumerate(train_loader, 0):

            net.train()

            model_out = net(
                Variable(data["left_aug"]), Variable(data["right_aug"])
            )
            REC_loss, disp_smooth_loss, lr_loss = get_loss(model_out)
      
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/b/db45ceae0c886369899de8daa7f3a918a2d7b004.png" data-download-href="https://discuss.pytorch.org/uploads/default/db45ceae0c886369899de8daa7f3a918a2d7b004" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/b/db45ceae0c886369899de8daa7f3a918a2d7b004_2_690x213.png" alt="image" data-base62-sha1="vhM7UnRXItPd6xcVAumVMEuIMWU" width="690" height="213" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/b/db45ceae0c886369899de8daa7f3a918a2d7b004_2_690x213.png, https://discuss.pytorch.org/uploads/default/original/3X/d/b/db45ceae0c886369899de8daa7f3a918a2d7b004.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/d/b/db45ceae0c886369899de8daa7f3a918a2d7b004.png 2x" data-dominant-color="FDFCFC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">811×251 20.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: RoySadaka
cooked: <p>What is the x axis? is it epochs? or iterations?</p>
<p>I would say, yes it’s possible, it depends on a few things though</p>
<ol>
<li>
<p>If x axis is iterations, and you calculate the total mean of the loss from the start until that iteration, so when the epoch is done and you start a new epoch, you start calculating a new mean, so you don’t drag your history from your previous epoch, that’s why you see this big drop.</p>
</li>
<li>
<p>It could be that your data loader has few batches of “easy” samples at the end</p>
</li>
<li>
<p>Try shuffle the data and post the plot of the result, is it reproducing?</p>
</li>
</ol>
<p>Roy</p>

username: 111392
cooked: <p>x is iteration</p>
<ol>
<li>I calculate mean of loss</li>
</ol>
<p>now I got reason</p>
<p>thanks</p>
Create two copies from the a model?

username: Erik_Bertolino
cooked: <p>Grettings.</p>
<p>Suppose I have a model and a certain data. I train the model on the data, now I want to create two independent copies of said trained model. With independent I mean that changes done to one copy of the model should not affect the other copy, or the original model.</p>
<p>I tried to save the trained model with torch.save(model, PATH) and then simply model_copy = torch.load(PATH). But this ends up that these two are “connected”. Is my programming off or why is this the case?</p>
<p>Best regards</p>
<p>Erik`</p>

username: Naruto-Sasuke
cooked: <p>How about doing this:</p>
<pre><code class="lang-python">import copy

net_a = torch.load(PATH)
net_b = copy.deepcopy(net_a)
</code></pre>
How to achieve same dimension or padding='same' after Conv2d in Pytorch?

username: krishna511
cooked: <p>Hello there,<br>
I am using following code in model ,</p>
<p>I=torch.randn(40,1,128,186)<br>
m=nn.Sequential(<br>
nn.Conv2d(1,16, kernel_size=5, stride=1),<br>
nn.BatchNorm2d(16),<br>
nn.ReLU())<br>
O=m(I)</p>
<p>I want my output to be of same size as input ie (40,1,128,186)<br>
I tried using</p>
<p>O=m(F.pad(I,(0,0,2,2)))<br>
But getting (40, 16, 128, 182)</p>
<p>Can anyone help plz.</p>

username: Abhilash_Srivastava
cooked: <p>From what I know, Pytorch doesn’t support this as an inbuilt option, TensorFlow does. Checkout this <a href="https://github.com/pytorch/pytorch/issues/3867#issuecomment-507010696" rel="noopener nofollow ugc">discussion</a> which mentions how dynamic loading makes it hard.<br>
However, there could be ways to hack it by combining asymmtric padding layers with conv2d layers. I wouldn’t bother doing it, unless super useful and just go with the inbuilt padding options. More discussion <a href="https://discuss.pytorch.org/t/is-asymmetric-padding-of-style-same-available-in-pytorch/4354/8">here</a>.</p>
How to multiple a part of tensor with another tensor?

username: Johnson_Mark
cooked: <p>I have a tensor <code>x</code> size of <code>BxCxHxW</code> and a tensor y size of <code>B/2 xCxHxW</code>. I want to perform multiplication between tensor x and tensor y such as the first part of tensor x (from 0 to B/2) will be modified by the result of the multiplication <strong>during forwarding</strong> , while the last part of tensor x is unchanged as the bellow figure.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/8/5/85c9d7096c718a0b3a3c98e65423b18e90150504.png" alt="Screenshot from 2020-10-22 01-15-57" data-base62-sha1="j5xWvLJ1XKg3ynBOCKeWvXqpSks" width="588" height="288"></p>
<p>How to do it in pytorch? I have my way</p>
<pre><code class="lang-auto">import torch
bx,cx,h,w = 4,2,3,3
by,cy,h,w = 2,2,3,3
tensor_x = torch.rand((bx,cx,h,w), requires_grad=True)
tensor_y = torch.rand((by,cy,h,w), requires_grad=True)
tensor_x_first = tensor_x[:by,...]  * tensor_y
tensor_x[:by,...] = tensor_x_first
</code></pre>
<p>I got the error during backward</p>
<pre><code class="lang-auto">RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [4, 2, 3, 3]], which is output 0 of SliceBackward, is at version 1; expected version 0 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).

</code></pre>

username: KFrank
cooked: <p>Hello Johnson!</p>
<aside class="quote no-group" data-username="Johnson_Mark" data-post="1" data-topic="100191" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/johnson_mark/48/29871_2.png" class="avatar"> Johnson_Mark:</div>
<blockquote>
<pre><code class="lang-auto">tensor_x_first = tensor_x[:by,...]  * tensor_y
tensor_x[:by,...] = tensor_x_first
</code></pre>
</blockquote>
</aside>
<p>Try:</p>
<pre data-code-wrap="python"><code class="lang-python">tensor_z = tensor_x * torch.cat ((tensor_y, torch.ones_like (tensor_y)), dim = 0)
</code></pre>
<p>Best.</p>
<p>K. Frank</p>
Pixel size change using MONAI

username: Mahdieh
cooked: <p>Hello,</p>
<p>I am working with 3d CT images. I am trying to change pixel dimension from [1 1.36 1.36 1.36] to [1 1 1 1] using MONAI package. I used : Spacingd(keys=[“image”, “label”], pixdim=(1, 1, 1), mode=(“bilinear”, “nearest”))<br>
When I print pixel dimension using : print(f" :\n{data_dict[‘image_meta_dict’][‘pixdim’]}"), the output is still [1 1.36 1.36 1.36] while the affine matrix reflects the new pixel size. Would appreciate any comments on this…<br>
<a class="mention" href="/u/monai">@MONAI</a></p>

username: wyli
cooked: <p>Hi! those transforms currently only rely on the “affine” key to track the coordinate system changes, ‘pixdim’ is not directly used at the moment.</p>
RuntimeError: cholesky_cuda: For batch 0: U(16,16) is zero, singular U

username: Shanya_Anand
cooked: <p>I am trying to find KL divergence between two diagonal Gaussian distribution lets say N1(prior) and N2(posterior). In code, this looks something like`</p>
<p>_Σ = T.bmm(I, ((_σ).permute(1, 2, 0).clone().repeat(1, 1, I.size()[1]))) + I<br>
_Σ_pr = T.bmm(I, ((_σ_pr).permute(1, 2, 0).clone().repeat(1, 1, I.size()[1]))) + I<br>
Dkl = kl_divergence(MultivariateNormal(_μ.squeeze(), _Σ.squeeze()),MultivariateNormal(_μ_pr.squeeze(), _Σ_pr.squeeze()))</p>
<p>Where I is an identity matrix with dim as [batch_size, m, m](m is the dimension of Gaussian distribution)(and I[x,:, :] = identity, for all x) and _σ, _σ_pr and _μ, _μ_pr are the variance and mean of diagonal Gaussians. _Σ and _Σ_pr are covariance matrix with diagonal elements as the variance of corresponding Gaussians. I am trying to find KL divergence of a batch at once rather than running a for loop to avoid this error. Is this because one of the diagonal elements is zero in covariance matrix so thus when PyTorch tries to find the inverse of it, it’s getting an error. If so then I also tried to add a number so that I can make that zero diagonal elements non-zero.<br>
I will be really grateful if someone gives me more efficient or working way of doing this.<br>
PS : I am using Google Colab</p>

username: Shanya_Anand
cooked: <p>The problem was actually with using torch.bmm in order to make diagonal covariance matrix, to be precise it wasn’t working the way I assumed it to worked so instead of using torch.bmm, I used torch.diag_embed()</p>
I want to split my audio dataset into train and test. can datasetLoader do this for me?

username: krishna511
cooked: <p>I just started coding in Pytorch. I have converted my wav files into text using glob library. But now I want to split that text file into train and test. Actually the dataset is very small and imbalanced. To be more clear it has 7 classes in file name only. But different classes have different samples like 100, 50 etc. Not How to split it into Train and Test<br>
can anybody help plz</p>

username: ptrblck
cooked: <p>If you want to randomly split your <code>Dataset</code>, you could use <code>torch.utils.data.random_split</code>.<br>
Alternatively, if you want to apply a stratified split, you could use <a href="https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html"><code>sklearn.model_selection.train_test_split</code></a> and pass the targets as the <code>stratify</code> argument.</p>

username: krishna511
cooked: <p>Thank you so much sir</p>
Custom Loss Function that requires numpy

username: Gerard_Jover_Pujol
cooked: <p>I have to implement a custom loss function to estimate some weights using the the total variance of a set of images. The problem is that my function requires using np.gradient(), and that breaks the computation graph. Does anyone know how to solve this?<br>
Here’s whatI have so far:</p>
<p>class TotalVariation(Function):</p>
<pre><code>@staticmethod 
def forward(ctx, projections, meanFF, FF, DF, x):
    FF_eff = torch.zeros((FF.shape[1], FF.shape[2]))
    for i in range(len(FF)):
        FF_eff  = FF_eff + x[i] * FF[i]
    logCorProj=(projections-DF)/(meanFF+FF_eff)*torch.mean(
        torch.flatten(meanFF)+torch.flatten(FF_eff))
    # tmp = torch.mean(logCorProj)
    # return tmp
    # pytorch doesn't have a np.gradient equivalent
    logCorProj = logCorProj.clone().detach().numpy()
    Gx, Gy = np.gradient(logCorProj)
    Gx, Gy = torch.tensor(Gx), torch.tensor(Gy)
    mag = (Gx**2 + Gy**2)**(1/2)
    cost = torch.sum(torch.flatten(torch.from_numpy(mag)))
    ctx.save_for_backward(cost)
    return cost

@staticmethod 
def backward(ctx, grad_outputs):
    result, = ctx.saved_tensors
    return result, None, None, None, None</code></pre>

username: albanD
cooked: <p>Hi,</p>
<p>If you use a custom Function, the backward should compute the gradient wrt the inputs by using the gradient wrt the outputs that are given to you.<br>
As you don’t use any advanced option of np.gradient, it might just be simpler to re-implement it by hand using pytorch’s ops as you simply compare the adjacent values in the given input.</p>

username: Gerard_Jover_Pujol
cooked: <p>Thanks, I’ve implemented a gradient function with PyTorch and now it’s working.</p>
Cuda runtime error: arguments are located on different GPUs

username: Tomojit
cooked: <p>Hi - I have a machine which has two GPUs. I noticed that if I run my code on the the second device (cuda id=1) the code runs for sometimes then error out with the following error message. I’m not doing any memory sharing or multiple GPU programming. I’m using the torch version ‘1.2.0’. Is this an issue with this version? What is a fix for this error?</p>
<p>RuntimeError: arguments are located on different GPUs at /pytorch/aten/src/THC/generic/THCTensorMathBlas.cu:260</p>
<p>Thank you,<br>
Tomojit</p>

username: JuanFMontesinos
cooked: <p>There is probably a hard-coded device <code>.cuda()</code> in some part of the code.<br>
If you don’t want to inspect the code, you can run<br>
<code>CUDA_VISIBLE_DEVICES=1 python path_to_the_file.py</code><br>
This environment variable will make that python program to consider the only gpu is the gpu 1.</p>
Index_put_ does not work correctly with sparse tensors libtorch

username: rubique
cooked: <p>Dear sir or madam,<br>
It seems like function index_put_ (used for <a href="https://pytorch.org/cppdocs/notes/tensor_indexing.html" rel="noopener nofollow ugc">setting values in tensors</a>) does not do anything with sparse tensors.</p>
<p>In the following code I try to insert value 42 in position (0, 0) of a sparse tensor:</p>
<pre><code class="lang-auto">  auto sparse_tensor = torch::zeros({2, 2}, torch::kSparse);

  std::cout &lt;&lt; "Before put: " &lt;&lt; sparse_tensor.index({0, 0}) &lt;&lt; std::endl;
  sparse_tensor.index_put_({0, 0}, 42);
  std::cout &lt;&lt; "After put: " &lt;&lt; sparse_tensor.index({0, 0}) &lt;&lt; std::endl;


</code></pre>
<p>The output of the code shows that no value was inserted:</p>
<pre><code class="lang-auto">Before put: 0
[ CPUFloatType{} ]
After put: 0
[ CPUFloatType{} ]


</code></pre>
<p>While for dense matrices the function works as intended:</p>
<pre><code class="lang-auto">  auto dense_tensor = torch::zeros({2, 2}, torch::kStrided);

  std::cout &lt;&lt; "Before put: " &lt;&lt; dense_tensor.index({0, 0}) &lt;&lt; std::endl;
  dense_tensor.index_put_({0, 0}, 42);
  std::cout &lt;&lt; "After put: " &lt;&lt; dense_tensor.index({0, 0}) &lt;&lt; std::endl;
</code></pre>
<p>Giving the following output:</p>
<pre><code class="lang-auto">Before put: 0
[ CPUFloatType{} ]
After put: 42
[ CPUFloatType{} ]
</code></pre>
<p>I build test application with (nightly/cpu/libtorch-cxx11-abi-shared-with-deps-latest.zip) binaries from this post: <a href="https://github.com/pytorch/pytorch/issues/17492#issuecomment-524692441" rel="noopener nofollow ugc"> Libtorch binaries compiled with flag _GLIBCXX_USE_CXX11_ABI = 1</a>.</p>
<p>Could you please help me understand whether I do something incorrectly or it is a bug?<br>
Thank you in advance for any assistance!</p>