akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Possible to reweight GPU load with DataParallel?

Entropy: <p>Say I am training with 4 GPUs but GPU 1 is also in charge of doing some other stuff that I need to set aside some memory for. Is it possible to give GPUs 2-4 a bigger load and GPU 1 a smaller load instead of splitting a batch among all GPUs evenly (currently, this is leading to an OOM error on GPU 1 as expected)?</p>

ptrblck: <p>I don’t think this is easily doable in <code>nn.DataParallel</code>, but should be possible using <code>DistributedDataParallel</code>. Note that the former is also deprecated while the latter is the recommended approach, too.</p>
S3FileLister fails to connect to s3

tomerroditi: <p>I am having issues connecting to S3 with the S3FileLister.<br>
my code:</p>
<pre><code class="lang-auto">from torchdata.datapipes.iter import IterableWrapper
s3_prefix = ['s3://mybucket/myfolder/']
dp = IterableWrapper(s3_prefix).list_files_by_s3()
list(dp)
</code></pre>
<p>the error:</p>
<pre><code class="lang-auto">ValueError                                Traceback (most recent call last)
Cell In[6], line 32
     30 s3_prefix = ['s3://vehicle-data-db/' + key_prefix for key_prefix in [f'volvo/april_{i}/' for i in range(1, 3)]]
     31 dp = IterableWrapper(s3_prefix).list_files_by_s3()
---&gt; 32 list(dp)

File ~\anaconda3\envs\ca\lib\site-packages\torch\utils\data\datapipes\_hook_iterator.py:173, in hook_iterator.&lt;locals&gt;.wrap_generator(*args, **kwargs)
    171         response = gen.send(None)
    172 else:
--&gt; 173     response = gen.send(None)
    175 while True:
    176     datapipe._number_of_samples_yielded += 1

File ~\anaconda3\envs\ca\lib\site-packages\torchdata\datapipes\iter\load\s3io.py:73, in S3FileListerIterDataPipe.__iter__(self)
     71 for prefix in self.source_datapipe:
     72     while True:
---&gt; 73         urls = self.handler.list_files(prefix)
     74         for url in urls:
     75             if match_masks(url, self.masks):

ValueError: Encountered network error when sending http request
This exception is thrown by __iter__ of S3FileListerIterDataPipe(length=-1, masks='', source_datapipe=IterableWrapperIterDataPipe)
</code></pre>
<p>when trying to access this bucket with boto3 it works as expected.<br>
my credentials file is located at <code>C:\Users\myuser\.aws</code><br>
what modifications do i need to perform to make it work?</p>

tomerroditi: <p>i didnt specified the region, fixed code that worked for me:</p>
<pre><code class="lang-auto">from torchdata.datapipes.iter import IterableWrapper
s3_prefix = ['s3://mybucket/myfolder/']
dp = IterableWrapper(s3_prefix).list_files_by_s3(region='myregion')
list(dp)
</code></pre>
Question about the implementation of Nesterov Accelerated Gradient in PyTorch

Entropy: <p>This is more of a conceptual question since I recently learned about this optimization algorithm. All of the code below is just pseudocode (not actual python). We know that the classical momentum update is given by</p>
<pre><code class="lang-auto">    g(t) = mu * g(t-1) + grad loss(theta(t-1)) 
theta(t) = theta(t-1) - alpha * g(t)
</code></pre>
<p>where <code>mu</code> is the momentum parameter and <code>alpha</code> is the learning rate.<br>
The Nesterov Accelerated Gradient update is given by</p>
<pre><code class="lang-auto">    g(t) = mu * g(t-1) + grad loss(theta(t-1) - mu * alpha * g(t-1)) 
theta(t) = theta(t-1) - alpha * g(t)
</code></pre>
<p>To make things somewhat simpler, we can define</p>
<pre><code class="lang-auto">theta_new(t) = theta(t) - alpha * mu * g(t)
</code></pre>
<p>Then, the update equations above can be expressed as</p>
<pre><code class="lang-auto">        g(t) = mu * g(t-1) + grad loss(theta(t-1)) 
    theta(t) = theta(t-1) - alpha * g(t) 
theta_new(t) = theta(t) - alpha * mu * g(t)
             = theta(t-1) - alpha * g(t) - alpha * mu * g(t)
             = theta_new(t-1) + alpha * mu * g(t-1) - alpha * g(t) - alpha * mu * g(t) 
             = theta_new(t-1) - alpha * (g(t) - mu * g(t-1) + mu * g(t)) 
             = theta_new(t-1) - alpha * (grad loss(theta_new(t-1)) + mu * g(t))
</code></pre>
<p>I apologize if this notation is not standard: it’s just what we used in my deep learning class.</p>
<p>Unless I am mistaken, it appears that the PyTorch implemenation of SGD (documentation here: <a href="https://pytorch.org/docs/stable/generated/torch.optim.SGD.html" class="inline-onebox" rel="noopener nofollow ugc">SGD — PyTorch 1.10 documentation</a>) uses the update for <code>theta_new</code>  directly on the model parameters <code>theta</code> without explicitly defining the relationship between the two. I know that the PyTorch implementation works, so I am wondering where the disconnect is between the derivation above and what is present in the source code. Thank you!</p>

Entropy: <p>I know I am a month late but bumping this!</p>
A bug from torch.nn.ConstantPad1d

MartinZhang: <p>Today I use the<code> torch.nn.ConstantPad1d</code> as follow:</p>
<pre><code class="lang-auto">import torch

layer = torch.nn.ConstantPad1d((2, 2, 2, 2), 0)

data = torch.randn(1, 3, 224)

result = layer(data)

print(result.shape) # 1, 7, 228
</code></pre>
<p>I think this is the wrong way to use it.</p>
<p>Why does it work?</p>

eqy: <p>This is because all of the ConstantPadNd functions call <code>torch.nn.functional.pad</code> under the hood without checking the length of the padding, and <code>torch.nn.functional.pad</code> doesn’t have any information about what the <code>nn.Module</code>’s dimensionality is. This might be sorta in the “it’s a feature not a bug” category as it is not technically doing an “invalid” computation:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch
&gt;&gt;&gt; a = torch.randn(1, 3, 224)
&gt;&gt;&gt; torch.nn.functional.pad(a, (2,2,2,2)).shape
torch.Size([1, 7, 228])
&gt;&gt;&gt; torch.nn.functional.pad(a, (2,2)).shape
torch.Size([1, 3, 228])
&gt;&gt;&gt; torch.nn.functional.pad(a, (2,2,3,3)).shape
torch.Size([1, 9, 228])
&gt;&gt;&gt; torch.nn.functional.pad(a, (2,2,3,3,6,6)).shape
torch.Size([13, 9, 228])
&gt;&gt;&gt; torch.nn.functional.pad(a, (2,2,3,3,6,6,8,8)).shape
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
RuntimeError: Padding length too large
&gt;&gt;&gt;
</code></pre>

MartinZhang: <p>Thank you for your reply <img src="https://discuss.pytorch.org/images/emoji/apple/grin.png?v=12" title=":grin:" class="emoji" alt=":grin:" loading="lazy" width="20" height="20">. Maybe there is no limit dimension for<code> torch.nn.functional</code> with the <code>mode='constant'</code>, other patterns are dimensionally constrained,like <code>reflect or replicate</code>. <img src="https://discuss.pytorch.org/images/emoji/apple/rofl.png?v=12" title=":rofl:" class="emoji" alt=":rofl:" loading="lazy" width="20" height="20"></p>
Using dataloader in a celery task

priyathamkat: <p>I’m trying to use PyTorch inside a Celery container and running into some issues related to threading. Initializing a DataLoader with num_workers greater than 0 successfully starts a new thread - but instead of acting as a dataloader, the new thread runs as a new celery worker and waits for a new task to be submitted! Is there a way to control how a new thread is spawned?</p>

ptrblck: <p>The <code>Dataloader</code> will spawn new processes if <code>num_workers&gt;0</code> is used and won’t use threads.<br>
I’m not familiar with Celery and don’t know about its specific requirements etc.</p>
Validation_step_output is empty

El-Hassan-Hajbi: <p>I use google tft model from pytorch.</p>
<p>My question is : what does validation_step_output tft’s attribute contain ? When I debug my code I see that is still empty when calling <strong>on_validation_epoch_end</strong>, why??</p>
<pre><code class="lang-auto">def on_validation_epoch_end(self):
        self.on_epoch_end(self.validation_step_outputs)
        self.validation_step_outputs.clear()
</code></pre>
<p>here I get self.validation_step_outputs empty list as mentioned before.</p>
<p>while this function is responsible of adding the val_loss into the list I think</p>
<pre><code class="lang-auto">def validation_step(self, batch, batch_idx):
        x, y = batch
        log, out = self.step(x, y, batch_idx)
        log.update(self.create_log(x, y, out, batch_idx))
        self.validation_step_outputs.append(log)
        return log
</code></pre>

ptrblck: <p>Is this a double post from <a href="https://discuss.pytorch.org/t/index-out-of-range-from-trainer-fit/184930">here</a>?</p>
Quantization accuracy debugging for custom LSTM PTQ

Ahmed_Louati: <p>Hello,<br>
I am trying to debug custom LSTM PTQ using <code>torch.ao.ns._numeric_suite_fx</code>.<br>
This is the code after custom quantization:</p>
<pre><code class="lang-auto"># extract weight pairs
weight_comparison = ns.extract_weights('a', m, 'b', converted)

ns.extend_logger_results_with_comparison(
    weight_comparison, 'a', 'b', torch.ao.ns.fx.utils.compute_sqnr,
    'sqnr')
</code></pre>
<p>I got this error:</p>
<pre><code class="lang-auto">    raise GraphMatchingException(msg)
torch.ao.ns.fx.graph_matcher.GraphMatchingException: 
The subgraphs
(NSSubgraph(start_node=my_lstm, end_node=my_lstm, base_op_node=my_lstm), &lt;class 'torch.nn.modules.rnn.LSTM'&gt;) and
(NSSubgraph(start_node=my_lstm, end_node=my_lstm, base_op_node=my_lstm), &lt;class 'torch.ao.nn.quantized.modules.rnn.LSTM'&gt;)
are not related. Please ensure that the two models you pass in have the same number
of subgraphs, and each pair of subgraphs is related to each other.
</code></pre>
<p>Does this mean that custom LSTM quantization is not supported for those APIs?<br>
Or I should manually match quantized.LSTM and nn.LSTM ops?</p>

jerryzh168: <p>probably means numeric suite is not yet supporting comparing nn.LSTM and nn.quantized.LSTM. cc <a class="mention" href="/u/vasiliy_kuznetsov">@Vasiliy_Kuznetsov</a> could you comment on the plan on the support?</p>

Vasiliy_Kuznetsov: <p>Numeric Suite for workflows such as Eager mode quantization and FX graph mode quantization is not something the core quantization team actively works on because we expect PT2.0 export quantization to be more aligned with our future plans, but we would welcome contributions.</p>
Quantizing an existing object detector with ResNet backbone

talh: <ul>
<li>Working on Windows with Pytorch 1.9.1+cu102</li>
<li>I have an object detector that runs with a Torchvision FP32 ResNet-18 backbone.</li>
<li>Added <code>QuantStub</code> and <code>DequantStub</code> to my model, calling the quant stub on the input and the dequant stub on the output (in the model’s forward method)</li>
<li>Quantized the model using fbgemm</li>
</ul>
<p>When the ResNet model tries to perform <code>out += identity</code> (From Torchvision implementation, <code>BasicBlock</code> class), I get the following error:</p>
<pre><code>File "d:\anaconda3\lib\site-packages\torchvision\models\resnet.py", line 80, in forward
    out += identity
NotImplementedError: Could not run 'aten::add.out' with arguments from the 'QuantizedCPU' backend. This could be because the operator doesn't exist for this backend, or was omitted during the selective/custom build process (if using custom build). If you are a Facebook employee using PyTorch on mobile, please visit https://fburl.com/ptmfixes for possible resolutions. 'aten::add.out' is only available for these backends: [CPU, CUDA, Meta, MkldnnCPU, SparseCPU, SparseCUDA, SparseCsrCPU, BackendSelect, Named, ADInplaceOrView, AutogradOther, AutogradCPU, AutogradCUDA, AutogradXLA, UNKNOWN_TENSOR_TYPE_ID, AutogradMLC, AutogradHPU, AutogradNestedTensor, AutogradPrivateUse1, AutogradPrivateUse2, AutogradPrivateUse3, Tracer, Autocast, Batched, VmapMode].
</code></pre>
<p>How can I overcome it?</p>

Vasiliy_Kuznetsov: <p>Check out here (<a href="https://pytorch.org/docs/stable/quantization.html#common-errors" class="inline-onebox" rel="noopener nofollow ugc">Quantization — PyTorch 1.9.1 documentation</a>) for a description of this error, it means you are passing a quantized tensor to a fp32 kernel.</p>
<p>For eager mode quantization, you could edit the model code to replace the addition with <code>FloatFunctional</code>.</p>
<p>For a backbone such as ResNet18, you could also use FX graph mode quantization which will do the syntax transforms for you automatically.  You could check out here (<a href="https://pytorch.org/docs/stable/quantization.html#quantization-api-summary" class="inline-onebox" rel="noopener nofollow ugc">Quantization — PyTorch 1.9.1 documentation</a>) for an API example (scroll down to FX Graph Mode Quantization).</p>

kent252: <p>Hi, I face same issue, did u fix that ?</p>

jerryzh168: <p>this is a user error that can only be fixed at user side, see <a href="https://pytorch.org/docs/stable/quantization.html#passing-a-quantized-tensor-into-a-non-quantized-kernel" class="inline-onebox" rel="noopener nofollow ugc">Quantization — PyTorch 2.0 documentation</a> for instructions on how to fix</p>
Create a matrix based on vector indices

ernest-s: <p>I have a one-dimensional tensor <code>a = torch.tensor([0,2,3,4])</code>.<br>
From this tensor, I want to create another two-dimensional tensor which looks like this:</p>
<pre><code class="lang-auto">tensor([[1, 1, 0, 0, 0, 0, 0, 0, 0],
        [1, 1, 0, 0, 0, 0, 0, 0, 0],
        [0, 0, 1, 1, 1, 0, 0, 0, 0],
        [0, 0, 1, 1, 1, 0, 0, 0, 0],
        [0, 0, 1, 1, 1, 0, 0, 0, 0],
        [0, 0, 0, 0, 0, 1, 1, 1, 1],
        [0, 0, 0, 0, 0, 1, 1, 1, 1],
        [0, 0, 0, 0, 0, 1, 1, 1, 1],
        [0, 0, 0, 0, 0, 1, 1, 1, 1]])
</code></pre>
<p>The output tensor’s dimensions should be equal to <code>(torch.sum(a), torch.sum(a))</code> which is <code>(9,9)</code>. The tensor <code>a</code> determines which elements of the output should be <code>1</code>. Since, the first two elements of a are <code>0</code> and <code>2</code>, the values in <code>output[0:0+2, 0:0+2]</code> should be <code>1</code>. And <code>output[2:2+3, 2:2+3]</code> should be <code>1</code> since the next two elements are <code>2</code> and <code>3</code>. And finally <code>output[2+3:2+3+4, 2+3:2+3+4]</code> should be <code>1</code>.<br>
If there an efficient way of doing this in PyTorch?</p>

ernest-s: <p><code>repeat_interleave</code> does the trick<br>
<code>torch.repeat_interleave(torch.repeat_interleave(torch.eye(3), a[1:], dim=0), a[1:], dim=1)</code></p>

ptrblck: <p><code>torch.blog_diag</code> should also work:</p>
<pre><code class="lang-python">a = torch.tensor([0,2,3,4])
blocks = []
for a_ in a:
    tmp = torch.ones(a_, a_)
    blocks.append(tmp)

torch.block_diag(*[b for b in blocks])
# tensor([[1., 1., 0., 0., 0., 0., 0., 0., 0.],
#         [1., 1., 0., 0., 0., 0., 0., 0., 0.],
#         [0., 0., 1., 1., 1., 0., 0., 0., 0.],
#         [0., 0., 1., 1., 1., 0., 0., 0., 0.],
#         [0., 0., 1., 1., 1., 0., 0., 0., 0.],
#         [0., 0., 0., 0., 0., 1., 1., 1., 1.],
#         [0., 0., 0., 0., 0., 1., 1., 1., 1.],
#         [0., 0., 0., 0., 0., 1., 1., 1., 1.],
#         [0., 0., 0., 0., 0., 1., 1., 1., 1.]])
</code></pre>

ernest-s: <p>Thanks a lot <a class="mention" href="/u/ptrblck">@ptrblck</a>. Is there any way we can do it without the loop? This function has to go inside a neural network.</p>

ptrblck: <p>You have to pass the tensors as arguments somehow to <code>block_diag</code>. If you can create these tensors separately somehow without a loop, it would also work.<br>
Btw. you can also just use <code>torch.block_diag(*blocks)</code>. In my previous code I had another condition to filter for the empty tensor, which wasn’t needed.</p>
Normal distribution: A log_prob dimension problem

Mehdi: <p>Hello,</p>
<p>I’m trying to work my way through Mixtures Density Network, and while I managed to figure it out for a single feature for the output, I’m stuck when it comes to several features. Among my problems, here’s the one concerning the Normal distribution class.</p>
<p>Here’s an example:</p>
<pre><code class="lang-auto">import torch
from torch.distributions import Normal 

means = torch.zeros(1,2)
stds = torch.ones(1,2)

dist = Normal(means, stds)

print('Sample is size {}'.format(dist.sample().shape))
print('Log prob of {} = {}'.format(torch.zeros(1,2), dist.log_prob(torch.zeros(1,2))))
</code></pre>
<p>Outputs:</p>
<pre><code class="lang-auto">Sample is size torch.Size([1, 2])
Log prob of tensor([[ 0.,  0.]]) = tensor([[-0.9189, -0.9189]])
</code></pre>
<p>Shouldn’t the log probability be of a single dimension ? Am I (certainly but what) doing something wrong ?</p>
<p>Thanks a lot !</p>

Mehdi: <p>Never mind, I figured it out !<br>
I think that Normal generates one normal distribution per feature. Hence, you have as many log_probs as you have distributions.<br>
However, in the case of multi-features, here’s what I propose:</p>
<ul>
<li>Either keep Normal, but sum log probs (or multiply probs) to get the probability you seek given multifeatures</li>
<li>Use a Multivariate Normal, which avoids you to gather the probabilites yourself.</li>
</ul>
<p>If anyone has other insights, or if I’ve made a mistake, I’d be delighted to hear it !</p>
<p>Thanks !</p>

Mingming_Qiu: <p>dist.log_prob(a) should have kept the same dimension as a. The essence of dist.log_prob() can be explained as the following example:</p>
<pre><code class="lang-auto">prob = torch.rand(5)
m=Bernoulli(prob)
act = m.sample()

act
tensor([0., 0., 1., 0., 0.])

prob
tensor([0.4880, 0.5403, 0.1671, 0.1158, 0.1695])

m.log_prob(act)
tensor([-0.6694, -0.7773, -1.7889, -0.1230, -0.1857])

torch.log(prob)
tensor([-0.7175, -0.6155, -1.7889, -2.1562, -1.7751])

torch.log(1-prob)
tensor([-0.6694, -0.7773, -0.1829, -0.1230, -0.1857])
</code></pre>
<p>So, why, in your example, for torch.zeros(1,2), you said that the log_prob is a single value instead of being with size (1,2)?</p>
How to reduce the loss in a simple training any further

Jazzy66: <p>I will post my simplified code first:</p>
<pre><code class="lang-auto">import torch
import torch.nn.functional as F
import numpy as np

def argmax(x, axis=-1):
    return F.one_hot(torch.argmax(x, dim=axis), list(x.shape)[axis]).float()

loss_function = torch.nn.MSELoss()

def optimization(V, U, E, beta, learning_rate):
    V = V.cuda()
    U = U.cuda()
    E = E.cuda()

    V_optim = V.detach().clone()
    V_optim.requires_grad = True

    optimizer = torch.optim.Adam([V_optim], lr=learning_rate)

    dif    =  1
    epoch  =  1

    num_epochs = 200
    for epoch in range(num_epochs):
        optimizer.zero_grad()
        Q_R = argmax(U + beta * torch.transpose(V_optim, 0, 1))
        loss = loss_function(V_optim, torch.matmul(Q_R * U, E) + beta * torch.matmul(Q_R, V_optim))
        loss.backward(retain_graph=True)
        optimizer.step()
        
        print(f"Epoch {epoch}, Loss: {loss}")

    return V_optim

if __name__ == "__main__":
    n = 100
    beta = 0.98 
    alpha = 0.03
    delta = 1

    kss = ((1 / beta - (1 - delta)) / alpha) ** (1 / (alpha - 1))
    k = np.linspace(0.5 * kss, 1.4 * kss, n)

    k_reshaped = k.reshape(-1, 1)
    c = k_reshaped ** alpha + (1 - delta) * k_reshaped - k
    c[c &lt; 0] = 1e-11
    c = np.log(c)

    V = (np.log(kss ** alpha - delta * kss) / (1 - beta)) * torch.ones(n, 1, requires_grad=True)
    U = torch.tensor(c, dtype=torch.float32)
    E = torch.ones(n, 1)

    learning_rate = 0.002

    optimal_V = optimization(V, U, E, beta, learning_rate)
    print(optimal_V)
</code></pre>
<p>According to the attached code, the (mean) loss after 200 iterations is</p>
<pre><code class="lang-auto">    Epoch 199, Loss: 3.335637765999877e-09
</code></pre>
<p>But the maximum single loss element in the last iteration result is</p>
<pre><code class="lang-auto">    tensor(0.0006, device='cuda:0', grad_fn=&lt;MaxBackward1&gt;)
</code></pre>
<p>Based on this simple learning I wonder:</p>
<p>(1) is there any solution to reduce the loss, especially the single loss, any further?</p>
<p>(2) is this a problem concerning the choice of loss function type or the format of inputs?</p>
<p>Many thanks.</p>
How to multiply two matrices but only summing only over topK largest values

quang: <p>I have to real-valued matrix matrices A and B with shape (m,n) and (n,p) respectively and a pre-defined constant K. The goal is to calculate the new “topk multiplication” function, where entry (i,j) in the output is calculated as follows:</p>
<ol>
<li>Perform element-wise multiplication between the i-th row of A and the j-th column of B</li>
<li>Summing only top K largest values from the result in step 1).</li>
</ol>
<p>Here, if K = n, then this operation reduces to the standard matrix multiplication. My interest is calculating this topk multiplication function efficiently, especially when the inputs are in batch. An example of the naive implementation can be found below. Thank you!</p>
<pre><code class="lang-auto">import torch

a = torch.rand(2,3,5)
b = torch.rand(2,5,2)
K = 2

out = torch.zeros(2,3,2)

for i in range(2):
    for j in range(3):
        for k in range(2):
            tmp = torch.mul(a[i,j,:], b[i,:,k])
            v, _ = torch.topk(tmp, K)
            out[i,j,k] = v.sum()
</code></pre>

KFrank: <p>Hi Quang!</p>
<aside class="quote no-group quote-modified" data-username="quang" data-post="1" data-topic="184905" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/q/48db29/48.png" class="avatar"> quang:</div>
<blockquote>
<p>The goal is to calculate the new “topk multiplication” function</p>
<pre><code class="lang-auto">for i in range(2):
    for j in range(3):
        for k in range(2):
            tmp = torch.mul(a[i,j,:], b[i,:,k])
            v, _ = torch.topk(tmp, K)
            out[i,j,k] = v.sum()
</code></pre>
</blockquote>
</aside>
<p>If you align the indices correctly and use broadcasting, you can replace<br>
the loops with a single element-wise tensor multiplication:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; print (torch.__version__)
2.0.0
&gt;&gt;&gt;
&gt;&gt;&gt; _ = torch.manual_seed (2023)
&gt;&gt;&gt;
&gt;&gt;&gt; a = torch.rand(2,3,5)
&gt;&gt;&gt; b = torch.rand(2,5,2)
&gt;&gt;&gt; K = 2
&gt;&gt;&gt;
&gt;&gt;&gt; out = torch.zeros(2,3,2)
&gt;&gt;&gt;
&gt;&gt;&gt; for i in range(2):
...     for j in range(3):
...         for k in range(2):
...             tmp = torch.mul(a[i,j,:], b[i,:,k])
...             v, _ = torch.topk(tmp, K)
...             out[i,j,k] = v.sum()
...
&gt;&gt;&gt; outB = (a.unsqueeze (3) * b.unsqueeze (1)).topk (K, dim = 2)[0].sum (dim = 2)
&gt;&gt;&gt;
&gt;&gt;&gt; torch.equal (out, outB)
True
</code></pre>
<p>(You could instead use <code>einsum()</code> to “align the indices,” but using<br>
<code>.unsqueeze()</code> seems a little better to me stylistically.)</p>
<p>Best.</p>
<p>K. Frank</p>

quang: <p>Hi Frank</p>
<p>Your solution worked well, thanks so much!</p>
<p>Quang</p>
"Concretize" the result of calling grad()

csid: <p>Hello!</p>
<p>I was wondering if it would be possible to “concretize” the result of the grad() call. I will explain.</p>
<p>Say we have some function sin(x^2) implemented in pytorch. When I call grad() it computes a graph that ultimately represents the expression cos(x^2)<em>2</em>x . Is it possible to get pytorch to return this expression to me ‘undecorated’ as opposed to as a graph of backward() → backward() calls? e.g. instead return a graph of x → [mul by 2] → [mul by …[cos …[mul by 2 ]]] etc. This way I can pass it around and differentiate it again without worrying about how it was generated (using grad).</p>

soulitzer: <p>What problems do you run into when differentiating it again? The graph will always contain blahbackward calls, and those are the functions that are actually run when you differentiate it - if you do cos(x^2)2x yourself you should expect to get the same graph.</p>

csid: <p>I get an error when calling <code>model(*unpack_inputs())</code><br>
It says <code>RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn</code><br>
even though the original input has <code>.requires_grad_()</code></p>
Torch.onnx.export a fp16 model but get the output tensor fp32

Chace_Young: <p>Hi there，i’ve just upgraded my torch from version 1.12.1 to 2.0.1.<br>
however, i get the problem as mentioned in the title, that the output tensor in the exported onnx become a fp32 one with the others fp16.<br>
the code as follows,<br>
torch.onnx.export(<br>
deploy, opset_version=11, args=(dummy_a, dummy_b), f=‘export.onnx’,<br>
input_names=[“a”, “b”], output_names=[“c”], verbose=False<br>
)</p>
Conv1d: on cuda, increase batch size, forward slower

Yaw_Young: <pre><code class="lang-auto">    import torch
    import datetime
    m = nn.Conv1d(16, 512, 3, stride=2).cuda()
    input = torch.randn(20, 16, 24000).cuda()
    for _ in range(40):
        torch.cuda.synchronize()
        t1 = datetime.datetime.now()
        output = m(input)
        torch.cuda.synchronize()
        t2 = datetime.datetime.now()
        print((t2-t1).microseconds / 1000, 'ms')

</code></pre>
<p>When the batch size = 20, forward takes 12ms.<br>
When I increase the batch size to 60, the forward takes 26ms.<br>
Is this normal? I think when using GPU, as long as the cuda memory is enough, increasing batch size won’t make forward slower.</p>

ptrblck: <aside class="quote no-group" data-username="Yaw_Young" data-post="1" data-topic="184923">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/yaw_young/48/62569_2.png" class="avatar"> Yaw_Young:</div>
<blockquote>
<p>I think when using GPU, as long as the cuda memory is enough, increasing batch size won’t make forward slower.</p>
</blockquote>
</aside>
<p>That’s not the case as you need to move more data around and are increasing the compute. Could you explain why the runtime should be static?</p>

Yaw_Young: <p>My understanding may be naive. I think that GPUs can achieve parallel operations between batches, so increasing the batch size does not increase the time required for an iteration, as long as the GPU memory size allows.</p>

ptrblck: <p>This is wrong, since CUDA kernels are written in a way to utilize all compute resources even for smaller batch sizes.</p>

Yaw_Young: <p>Got it, thank you ~!</p>
Torch.fft.rfft consumes too high gpu memory

pepper8362: <p>Hi,</p>
<p>I have the following code:</p>
<pre><code class="lang-auto">import torch

def get_memory_mb(data):
    return data.numel() * data.element_size() / (1024*1024)

n_fft = 153999
with torch.no_grad():
    rir = torch.randn(64, 6, 10, 10000).float().cuda()
    rir_fft = torch.fft.rfft(rir, n=n_fft, dim=-1)
    print("rir memory should be: {} MB".format(get_memory_mb(rir)))
    print("rir_fft memory should be: {} MB".format(get_memory_mb(rir_fft)))
</code></pre>
<p>The output is as follows:</p>
<pre><code class="lang-auto">rir memory should be: 146.484375 MB
rir_fft memory should be: 2255.859375 MB
</code></pre>
<p>According to the output, I would expect the code consumes no more than 3GB GPU memory. But it actually consumes around 16 GB GPU memory. Is there anything wrong in my way of using <code>torch.fft.rfft</code> without auto grad? And is there anyway to reduce the memory usage of this function (I’m trying to use it with GPU as an efficient way to calculate 1d convolution between two large 1d tensors)?</p>

pepper8362: <p>I was able to alleviate the problem using <code>torch.cuda.empty_cache()</code> to release some memory after fft</p>
'MSELossBackward0' returned nan values in its 0th output

Solatis34: <p>I have some difficulties with this error.<br>
It happens during the first epoch after 62 batch_idx. When I debug my code, it says avg_cost becomes nan just after batch_idx is 62.</p>
<p>Here is my code</p>
<pre><code class="lang-auto">def train_model(model, train_df, num_epochs = None, lr = None, verbose = 20, patience = 10):
    criterion = nn.MSELoss(reduction='sum')
    optimizer = optim.Adam(model.parameters(), lr = learning_rate)
    nb_epochs = num_epochs

    train_hist = np.zeros(nb_epochs)

    for epoch in range(nb_epochs):
        avg_cost = 0
        total_batch = len(train_df) 

        for batch_idx, samples in enumerate(train_df):
            x_train, y_train = samples

            model.reset_hidden_state()

            # H(x)
            outputs = model(x_train)

            # cost
            loss = criterion(outputs, y_train)

            optimizer.zero_grad()
            loss.backward()
            optimizer.step()

            avg_cost += loss / total_batch

        train_hist[epoch] = avg_cost

        torch.autograd.set_detect_anomaly(True)

        if epoch % verbose == 0:
            print('Epoch: ', '%03d' % (epoch), 'train loss : ', '{: .4f}'.format(avg_cost))

        if (epoch % patience == 0) &amp; (epoch != 0):
            
            if train_hist[epoch-patience] &lt; train_hist[epoch]:
                print('\nEarly Stopped')
                break

    return model.eval(), train_hist
</code></pre>
<p>I think something is wrong with normalization, (I used MinMaxScalar() function to normalize my input data) but I don’t know the real cause of error.</p>
<p>How do you suggest me to delete my error?</p>
AssertionError: Mismatch in batch_size and euler_angles shape

REDHWAN_AL-GABRI: <p>I tried to compute the Euler angles from the quaternion using this code:</p>
<pre><code class="lang-auto">def compute_euler_angles_from_quaternion(quaternions, sequence='xyz'):
    """
    Convert a batch of quaternions to Euler angles.

    Args:
        quaternions: Tensor of shape (batch_size, 4). Batch of quaternions.
        sequence: String specifying the rotation sequence. Default is 'xyz'.

    Returns:
        euler_angles: Tensor of shape (batch_size, 3). Batch of Euler angles.
    """
    batch_size = quaternions.shape[0]
    q = quaternions.detach().cpu().numpy()  # Convert to NumPy array

    rotations = Rotation.from_quat(q)
    euler_angles = rotations.as_euler(sequence, degrees=False)

    euler_angles = torch.tensor(euler_angles, device=quaternions.device)
    euler_angles = euler_angles.view(batch_size, 3)

    return euler_angles
</code></pre>
<p>But I got this issue when  batch sizes =  1 or 2, or 16:</p>
<pre><code class="lang-auto">RuntimeError: shape '[4, 3]' is invalid for input of size 3
</code></pre>
<p>I tried to fix it using this code as I found some solution <a href="https://stackoverflow.com/questions/60439570/beginner-pytorch-runtimeerror-shape-16-400-is-invalid-for-input-of-size/60439832#60439832" rel="noopener nofollow ugc">here</a>:</p>
<pre><code class="lang-auto">def compute_euler_angles_from_quaternion(quaternions, sequence='xyz'):
    batch_size = quaternions.shape[0]  # Ensure that batch_size is correctly calculated
    print("Batch size:", batch_size)

    q = quaternions.detach().cpu().numpy()  # Convert to NumPy array

    rotations = Rotation.from_quat(q)
    euler_angles = rotations.as_euler(sequence, degrees=False)
    print("Shape of euler_angles:", euler_angles.shape)

    if batch_size == 1:
        euler_angles = euler_angles.reshape(1, -1)  # Reshape single quaternion to (1, 3)

    euler_angles = torch.tensor(euler_angles, device=quaternions.device)

    assert batch_size == euler_angles.shape[0], "Mismatch in batch_size and euler_angles shape"
    euler_angles = euler_angles.view(batch_size, 3)  # Reshape to [batch_size, 3]

    return euler_angles
</code></pre>
<p>I got this issue:</p>
<pre><code class="lang-auto">Batch size: 4
Shape of euler_angles: (3,)
Traceback (most recent call last):
  File "test_quat.py", line 147, in &lt;module&gt;
    euler = utils.compute_euler_angles_from_quaternion(
  File "/home/redhwan/2/HPE/quat/utils.py", line 318, in compute_euler_angles_from_quaternion
    assert batch_size == euler_angles.shape[0], "Mismatch in batch_size and euler_angles shape"
AssertionError: Mismatch in batch_size and euler_angles shape
</code></pre>
<p>I would like the model to work at any batch size.<br>
Please help.</p>
Writing a compiler for PyTorch Models

Moe: <p>I am currently working with a custom accelerator for machine learning models. Currently models are trained in PyTorch to get the weights and then implemented using a c driver for the accelerator. I am interested in writing a compiler that consumes a PyTorch model and outputs driver calls. My question is whether torch script or another PyTorch IR provides a sufficient API that exposes the tensor operations so I can iterate over them into function calls. Any advice about feasibility or implementation would be highly useful.</p>
Augmentations Used in Training Various Models in Timm: Where to Find Them?

PetrDvoracek: <p>I recently came across the paper, <a href="https://arxiv.org/abs/2110.00476" rel="noopener nofollow ugc">“ResNet Strikes Back: An Improved Training Procedure in Timm”</a>. This paper describes the improved methodology for training ResNets, which notably includes the use of specific <strong>data augmentations</strong>.</p>
<p>My question is <strong>whether this training procedure, especially the augmentation techniques, is also employed for other models within the Timm library</strong>. The models I’m particularly interested in are <em>resnet*d, efficientnet_b*, convnext_*, vit_*, swin_*, and mobilenetv3_small_*</em>.</p>
<p>Could anyone please direct me to <strong>relevant publications,</strong> or s<strong>ections within the Timm codebase</strong>, which detail the specific augmentations employed in training these aforementioned models?</p>
<p>Thank you in advance for your assistance!</p>
Implementing torchvision focal loss

hossein: <p>Hi,<br>
I have two questions regarding the focal loss in torchvision library (<a href="https://pytorch.org/vision/main/_modules/torchvision/ops/focal_loss.html#sigmoid_focal_loss" rel="noopener nofollow ugc">focal loss</a>)</p>
<ol>
<li>Can I use it for text classification?<br>
if yes:</li>
<li>I have an imbalanced dataset in which class 0 has 100 examples and class 1 has 30. How should I set the alpha parameter?</li>
</ol>
<p>Thanks</p>
Autograd - adaptation in dimension 2 and more

Stephane: <p>Dears,<br>
I am trying to solve a mathmatical problem through neural network.</p>
<p>Let’s have a fonction u whose second derivative is called f.</p>
<p>The loss function will compare the second derivative of the neuronal fonction unn (aiming at imitating u) and f.<br>
End of the process is to have unn very near to u (we implement an adequation percentage between both curves to assess the results).</p>
<p>This program works fine in dim 1 (with the function autograd) but in dim 2 and in dim 3, it does not work due (probably) to the misuse of the autograd function.</p>
<p>In annex, I put a picture of the part of the program which does not work, and can share if need be the full  program as well (.ipynb).</p>
<p>Thank you if anyone can help here.</p>
<p>Best</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/3/f3815ecb850b31629d559d66173f47f41aeafdf6.png" data-download-href="https://discuss.pytorch.org/uploads/default/f3815ecb850b31629d559d66173f47f41aeafdf6" title="MicrosoftTeams-image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/3/f3815ecb850b31629d559d66173f47f41aeafdf6.png" alt="MicrosoftTeams-image" data-base62-sha1="yK9coode8UYxPQgtwpSDff4mAD4" width="690" height="303" data-dominant-color="F0F1F1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">MicrosoftTeams-image</span><span class="informations">1131×498 25.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

Stephane: <p>I see absolutely no answer.<br>
Does it mean that this is not achievable according to you ?<br>
Would appreciate someone’s thoughts</p>

Stephane: <p>Dears,<br>
I just received in the meantime information that Pytorch autograd is only designed in dimension 1, reason why it cannot work in dimension 2.<br>
So this post is not useful anymore.<br>
Thank you very much</p>
Detectron2 confidence score for negative classes

Jeff-Gui: <p>I’m using detectron2 maskRCNN for semantic segmentation. The standard output of model prediction only includes the confidence of predicted class. Since I’m going to smooth classification across time, it will be helpful to know the confidence/probability of other classes. I wonder if this is feasible.</p>

hamzagorgulu: <p>Trying to resolve the same problem. Here there is a similar session: <a href="https://discuss.pytorch.org/t/how-to-get-mask-scores-for-image-segmentation-in-detectron2/84209/6" class="inline-onebox">How to get mask scores for image segmentation in Detectron2? - #6 by dangmanhtruong1995</a></p>
Adding MSE and Cross entropy loss

Salman_Abbasi: <p>Hi</p>
<p>I have one question about the regression and classification loss. I have trained a network which outputs denoised data and its corresponding events picked in binary(0 or 1). At the moment I am using only MSE loss and then set some threshold criteria for binary data, to convert to 0 or 1.</p>
<p>Is it possible that I have two types of losses(one classification and other regression) added and backpropagated for this type of network? If not, what are the other alternatives?</p>
<p>Thanks<br>
Sal</p>

ptrblck: <p>Yes, you can calculate and sum different losses, which can then be used to compute the gradients and train the model.</p>

Salman_Abbasi: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> . Thank you for the helpful response.</p>
How to Gather Prediction Results from Different Sub-Processes?

UgurA: <p>I am currently working with a Distributed Data Parallel (DDP) setup for deep learning models. You can find the details of my setup at [this link] (<a href="https://discuss.pytorch.org/t/cant-distribute-data-to-all-gpus-with-ddp/184749" class="inline-onebox">Can't distribute data to all GPUs with DDP</a>) . In my setup, I use a Trainer object to make predictions for batches and then merge all the predictions into one convenient dictionary object. Each prediction is associated with a unique ID, which is why I chose to use a dictionary.</p>
<p>Once I have created the dictionary containing all the predictions for the dataset, I proceed to calculate performance metrics. However, I encounter an issue with the DDP setup: the predictions are split into two distinct dictionaries, each residing on a different GPU. This poses a challenge, as I need to gather all the predictions into a single dictionary to make accurate performance metric calculations.</p>
<p>I am seeking guidance on how to efficiently merge these two separate dictionary objects that reside on different GPUs into one dictionary.  The merged dictionary can be reside on CPU.  Any insights or solutions to address this problem would be greatly appreciated!</p>

kumpera: <p>You can exchange them between all ranks by using all_gather_object:<br>
<a href="https://pytorch.org/docs/master/distributed.html#torch.distributed.all_gather_object" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/master/distributed.html#torch.distributed.all_gather_object</a></p>
DeepLabV3+ Transfer Learning

bjaeger: <p>Does anybody has a good understanding of the deeplabv3+ module or has some experience when applying transfer learning on semantic segmentation tasks which are based on the deeplabv3+ model?</p>
<p>The deeplabv3+ module consist of three main modules: backbone, ASPP and decoder</p>
<p>Does it make sense to freeze layers of the ASPP or decoder module during training? Or is it advisable to only freeze layers/layergroups of the backbone (e.g. resnet101)?</p>
Torch to onnx supported operators dict

daniellevi: <p>Hi<br>
I would like to create a script which takes the name of a specific torch operator, and tells me if this operator is supported by onnx. That’s important for me that I could do it using internal torch.onnx APIs, without using the list provided by the torch’s website. The script should be up to date according to the torch version I’m using in the script, so managing such a dict manually according to the website list is impractical.</p>
<p>Is there such an API?<br>
I tried to explore the implementation of torch.onnx.export, but it contains lots of hidden APIs and I couldn’t find something good enough.</p>

daniellevi: <p>Update- I found such an internal method, but I also need to make the exact conversion from torch operator’s name to the converted onnx operator’s name.</p>
Cpu vs gpu (cuda) segmentation results difference

kabbas570: <p>Hello PyTorch Community,<br>
I am inferring a pre-trained model trained on some segmentation task.<br>
If I evaluate cuda (GPU) (on my GPUs cluster), I get different results than if I do it on my local machine on Windows CPU.<br>
Here are some visual results; I found two reasons behind this, but the visual results should vary that much; not sure.</p>
<ul>
<li>Floating point precisions</li>
<li>Difference in the execution order of the operations</li>
</ul>
<p>Any clue will be appreciated.<br>
Thank You<br>
Cheers<br>
Abbas</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/0/80e1f2ec7515f5d4385cea3d7189e85f0f77296f.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/80e1f2ec7515f5d4385cea3d7189e85f0f77296f" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/0/80e1f2ec7515f5d4385cea3d7189e85f0f77296f_2_690x119.jpeg" alt="image" data-base62-sha1="io9cZuX5hxld0Ef9FZMmPsLhm5V" width="690" height="119" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/0/80e1f2ec7515f5d4385cea3d7189e85f0f77296f_2_690x119.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/8/0/80e1f2ec7515f5d4385cea3d7189e85f0f77296f_2_1035x178.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/8/0/80e1f2ec7515f5d4385cea3d7189e85f0f77296f_2_1380x238.jpeg 2x" data-dominant-color="A7A28C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1509×261 92.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

eqy: <p>This is potentially expected, especially if the model was trained on CPU and then moved to GPU. If the GPUs are Ampere (sm80) or newer, you might want to check if setting <code>NVIDIA_TF32_OVERRIDE=0</code> changes the results somewhat (this env var turns off the use of the TF32 dtype internally).</p>
<p>If there is a serious degradation of accuracy and the above env var doesn’t meaningfully improve your results, I would check the layers used by your model one-by-one to see where the differences are coming from.</p>

kabbas570: <p>Hello, thanks for your reply.<br>
The model is trained on GPU and then tested on both CPU and GPU.<br>
The GPU is A100.<br>
Thanks for the suggestions, will try it.<br>
Cheers<br>
Ababs</p>
Applying convolution across batched sequences

Yugen_Omer_Korat: <p>Hello,</p>
<p>I have batched and sequenced 2d data, meaning I am working with tensors of size (batch_size, max_seq_len, d1, d2). I want to use convolution to reduce it to (batch_size, max_seq_len, new_dim). The layer that does this doesn’t care about sequences, meaning all elements of the sequence are passed through the same weights. The issue is that Conv1d and Conv2d do not accept 4d inputs, so I would have to first reshape them into (batch_size * max_seq_len, d1, d2).</p>
<p>My question is, is there a more efficient way to do this? Or is it ok? The worry is that batch_size*max_seq_len ends up equalling 2400, which seems to big for a batch and it might cause my computation to run too slowly.</p>

eqy: <p>I would just attempt to run your computation as-is and check if the throughput you are getting is acceptable given e.g., your hardware’s peak FLOP throughput. I would be more concerned by the effectively small channel dimension if you intend to interpret your data as (N, 1, d1, d2), as this would be a fairly uncommon setup.</p>

Yugen_Omer_Korat: <p>There is no (N, 1, d1, d2). I squeeze it to (N, d1, d2) where N=batch_size*max_seq_len. (N, 1, d1, d2) is impossible bc like I said Conv1d/2d doesn’t accept 4d input, which is why I have to squeeze in the first place.</p>
What is torch.renorm

JBoRu: <p>I’m confused about the doc of torch.renorm and can you tell me what use of renorm and how the example in doc as follows compute.</p>
<pre><code class="lang-auto">&gt;&gt;&gt; x
tensor([[ 1.,  1.,  1.],
        [ 2.,  2.,  2.],
        [ 3.,  3.,  3.]])
&gt;&gt;&gt; torch.renorm(x, 1, 0, 5)
tensor([[ 1.0000,  1.0000,  1.0000],
        [ 1.6667,  1.6667,  1.6667],
        [ 1.6667,  1.6667,  1.6667]])
</code></pre>

arunppsg: <p>I am also having trouble understanding <code>torch.renorm</code>. Here is an example which confuses me:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch
&gt;&gt;&gt; x = torch.randn(3, 3)
&gt;&gt;&gt; torch.norm(x, p=2, dim=1)
tensor([2.2026, 2.2433, 1.0954])
&gt;&gt;&gt; y = torch.renorm(x, p=2, dim=1, maxnorm=2)
&gt;&gt;&gt; torch.norm(y, p=2, dim=1)
tensor([2.0249, 1.7163, 0.8764])
</code></pre>
<p>Since I have specified <code>maxnorm</code> to be 1, I expect all values of y norm to be less than or equal to 1 along <code>dim=1</code> for <code>p=2</code> but in this case, the first value is <code>2.0249</code>. But if I do</p>
<pre><code class="lang-auto">&gt;&gt;&gt; torch.norm(y, p=2, dim=0)
tensor([2.0000, 0.8703, 1.7483])
</code></pre>
<p>all values are less than the maxnorm. How does <code>torch.renorm</code> works and along what dimension does it renormalizes the array?</p>
<p>Something like a mathematical formulation of the <code>renorm</code> will be super helpful.</p>

arunppsg: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> can this be a bug?</p>

ptrblck: <p>No, I don’t think it’s a bug and it seems you are using the <code>dim</code> argument wrong in <code>renorm</code>:</p>
<pre data-code-wrap="python"><code class="lang-python">x = torch.randn(3, 3)
x.norm(p=2, dim=1)
# tensor([1.2906, 2.8629, 1.7181])
torch.renorm(x, p=2, dim=0, maxnorm=2).norm(p=2, dim=1)
# tensor([1.2906, 2.0000, 1.7181])

x = torch.randn(3, 3) * 5
x.norm(p=2, dim=1)
# tensor([14.8308,  5.6469,  4.5213])
torch.renorm(x, p=2, dim=0, maxnorm=2).norm(p=2, dim=1)
# tensor([2., 2., 2.])
</code></pre>
<p>as it’s:</p>
<blockquote>
<p>the dimension to slice over to get the sub-tensors</p>
</blockquote>

arunppsg: <p>I also made a numpy implementation of renorm to understand better about it.</p>
<pre><code class="lang-auto">import numpy as np

def renorm(x, p, dim, maxnorm):
    x_view = np.rollaxis(x, dim, 0)
    n = x.shape[dim]
    norms = []
    for i in range(n):
        norms.append(np.linalg.norm(x_view[i,:], ord=2))

    factors = []
    for norm in norms:
        if norm &gt; maxnorm:
            factors.append(maxnorm/norm)
        else:
            factors.append(1)
    factors = np.array(factors)
    return x * factors.reshape(-1, 1), factors
</code></pre>