akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to create a tensor with shape of Nones?

Asya: <p>I want to create tensor of shape=(None, 32, 32, 3)<br>
how can I do so?</p>

Samuel_Bachorik: <p>Tell what are you trying to do please.</p>

Asya: <p>im trying to create an input tensor in pytorch similar to <code>Input(shape=(32,32,3))</code> in tesorflow</p>

ptrblck: <p>You don’t need to use placeholder variables in PyTorch and can directly pass tensors to the model without specifying their shapes beforehand (similar to what you would be using in numpy).</p>

Cindy5: <p>I need to create a temporary variable (of tensor type) for … K.zeros throws an error saying it cannot <em>create a tensor with shape None</em></p>
<blockquote>
<p>.</p>
</blockquote>

ptrblck: <p>You can create an empty tensor via <code>x = torch.tensor([])</code>, but I still don’t understand why you would need to create such a tensor, as placeholders are not used in PyTorch and you can directly use valid tensors instead so could you explain your use case a bit more, please?</p>

Cindy5: <aside class="quote no-group quote-modified" data-username="Cindy5" data-post="5" data-topic="132421">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/dfb087/48.png" class="avatar"> Cindy5:</div>
<blockquote>
<p>I need to create a temporary variable (of tensor type) for K<a href="https://www.dgcustomerfirst.page/" rel="noopener nofollow ugc">.</a>zeros throws an error saying it cannot <em>create a tensor with shape None</em></p>
</blockquote>
</aside>
<p>Thanks for the information</p>

miandrew: <aside class="quote no-group" data-username="ptrblck" data-post="4" data-topic="132421" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>You don’t need to use placeholder variables in PyTorch and can directly pass tensors to the model without specifying their shapes beforehand (similar to what you would be using in numpy)<a href="https://www.budgetsavvydiva.com/2024/08/unlocking-the-benefits-of-dihydroberberine-a-game-changer-for-health-and-wellness/" rel="noopener nofollow ugc">.</a></p>
</blockquote>
</aside>
<p>Thanks for best reply.</p>
Adding Variable in Tensor Class

sabiha_afroz: <p>I wanted to add an ID variable in Tensor class (<a href="https://github.com/pytorch/pytorch/blob/main/torch/_tensor.py" class="inline-onebox" rel="noopener nofollow ugc">pytorch/torch/_tensor.py at main · pytorch/pytorch · GitHub</a>). I overloaded an init function inside the class Tensor.</p>
<pre><code class="lang-auto">class Tensor(torch._C.TensorBase):
    def __init__(self, other):
        super().__init__()
        print("Initializing Tensor...")
        self.ts_id = random.randint(0, 3000)
        print(f"Tensor ID: {self.ts_id}")
</code></pre>
<p>I observed that only Parameter tensors are only invoking the <strong>init</strong> method. For intermediate tensors, they are not calling this init method. My question is do intermediate tensors use another Tensor class for initialization?</p>
Deepspeed flops profiler for llama-3-8B model on compile mode

Arunima_Ghosh: <p>Hi all,<br>
I want to find out the total number of flops of an inference flow of llama-3-8B model in compile mode using deepspeed flops profiler. I am using the following code for this purpose:</p>
<p>model.generate = torch.compile(model.generate,backend=“aot_eager”)<br>
prof = FlopsProfiler(model)<br>
prof.start_profile()<br>
input_ids= tokenizer(batch_sentences,truncation=True,padding=“max_length”,max_length=256, return_tensors=“pt”).to(device=“cpu”)<br>
input_ids = input_ids[‘input_ids’]<br>
start_time1 = time.time()<br>
with torch.no_grad():<br>
outputs = model.generate(input_ids,generation_config=generation_config)<br>
end_time1 = time.time() <span class="hashtag-raw">#time</span> taken to generate inference</p>
<pre><code>prof.stop_profile()
</code></pre>
<p>tot_flops = prof.get_total_flops(as_string=False)<br>
print(tot_flops)</p>
<p>If I print this tot_flops variable to get the total number of flops, I am getting a value similar like 1234<em>56i</em>87j instead of a numeric number. Can you please help me to understand what is wrong in my code? How to get the total number of flops?</p>
Passing a SparseTensor object to an optimzer

alireza378: <p>So I’m just playing around with pytorch-sparse with some toy examples to get a sense of it as there isn’t a complete documentation for that.<br>
Suppose that I have a sparse matrix A, and given two vector x and b. I know Ax=b.<br>
Given b and x and the exact location of nnz in A, I want to find A through SGD, here is the pseudocode I wrote for that and seems it works.</p>
<pre><code class="lang-auto">import numpy as np
from torch_sparse import SparseTensor
import torch
n = 10
lr = 0.01
row = np.random.choice(n, n, replace=True)
col = np.random.choice(n, n, replace=True)
data = np.random.choice(np.arange(1, n + 1), n, replace=True)
A_ground_truth = SparseTensor(row = torch.Tensor(row).to(torch.long), col = torch.Tensor(col).to(torch.long), value = torch.Tensor(data).to(torch.float), sparse_sizes=(n, n))
A_approx = SparseTensor(row = torch.Tensor(row).to(torch.long), col = torch.Tensor(col).to(torch.long), sparse_sizes=(n, n)).requires_grad_(True) # This is the matrix A that we want to learn
x = torch.randn(n,1) # This is the vector x that we have, we know Ax = b
b = A_ground_truth.matmul(x) # This is the vector b that we have, we know Ax = b
for i in range(10000):
    output = A_approx.matmul(x)
    l = torch.norm(b - output)
    l.backward()
    row,col,value = A_approx.coo()
    print(f"Epoch: {i+1}, Loss: {l.item():.3f}")
    newvalue = value - lr*value.grad
    A_approx = SparseTensor(row = torch.Tensor(row).to(torch.long), col = torch.Tensor(col).to(torch.long), value=newvalue.detach(), sparse_sizes=(n, n)).requires_grad_(True)
</code></pre>
<p>However, I have some questions regard that:</p>
<ol>
<li>As you can see, in each epoch, I create <code>A_approx</code> again from the updated values, are there any solution to update <code>A_approx</code> values on the fly?</li>
<li>What if I want to use an optimizer in <code>torch.optim</code> like <code>torch.optim.ADAM</code>? How can I pass the <code>val</code> in <code>A_approx</code> to the optimizer? Should I pass <code>val</code> or I can pass the whole <code>SparseTensor</code> object? Could you please provide me an snippet for that?</li>
</ol>

ptrblck: <p>You can check <a href="https://pytorch.org/docs/stable/generated/torch.optim.SparseAdam.html#torch.optim.SparseAdam"><code>optim.SparseAdam</code></a> which accepts sparse parameters.</p>
Torch.cholesky_solve cannot be called by torch.linalg.cholesky_solve

Kevin96: <p>Hi,</p>
<p>It looks like <code>torch.cholesky_solve</code> is not implemented inside <code>torch.linalg</code>. Namely, the function cannot be called via <code>torch.linalg.cholesky_solve</code>. Given that other functions related to Choleksy decomposition have been moved to <code>torch.linalg</code> (e.g., <code>torch.cholesky</code> is deprecated in favor of <code>torch.linalg.cholesky</code>), what are the motivations to keep the Cholesky solve inside <code>torch</code>?</p>

ptrblck: <p><a href="https://github.com/pytorch/pytorch/issues/61658">This issue</a> tracks the namespace change, but seems to be stale.</p>
Need tensor subclass support in torch.export

qiqixiao: <p>Hi. I’m using torch.export to export a model with UInt4Tensor from torchao, and I see this error:</p>
<pre><code class="lang-auto">aot_export is not currently supported with traceable tensor subclass.
</code></pre>
<p>Wondering what is the plan to support tensor subclass with torch.export and any suggestion for my use case? Thanks</p>
<p>Here is my environment and reproduce script:<br>
torch==2.4.0<br>
torchao==0.1</p>
<pre><code class="lang-auto">import torch
import numpy as np
from torchao.dtypes.uint4 import UInt4Tensor


class ExampleModel(torch.nn.Module):
    def __init__(self):
        """Init"""
        super().__init__()
        x_uint8 = torch.randint(0, 16, (4, 8)).to(torch.uint8)
        self.x = UInt4Tensor.from_unpacked(x_uint8)

    def forward(self, input: torch.Tensor) -&gt; torch.Tensor:
        """Invoke."""
        transposed = self.x.view(8, 4).to(torch.uint8)
        return torch.add(input, transposed)


def main():
    input_tensor = torch.randn(8, 4)
    model = ExampleModel()
    with torch.no_grad():
        exported_program = torch.export.export(
            model.eval(), args=(), kwargs={"input":input_tensor},
        )
    print("===exported_program====")
    print(exported_program)

if __name__ == "__main__":
    main()
</code></pre>
`tolist` vs `numpy` with `requires_grad=True`

jakelevi1996: <p>What is the rationale for allowing the <code>t.tolist()</code> method to be called on a tensor <code>t</code> with <code>requires_grad=True</code>, but not <code>t.numpy()</code>? I’d have thought it would make sense to allow both or neither, but it doesn’t make sense to me why one would be allowed and not the other. Example:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch

t = torch.arange(5, dtype=torch.float32, requires_grad=True)
print(t, type(t), type(t[0]))
# &gt;&gt;&gt; tensor([0., 1., 2., 3., 4.], requires_grad=True) &lt;class 'torch.Tensor'&gt; &lt;class 'torch.Tensor'&gt;
print(t.tolist(), type(t.tolist()), type(t.tolist()[0]))
# &gt;&gt;&gt; [0.0, 1.0, 2.0, 3.0, 4.0] &lt;class 'list'&gt; &lt;class 'float'&gt;
print(t.numpy())
# &gt;&gt;&gt; RuntimeError: Can't call numpy() on Tensor that requires grad. Use tensor.detach().numpy() instead.
</code></pre>

KFrank: <p>Hi Jake!</p>
<aside class="quote no-group" data-username="jakelevi1996" data-post="1" data-topic="207578" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jakelevi1996/48/75757_2.png" class="avatar"> jakelevi1996:</div>
<blockquote>
<p>What is the rationale for allowing the <code>t.tolist()</code> method to be called on a tensor <code>t</code> with <code>requires_grad=True</code>, but not <code>t.numpy()</code>?</p>
</blockquote>
</aside>
<p>I speculate that it’s because <code>t.numpy()</code> shares memory with <code>t</code>, but<br>
that autograd can’t track changes to <code>t</code> that occur through <code>t.numpy()</code>.<br>
My thinking would be that this is sufficiently error-prone that it’s not<br>
supported.</p>
<p>Here we see how one can modify <code>t</code> through its shared memory:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'2.4.0'
&gt;&gt;&gt; t = torch.arange (5.)
&gt;&gt;&gt; tn = t.numpy()
&gt;&gt;&gt; tn[1] = 99
&gt;&gt;&gt; t
tensor([ 0., 99.,  2.,  3.,  4.])
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

jakelevi1996: <p>Makes sense, thank you <a class="mention" href="/u/kfrank">@KFrank</a> !</p>
Calculate Hessian with Jacobian

abc1: <p>I have a function that produces 3 outputs where each output is a function of x, y, z. I want to calculate the Hessian of that function with respect to inputs x, y, z. The built-in <a href="https://pytorch.org/docs/stable/generated/torch.autograd.functional.hessian.html" rel="noopener nofollow ugc">Hessian</a> takes in a function that produces a single output, so I’m thinking of calling Jacobian twice. However, the first time I call the Jacobian, the inputs are passed in and I get a tensor of numbers, and taking the Jacobian of that gets all 0s. Is there a way to not sub in the numbers the first time Jacobian is called or any other ways to do this? Thanks!</p>

AlphaBetaGamma96: <p>Hi <a class="mention" href="/u/abc1">@abc1</a>,</p>
<p>Can you share a minimal reproducible example?</p>

KFrank: <p>Hi abc!</p>
<aside class="quote no-group" data-username="abc1" data-post="1" data-topic="207576" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/abc1/48/63567_2.png" class="avatar"> abc1:</div>
<blockquote>
<p>The built-in <a href="https://pytorch.org/docs/stable/generated/torch.autograd.functional.hessian.html" rel="noopener nofollow ugc">Hessian</a> takes in a function that produces a single output</p>
</blockquote>
</aside>
<p><a href="https://pytorch.org/docs/2.4/generated/torch.func.hessian.html" rel="noopener nofollow ugc">torch.func.hessian</a> takes a <code>func</code> that can return a tuple.  I believe<br>
that this will do what you want.</p>
<p>Best.</p>
<p>K. Frank</p>
Pytorch model React Native

Suji: <p><span class="mention">@ReactMethod</span><br>
public void testModelLoading(String imagePath, String envName,final Callback successCallback, final Callback errorCallback) {<br>
try {<br>
String modifiedPath = imagePath.replace(“file://”, “”);<br>
String modelFilePath = assetFilePath(reactContext, “my_model_lite.ptl”);<br>
Log.i(“FCNVImagePyTorch”, “Model trying”);<br>
Module module = Module.load(modelFilePath);<br>
Log.i(“FCNVImagePyTorch”, “Model loaded successfully”);<br>
successCallback.invoke(“Model loaded successfully”);<br>
} catch (Exception e) {<br>
Log.e(“FCNVImagePyTorch”, "Exception: " + e.getMessage());<br>
errorCallback.invoke("Exception: " + e.getMessage());<br>
}<br>
}</p>
<p>I am trying to load a model which is of 4.6mb and app got crashed immediately after loading.<br>
When  I used adb logcat --buffer=crash to check logs</p>
<p>I got below</p>
<p>: Abort message: 'terminating with uncaught exception of type c10::Error: isTuple()INTERNAL ASSERT FAILED at “…/…/…/…/src/main/cpp/libtorch_include/arm64-v8a/ATen/core/ivalue_inl.h”:1306, please report a bug to PyTorch. Expected Tuple but got String<br>
08-05 12:56:20.965  4347  4347 F DEBUG   : Exception raised from toTuple at …/…/…/…/src/main/cpp/libtorch_include/arm64-v8a/ATen/core/ivalue_inl.h:1306 (most recent call first):</p>
<p>But another .ptl file of size 5 mb worked perfectly without any error. When I checked in Python. Both models loaded perfectly. Only in react native environment one is loading and the other one is failing with above error. Please help to resolve this issue</p>
Model forward pass in AMP gives NaN

MyNameIsArko: <p>Code:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
import torch.nn as nn

img = torch.randn(4, 3, 256, 256)
small_img = torch.randn(4, 3, 64, 64)

class SuperResolution(nn.Module):
    def __init__(self):
        super().__init__()

        self.m1 = nn.Sequential(
            nn.Conv2d(3, 128, 3, padding=1, bias=False),
            nn.ReLU(True),
            nn.Conv2d(128, 256, 3, padding=1, bias=False),
            nn.ReLU(True),
            nn.Conv2d(256, 512, 3, padding=1, bias=False),
            nn.ReLU(True),
        )
   
    def forward(self, x):
        print('x:', x.mean())
        z = self.m1(x)
        print('m1:', z.mean())
        print('\n====================\n')
        return x

device = torch.device('cuda')  # static for debugging
model = SuperResolution().to(device)
small_img = small_img.to(device)

print('Normal Precision\n')
output = model(small_img)

print('Automatic Mixed Precision\n')
with torch.autocast('cuda', enabled=True):
    output = model(small_img)
</code></pre>
<p>Output:</p>
<pre><code class="lang-auto">Normal Precision

x: tensor(0.0062, device='cuda:0')
m1: tensor(0.0014, device='cuda:0', grad_fn=&lt;MeanBackward0&gt;)

====================

Automatic Mixed Precision

x: tensor(0.0062, device='cuda:0')
m1: tensor(nan, device='cuda:0', dtype=torch.float16, grad_fn=&lt;MeanBackward0&gt;)

====================
</code></pre>
<p>Basically, with full precision it properly gives value as output of the network, but in autocast it gives nan.</p>
<p>Specs:</p>
<pre><code class="lang-auto">Fedora 40
Ryzen 7 4800H
NVIDIA GeForce GTX 1660 Ti
</code></pre>
<p>Nvidia-smi:</p>
<pre data-code-wrap="bash"><code class="lang-bash">+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 555.58.02              Driver Version: 555.58.02      CUDA Version: 12.5     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce GTX 1660 Ti     Off |   00000000:01:00.0  On |                  N/A |
| N/A   53C    P8              6W /   80W |    1675MiB /   6144MiB |     19%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
</code></pre>
<p>Pytorch installed with mamba (conda equivalent) as:</p>
<pre data-code-wrap="bash"><code class="lang-bash">mamba create --name torch python=3.11.9
mamba activate torch
mamba install pytorch torchvision torchtext torchaudio pytorch-cuda=12.4 -c pytorch -c nvidia
</code></pre>
<p>Environment created and pytorch installed today (05.08.2024)</p>
1Torch was not compiled with flash attention

Jack_Archer: <p>I am using the latest 12.1 version of Pytorch. If anyone knows how to solve this, please just take a couple of minutes out of your time to tell me what to do.</p>
<p>Update: It ran again correctly after recompilation. This forum is awful.</p>

wangqidong201901: <p>hello.bro. Did you solved the problems?Or how you recompile your Torch?</p>
<p>你是怎么样重新编绎Torch的？多谢哈。</p>
Gradient computation fails for 'HRNet_FeatureExtractor' due to an inplace operation

mohammadalihumayun: <p>Using latest torch version when i try to train HRNet_FeatureExtractor from modules.feature_extraction, i get following error</p>
<p><code>one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [4, 512, 4, 50]], which is output 0 of ReluBackward0, is at version 1; expected version 0 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. </code></p>
Reducing RAM usage when dealing with very large Tensor

JosephSav: <p>I am trying to build a function that does as follows:<br>
“”"<br>
Calculates a 3D tensor with the minimum distance from each pixel to data.</p>
<p>Inputs:<br>
* points: a numpy array of 2D coordinates and point values, normalized to<br>
be in the range [0,1]. The expected shape is [B, P, 2].<br>
* res: the resolution of the output tensor.<br>
Returns:<br>
A res x res square tensor with floating point values corresponding to the<br>
euclidean distance to the closest point in points. The return tensor<br>
shape is [B, res, res].<br>
“”"<br>
However, it should be able to process a resolution of up to 65,536. I have tried the following method, but it is really slow. Other things I have tried and I run out of RAM. Is there a better way to deal with large tensors like this, or perhaps is my actual algorithm too slow?<br>
Here is my code, I am doing chunk by chunk and writing it to disk:</p>
<pre><code class="lang-auto">import torch
import numpy as np
import matplotlib.pyplot as plt
import os

def create_points(batch_size, num_points):
  coords = np.random.rand(batch_size, num_points, 2)
  return coords

def min_dist(points, res):

  data_coords = torch.tensor(points, dtype=torch.float32) * (res - 1)

  B, P, _ = data_coords.shape

  chunk_size = 4096

  # Calculate number of chunks
  num_chunks = (res + chunk_size - 1) // chunk_size

  # min_dist_tensor_full = torch.zeros(B, res, res)
  min_dist_tensor_full = np.memmap('large_tensor.dat', dtype='float32', mode='w+', shape=(B, res, res))

  # Initialize or load the final tensor

  for i in range(num_chunks):
    for j in range(num_chunks):
      x_start = i * chunk_size
      x_end = min((i + 1) * chunk_size, res)
      y_start = j * chunk_size
      y_end = min((j + 1) * chunk_size, res)

      grid_x, grid_y = torch.meshgrid(
          torch.arange(x_start, x_end, dtype=torch.float32),
          torch.arange(y_start, y_end, dtype=torch.float32),
          indexing='xy'
      )

      grid_x = grid_x.flatten()
      grid_y = grid_y.flatten()
      grid_coords = torch.stack([grid_x, grid_y], dim=-1)

      chunk_size = x_end - x_start

      grid_coords_batch = grid_coords.unsqueeze(0).expand(B, -1, -1)

      dists = torch.cdist(grid_coords_batch, data_coords)

      min_dists, _ = dists.min(dim=2)

      min_dist_chunk = min_dists.view(B, chunk_size, chunk_size)

      min_dist_tensor_full[:, y_start:y_end, x_start:x_end] = min_dist_chunk.numpy()

  return min_dist_tensor_full
</code></pre>
<p>Here is how I visualise my result:</p>
<pre><code class="lang-auto">points = create_points(3, 4)
dist_data = min_dist(points, 65536)[:]

batch_size = dist_data.shape[0]
plt.figure(figsize=(12, 4 * batch_size))

for i in range(batch_size):
    plt.subplot(1, batch_size, i + 1)
    plt.imshow(dist_data[i], cmap='viridis')
    plt.title(f'Batch {i + 1}')

plt.tight_layout()
plt.show()
</code></pre>
Anomalous time gaps observed when using CUDA kernels in PyTorch

hexisyztem: <p>The issue is that there is a certain time gap between the completion of one kernel’s execution and the launch of the next kernel. From the profiling perspective, I can’t determine what is happening during this time.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/c/cce46b176c4011835de76377c0a75ee046ca9c42.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/cce46b176c4011835de76377c0a75ee046ca9c42" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/c/cce46b176c4011835de76377c0a75ee046ca9c42_2_690x312.jpeg" alt="image" data-base62-sha1="teyMQZjgIk7qxbPEX1Wym8LiqVI" width="690" height="312" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/c/cce46b176c4011835de76377c0a75ee046ca9c42_2_690x312.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/c/c/cce46b176c4011835de76377c0a75ee046ca9c42_2_1035x468.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/c/c/cce46b176c4011835de76377c0a75ee046ca9c42_2_1380x624.jpeg 2x" data-dominant-color="D0E5E6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1920×870 146 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>How can I analyze what is causing the time consumption during this period? Note: This part of the computation shown in the diagram is a simple encoder layer.</p>

hexisyztem: <p>I found that part of the gaps is caused by the host time consumed by torch.cuda.nvtx, but the remaining part still cannot find the specific cause of the time consumption. Is there any method to profile the host time?</p>
Ensemble of models and gradients

Bodo: <p>Suppose I have a very simple ensemble of models like this:</p>
<pre><code class="lang-auto">    class MyEnsemble(nn.Module):
        def __init__(self, modelA, modelB):
            super(MyEnsemble, self).__init__()
            self.modelA = modelA
            self.modelB = modelB
</code></pre>
<p>ModelA does some simple preprocessing and feeds its output into ModelB. Usually the gradient is then calculated during the backwardpass starting from modelB and going to modelA.</p>
<p>I wonder, what will happen if I do something like this ?</p>
<pre><code class="lang-auto"> def forward(self, x):
            x = self.modelA(x)
            x = torch.where(x &gt; 0.05, 255, 0)
            x = self.modelB(x)

            return x
</code></pre>
<p>The toch.where() function here is supposed to produce some kind of threshold for the output of modelA. This code does not throw an error or warning. However since this function is not differentiable I was wondering what would happen during the backwardpass? Will the gradients in modelA be unchanged? Is the statement ignored and the loss just computes the gradient for the output of modelA?</p>
<p>I think the general question is: Are model gradients computed individually or in conjuction?</p>
<p>Hoping for some insights on this one.</p>
I am trying to train a model by considering navier stokes equation as loss function. when i run the code i am getting an error " one of the differentiated tensors does not require grad"

Bharghav: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/2/12f5363e0df6ea224a75bc19df9130703d461a70.png" data-download-href="https://discuss.pytorch.org/uploads/default/12f5363e0df6ea224a75bc19df9130703d461a70" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/2/12f5363e0df6ea224a75bc19df9130703d461a70.png" alt="image" data-base62-sha1="2HHXqv9wsXrCBBWPqOjoP0CJWMM" width="507" height="500" data-dominant-color="242E37"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">917×903 47.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Custom Image Folder

efthamar: <p>Hello!</p>
<p>I want to create a custom <code>ImageFolder</code> were instead of the default <code>(sample, target)</code>, it will also return an annotated version of each data sample. Therefore <code>(sample, annotated_sample, target)</code>.</p>
<p>Below</p>
<pre><code class="lang-auto">class AugmentatedImageFolder(ImageFolder):
    """Augmentated Images dataset."""

    def __getitem__(self, index: int):
        """
        Args:
            index (int): Index

        Returns:
            tuple: (sample, augmented_sample, target) where target is class_index of the target class.
        """
        path, target = self.samples[index]
        sample = self.loader(path)

        if self.transform is not None:
            sample_augmented = self.transform(sample)
        if self.target_transform is not None:
            target = self.target_transform(target)

        return sample, sample_augmented, target
</code></pre>
<p>Is the above approach correct?</p>
<p>I run my experiments in an HPC, and I get the below memory error when I do this customization:<br>
<code>slurmstepd: error: Detected 2 oom_kill events in StepId=1819449.batch. Some of the step tasks have been OOM Killed.</code></p>
<p>Is there something in the custom class that could be affecting the memory?</p>
<p>Thank you! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

ptrblck: <p>It’s unclear what is causing the OOM issues but depending on the sample size and batch size the data loading could be responsible for it.<br>
Your code looks generally alright, but you should receive an error if the <code>if</code> conditions are not met since the returned variables would be undefined.</p>

efthamar: <p>Hello <a class="mention" href="/u/ptrblck">@ptrblck</a> , thank you for your response and your points.</p>
<p>I had some extra lines in the class that i skipped in my post, where I was composing multiple transforms using <code>transorms.Compose</code>  and the applied them to the data. It seems that this was causing the OOM issue, so I used another approach instead, and it worked fine.</p>

ptrblck: <p>Good to hear it’s working now! Just out of interest, which transformations were you using?</p>
Export libtorch trained model to load in python

tuero: <p>I was wondering if anyone has any ideas for how to export to disk a trained model in libtorch, then load in python pytorch? The environment I work in benefits from training in the C++ runtime, but it is often nice to run visualizations on the python end for insights.</p>
<p>I’ve seen this here but there doesn’t seem to me much options: <a href="https://discuss.pytorch.org/t/exporting-libtorch-model-and-loading-in-python/108240" class="inline-onebox">Exporting libtorch model and loading in Python</a></p>
<p>Ideally, the loading side on the python pytorch can be done with vanilla pytorch (no C++ extensions). Maybe a custom script which reads the weights from the exported file and sets the weights accordingly on the python end, assuming the network structure is the same and the weights are written in a deterministic way? Are there any documentation for how the weights are stored in the exported file? Any other ideas?</p>

tuero: <p>Here is what I currently have:</p>
<p>Assuming the model parameters are registered in the correct order (i.e. iterating over in C++ <code>model-&gt;named_parameters()</code> yields the same order as python <code>model.state_dict()</code>, in C++ I can save the weights as <code>torch::save(model-&gt;named_parameters().values(), "test.pt");</code>.</p>
<p>On the python side, I can do the following:</p>
<pre data-code-wrap="python"><code class="lang-python">t = torch.jit.load("test.pt")
tensor_list = list(t.parameters())

param_names = [name for name in model.state_dict()]
model_dict = model.state_dict()

for tensor, param_name in zip(tensor_list, param_names):
    model_dict[param_name] = tensor
    
model.load_state_dict(model_dict)
</code></pre>
<p>From my limited testing, this seems to work correctly.</p>
Interpret NCHW channel_last tensor as NHWC normal tensor?

111550: <p>I need to use conv layers and transformer FFNs (2 linear layers with an activation function in between) in an interleave manner, just like those in a ConvNext block.<br>
Assume N=num_images, C=num_channel, H=image_height, W=image_weight, in conv layers, I need the input &amp; output to be of shape [N, C, H, W] to use nn.Conv2d APIs, while in the FFN layers, I need the input &amp; output tensor to be of shape [N, H, W, C] to perform linear transformation in the channel dimension. This leads to redundant permutation of tensors (especially when stacking multiple such blocks), which I assume is slow.</p>
<p>The code would look like:</p>
<pre><code class="lang-auto">x = torch.randn(N, C, H, W)
x = conv2d(x, conv_kernel)  # [N, C, H, W]
x = x.permute(0, 2, 3, 1)  # [N, H, W, C]
x = ffn(x) # [N, H, W, C]
x = x.permute(0, 3, 1, 2)  # [N, C, H, W]
</code></pre>
<p>I’ll repeat such block several times, so I need to permute tensor x several times. Permutation doesn’t move data, it just calculate a proper stride for each dimension, but I’m sure if the FFN layer is efficient enough when axis C is discountinuous in memory?</p>
<p>If not, another option is to use the channel_last memory format to make dimension C continuous before &amp; after conv2d. But in this case, I don’t know how to convert a channel_last [B, C, H, W] tensor to normal contiguous [B, H, W, C] tensor to let the FFN works. I tried to view the channel_last [B, C, H, W] tensor directly, but it doesn’t work.</p>

111550: <p>I seem to get the answer to the second question (thus the original question): For a channel_last [B, C, H, W] tensor x, simply do <code>x.permute(0, 2, 3, 1)</code> will get a normal contiguous tensor.</p>
<p>But for the first question, I still do not know if a discontinuous dimension C is efficient in FFN layers.</p>