akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Cannot export call stacks for flamegraphs with Pytorch profiler?

Vedant_Roy: <p>Cross-posted as an issue: <a href="https://github.com/pytorch/pytorch/issues/89406" class="inline-onebox" rel="noopener nofollow ugc">stacks file from profiler is empty · Issue #89406 · pytorch/pytorch · GitHub</a>, but I’m 99% sure it’s not a bug and just me misusing the API somehow.</p>
<p>I have a small reproducible example below that should (theoretically) write out a .stacks file that I can feed into a flamegraph generator.</p>
<p>In reality, the file is empty.<br>
Does anyone know why?<br>
I can’t see anything obviously wrong with my usage of the API either.</p>
<pre><code class="lang-auto">import torch

class Mlp(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = torch.nn.Linear(100, 100)
        self.act = torch.nn.ReLU()
        self.fc2 = torch.nn.Linear(100, 100)

    def forward(self, x):
        return self.fc2(self.act(self.fc1(x)))

class Net(torch.nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.mlp1 = Mlp()
        self.mlp2 = Mlp()

    def forward(self, x):
        return self.mlp2(self.mlp1(x))

from torch.utils.data import DataLoader
import torchdata.datapipes.iter as dp
from torch.profiler import profile, ProfilerActivity

def make_mock_dataloader():
    pipe = dp.IterableWrapper([torch.rand(100) for _ in range(1000)])
    return DataLoader(pipe, batch_size=32, num_workers=2, drop_last=True)

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
net = Net().to(device=device)

target_traces = 2
traces_saved = 0

def trace_handler(prof: "torch.profiler.profile"):
    global traces_saved
    from os.path import join

    print("SAVING TRACE")

    tb_dir = join("./output2", "traces", str(traces_saved))
    handler = torch.profiler.tensorboard_trace_handler(
        tb_dir, worker_name=f"rank0"
    )
    handler(prof)

    prof.export_stacks(path=join(tb_dir, f"rank0.cuda.stacks"), metric="self_cuda_time_total")
    prof.export_stacks(path=join(tb_dir, f"rank0.cpu.stacks"), metric="self_cpu_time_total")

    # print(prof.events())

    traces_saved += 1
    if traces_saved == target_traces:
        prof.stop()

prof = profile(
    activities=[ProfilerActivity.CPU, ProfilerActivity.CUDA],
    # profile_memory=True,
    with_stack=True,
    # with_modules=True,
    # record_shapes=True,
    schedule=torch.profiler.schedule(
        skip_first=5, wait=1, warmup=5, active=5, repeat=target_traces
    ),
    on_trace_ready=trace_handler,
)
prof.start()

for idx, batch in enumerate(make_mock_dataloader()):
    print(f"idx: {idx}")
    batch = batch.to(device=device)
    out = net(batch)
    out.sum().backward()
    net.zero_grad()
    prof.step()
</code></pre>

TheExGenesis: <p>facing the exact same problem</p>

Vedant_Roy: <p>Did you ever manage to solve it?</p>
Normalizing a video dataset in pytorch

alexmehta: <p>I see plenty of guides on how to calculate std and mean for an image dataset, but how would you do so for videos in the standard format (B x L x C x W x H)?</p>

rinkujadhav2013: <p>Do you mean to ask what works best practically or do you wanna know how to calculate mean, std for an n-dim tensor in Pytorch?<br>
I guess it’s the former because the latter is too easy. I’ve never worked with a video dataset so I don’t know what works best practically. But usually with any kind of dataset, you should start with the easiest normalization and then make it more nuanced with experimentation.<br>
So I’d have tried normalisations in the following order:</p>
<ol>
<li>Simply divide each pixel by 255 to get values between 0 and 1. This is what you do with images as well. It’s a standard practice so I guess you’re doing it already.</li>
<li>Find the mean, std of the whole dataset. So, basically mean, std across all pixels of all frames of all videos i.e. across all <code>B,L,C,W,H</code> dims. Stadardise all pixels using this mean and variance.</li>
<li>Calculate mean, std for each video separately. So mean, std across <code>L,C,H,W</code> only.</li>
<li>Mean across each frame separately i.e. across <code>C,H,W</code> dimensions.</li>
</ol>
<p>I’d expect approach <code>1</code> or <code>3</code> to work best but that’s just a guess.</p>
Reproducibility problem - not working as expected after 3 years

vitruvi: <p>I have reproduced this experiment 2 years ago, that is partly hosted at the github link below.</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://github.com/ustc-bmec/Whole-Brain-Conv">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/ustc-bmec/Whole-Brain-Conv" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/345;"><img src="https://opengraph.githubassets.com/e9412bd7abe541ccb6181731552c442c2e1dbb5a53d672c35daeb2cb30dd48ab/ustc-bmec/Whole-Brain-Conv" class="thumbnail" width="690" height="345"></div>

<h3><a href="https://github.com/ustc-bmec/Whole-Brain-Conv" target="_blank" rel="noopener nofollow ugc">GitHub - ustc-bmec/Whole-Brain-Conv: A 3D Convolutional Neural Network for...</a></h3>

  <p>A 3D Convolutional Neural Network for whole-brain fMRI decoding - GitHub - ustc-bmec/Whole-Brain-Conv: A 3D Convolutional Neural Network for whole-brain fMRI decoding</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>At the moment, even the demo notebook does not work as expected.</p>
<p>To reproduce the issue:</p>
<ul>
<li>clone the repository</li>
<li>run sample.ipynb</li>
<li>observe the “pred_prob” output.</li>
</ul>
<p>The output of pred_prob is drastically different in the hosted notebook’s saved state than the one I run on my machine.</p>
<p>Expected result: tensor([[6.1537e-09, 1.0000e+00, 5.6260e-11, 2.5552e-15, 6.2870e-09, 4.2733e-11, .1202e-08]], device=‘cuda:0’)<br>
What I get: tensor([[0.0037, 0.2303, 0.0036, 0.0065, 0.0014, 0.0768, 0.6778]], device=‘cuda:0’)</p>
<p>Requirements are nibabel, pandas, numpy and pytorch.</p>
<p>What I tried:<br>
I created a new environment, tried combinations of previous cuda-pytorch combinations for 0.4.1 and 1.00 pytorch. Downgraded nibabel and numpy with pypi-timemachine at date 2018-12-12. But these didn’t work.</p>
<p>I can additionally say that, problem is not limited to loading the checkpoint. My code that used to work that runs from scratch does not work too.</p>
<p>Any advice is appreciated.</p>
Gradient of module output with regards to module input

alexandrumeterez: <p>Hi everyone,</p>
<p>First question here, I hope it’s the correct place.</p>
<p>I have a chain of nn.Sequential and I’m trying to extract 2 gradients from my model: gradient of the module output wrt the module input and the gradient of the module output wrt x (initial input).</p>
<p>In other words, given:</p>
<pre><code class="lang-auto">y1 = f1(x)
y2 = f2(y1)
# and so on
</code></pre>
<p>where f_i is an nn.Sequential (or any nn.Module), I’d like to extract dy2/dy1 (module output wrt module input) and dy2/dx (module output wrt x).</p>
<p>Thanks!<br>
Alex</p>

srishti-git1110: <p>Hi Alex,<br>
You might be looking for torch.autograd.grad :<br>
<a href="https://pytorch.org/docs/stable/generated/torch.autograd.grad.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.autograd.grad.html</a></p>
<p>Feel free to post in the thread if you face any errors.</p>

alexandrumeterez: <p>Hi Srishi,</p>
<p>Thanks for your help. After going over that doc page, I came up with this tiny piece of code to test it out:</p>
<pre><code class="lang-auto">model = nn.Sequential(
    nn.Sequential(
        nn.Linear(12, 14),
        nn.Linear(14, 16),
    ),
    nn.Sequential(
        nn.Linear(16, 16),
        nn.Linear(16, 20),
    ),
    nn.Sequential(
        nn.Linear(20, 20),
        nn.Linear(20, 20),
    ),
).to(device)

grads = []
def hook2(module, inputs, outputs):
#     import pdb
#     pdb.set_trace()
    grads.append(torch.autograd.grad(outputs, inputs[0]))
    
grads.clear()
for module in model.modules():
    if isinstance(module, nn.Sequential):
        module.register_forward_hook(hook2)
</code></pre>
<p>However, I’m getting an error:<br>
RuntimeError: grad can be implicitly created only for scalar outputs</p>
<p>Not sure how to proceed from here.</p>

srishti-git1110: <p>Hi <a class="mention" href="/u/alexandrumeterez">@alexandrumeterez</a> ,<br>
Yes, that’s expected behaviour when you try to use autograd to calculate gradient of a tensor that isn’t a single number (scalar).</p>
<p>In case you want to be able to calculate gradient of a multi-dim tensor, use this:</p>
<pre><code class="lang-python">import torch
inp = torch.tensor([4.0, 3.0, 2.0], requires_grad=True)
x = (inp*2)**3  # x is not a scalar
x.backward(torch.ones_like(x))
print(inp.grad) # tensor([384., 216.,  96.])
</code></pre>
<p>In fact, when <code>x</code> is scalar, <code>x.backward()</code> is a shortcut for <code>x.backward(torch.Tensor([1]))</code>.<br>
Hope this helps,<br>
S</p>

alexandrumeterez: <p>This is what I was looking for, thank you!</p>
Get key of nn.Sequential made with OrderedDict

cxv0519: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/7/b7d2fd22737635ebea6d147e5e62dd9f6354ec5c.png" data-download-href="https://discuss.pytorch.org/uploads/default/b7d2fd22737635ebea6d147e5e62dd9f6354ec5c" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/7/b7d2fd22737635ebea6d147e5e62dd9f6354ec5c.png" alt="image" data-base62-sha1="qebpWOwfrMapNzf72Sqz4Ybd0te" width="553" height="500" data-dominant-color="F3F3F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">622×562 16.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>In the case like image above, can I extract the name(I mean original key of OrderedDict) of the Sequential module?</p>
<p>It seems easy but not easy…</p>

cxv0519: <p>Oh, I founded nn.ModuleDict<br>
<a href="https://pytorch.org/docs/stable/generated/torch.nn.ModuleDict.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.nn.ModuleDict.html</a><br>
I’d better to use this than Sequential with OrderedDict…</p>
<p>Please share ideas<br>
Thank you</p>

srishti-git1110: <p>Hi, yes.<br>
<code>ModuleDict</code> can be used with Sequential and then queried using layer names, like so -</p>
<pre><code class="lang-python">model_dict = nn.ModuleDict({
                'fc1': nn.Linear(786, 120),
                'first_fc1_relu': nn.ReLU(),
                'fc2': nn.Linear(120, 10)
        })

model = nn.Sequential(model_dict)
print(model)
print(model[0]['fc1'] )
</code></pre>
<p>gives</p>
<pre><code class="lang-plaintext">Sequential(
  (0): ModuleDict(
    (fc1): Linear(in_features=786, out_features=120, bias=True)
    (first_fc1_relu): ReLU()
    (fc2): Linear(in_features=120, out_features=10, bias=True)
  )
)

Linear(in_features=786, out_features=120, bias=True)
</code></pre>
<p>With OrderedDict, you can access the specific layers by their names like this -</p>
<pre><code class="lang-python">model_dict = OrderedDict()
model_dict['fc1'] = nn.Linear(786, 120)
print(model_dict)
model = nn.Sequential(model_dict)
print(model)
print(model.fc1)
</code></pre>
<p>gives</p>
<pre><code class="lang-plaintext">Sequential(
  (fc1): Linear(in_features=786, out_features=120, bias=True)
)

Linear(in_features=786, out_features=120, bias=True)
</code></pre>
<p>In both cases, the layers’ parameters are registered properly.</p>
Backpropagation on random networks

Settur_Anantha_Sai_J: <p>When we define a very random graph like the input in going as input to the 5th node of layer 4 and we have many nodes and edges like that, will loss.backward() still work?</p>

srishti-git1110: <p>It should.<br>
Please post any code that you think might produce error.</p>

Settur_Anantha_Sai_J: <p>I cant share the code. But what happens in the code is that we convert an adjacency matrix into the network. The problem is if I use the code like that and concatenate the outputs of previous nodes and send it into the next node does it know where that input is coming from? I believe because of this the weights don’t change. As in there is no optimization occuring.</p>

srishti-git1110: <p><code>torch.cat</code> should not produce any error during the backward pass so you should be fine using it for concatenation. About converting “an adjacency matrix into the network”, I am not quite clear what you mean. Could you please post some code to elaborate?</p>

Settur_Anantha_Sai_J: <pre><code class="lang-auto">def forward(self,x):
    outputs=[torch.zeros_like(x[:,0].unsqueeze(1))]*self.n
    
    for i in range(x.shape[1]):
      outputs[i]=x[:,i].unsqueeze(1).to(device)
    
    node=0
    for i in range(self.input_size,len(self.inputs_count)):
      if(self.inputs_count[i]&gt;0):
        inpts=[]
        for j in range(self.n):
          if self.adjMat[j][i]!=0:
            inpts.append(outputs[j])
        outputs[i]=self.activation(self.nodes[node](torch.cat(inpts,1)))
        node+=1
    return self.ouput_activation(torch.cat(outputs[-self.outputs_count:],1)) if self.activation != None else torch.cat(outputs[-self.outputs_count:],1)
</code></pre>
<p>As u may see in this forward pass I split the input, then add it to a list of outputs. the outputs  contains all the outputs from all nodes. And input to a node is only from a select few nodes and hence i concatenate those nodes and push them into as input to the next node.</p>
<p>While training this network the change in weights is very slow. The training is very slow even for a simple XOR example. So my question is what could be the reason for this? Is the fact that the node does not know properly where the inputs are coming from causing a problem? If so, how should i fix it?</p>

Settur_Anantha_Sai_J: <p>My bad was doing an error in the network. Thanks for your help.</p>
Set current stream from external stream

Kuppharish: <p>My input image exists in a cuda stream which was created by <code>cudaStreamCreate</code>. Now I’m trying to set the model and other tensors to same stream in the following way,</p>
<blockquote>
<p>cudaStream_t cuda_stream;<br>
cudaStreamCreateWithPriority(&amp;cuda_stream, cudaStreamNonBlocking, -1);</p>
<p>at::cuda::CUDAStream myStream = at::cuda::getStreamFromExternal(cuda_stream, 0);<br>
at::cuda::setCurrentCUDAStream(myStream);</p>
</blockquote>
<p>This is throwing the following error</p>
<pre><code class="lang-auto">Unrecognized stream stream 3394290592 on device cuda:0 (I didn't recognize the stream type, EXT)
</code></pre>
<p>Is there any way to fix this?</p>
Does tensor.to(device) copy the data if the data is already present in GPU?

Kuppharish: <p>I’m working with libtorch and I’ve input data which is already present in GPU(cuda). I’m making a tensor from my data this way,</p>
<blockquote>
<p><code>torch::Tensor input_tensor = torch::from_blob(input_image.data,{1,engine_input_rows,engine_input_cols,engine_input_channels});</code></p>
</blockquote>
<p>If I don’t use input.to(‘cuda’) the system thinks that input tensor is in CPU. So if I do input.to(‘cuda:0’) does libtroch know that the data is already present in GPU and not cause me any transfer computation?</p>
DDP how to find unused parameters preventing `static_graph=True` from working?

ArchieGertsman: <p>I am trying to set <code>static_graph=True</code> in DDP, because I believe it should work in my case. However, during backward I get the error<br>
<code>RuntimeError: Your training graph has changed in this iteration, e.g., one parameter is unused in first iteration, but then got used in the second iteration. this is not compatible with static_graph set to True.</code><br>
I would like to see which parameter(s) are causing this.</p>
<p>So far, I’ve tried enabling debugging via</p>
<pre><code class="lang-auto">os.environ["TORCH_CPP_LOG_LEVEL"] = "INFO"
os.environ["TORCH_DISTRIBUTED_DEBUG"] = "DETAIL"
</code></pre>
<p>But the only relevant output I get is<br>
<code>[I logger.cpp:377] [Rank 0 / 2] [before iteration 1] Training ActorNetwork unused_parameter_size=0</code><br>
every time the model is called. <code>unused_parameter_size</code> always equals zero.</p>
<p>I’ve also tried logging the forward trace each time the model is called using a hook:</p>
<pre><code class="lang-auto">torch.nn.modules.module.register_module_forward_hook(
    lambda module, input, output: print(module)
)
</code></pre>
<p>but the traces are identical between the first two model calls (verified using <code>diff</code> command).</p>
<p>How can I track down the cause of the error during <code>backward</code>?</p>

irisz: <aside class="quote no-group" data-username="ArchieGertsman" data-post="1" data-topic="170142">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/archiegertsman/48/51153_2.png" class="avatar"> ArchieGertsman:</div>
<blockquote>
<p><code>RuntimeError: Your training graph has changed in this iteration, e.g., one parameter is unused in first iteration, but then got used in the second iteration. this is not compatible with static_graph set to True.</code></p>
</blockquote>
</aside>
<p>Have you set <code>find_unused_parameters=True</code> when initializing DDP? If not, could you try this?</p>
<p>Currently, <code>find_unused_parameters=True</code> must be passed into <a href="https://pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html#torch.nn.parallel.DistributedDataParallel" rel="noopener nofollow ugc"><code>torch.nn.parallel.DistributedDataParallel()</code> </a> initialization if there are parameters that may be unused in the forward pass.</p>
What is the usage of DDP output_device?

JuyiLin: <p>I find a parameter in DDP:<br>
<code>model = DistributedDataParallel(             model, device_ids=[train_device], output_device=train_device) </code><br>
What is the meaning of output_device? I read the source codes and doc but cannot figure it out. Does the different device has the same output_device?</p>

irisz: <p>Output_device is usually the rank, which tells DDP where to output.</p>
<p>For details, please check out the following documentations.<br>
<a href="https://pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html</a><br>
<a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/intermediate/ddp_tutorial.html</a></p>
How to get region proposals from Faster-RCNN?

dominic_simon: <p>I’m trying to find the region proposals created by the Region Proposal Network during inference time in the Faster-RCNN model. To be clear, I’m looking for either the bounding box coordinates of these region proposals or the full regions proposals themselves. I’ve tried looking into the many different networks that comprise the whole model (via forward hooks and looking at the model architecture), but I can’t seem to find what I’m looking for. If anyone can tell me where I can pull the coordinates or the regions themselves from, it would be much appreciated.</p>
Segmentation Fault while element access with data_ptr on GPU

HallerPatrick: <p>Hey,</p>
<p>I am running into problems I am not able to comprehend. Maybe you can help me out.</p>
<p>I am working on a C++ extension and reimplementing some functions for performance.<br>
In one of them I am trying to iterate through a 1-dimensional tensor and access each element in it.</p>
<pre><code class="lang-auto">torch::Tensor unpack(const torch::Tensor &amp;self) {
    TORCH_CHECK(self.dim() == 1, "Only one dimensional tensor allowed");
    TORCH_CHECK(self.is_contiguous(), "Tensor has to be conitguous");
    TORCH_CHECK(self.dtype() == torch::kInt64, "Tensor has to be long type");

    // Get pointer to data
    auto *self_ptr = self.data_ptr&lt;int64_t&gt;();

   // Access data
   // Crashed with GPU
    std::cout &lt;&lt; *self_ptr &lt;&lt; "\n"; // index 0 
    std::cout &lt;&lt; *self_ptr++ &lt;&lt; "\n"; // index 1
   
   ...

}
</code></pre>
<p>Sample input:</p>
<pre><code class="lang-auto">auto opts = torch::TensorOptions().dtype(torch::kInt64);
auto tensor = torch::ones({4}, opts);

unpack(tensor);
</code></pre>
<p>This all works fine when the input tensor is on the CPU, BUT if I try to pass a tensor that is on the GPU I am running into a segmentation fault. Note that the tensor is in the end passed from the python program, if that is relevant.</p>
<p>UPDATE:</p>
<p>I tested with moving the tensor back on the cpu and this works. Which is a workaround…<br>
Is there a way of accessing the data of the GPU tensor without the overhead off moving it?</p>
<p>In hope someone can help me out here.</p>
<p>Greetings,<br>
Patrick</p>

ptrblck: <p>You won’t be able to access device memory on the host via a direct pointer dereference and would either need to move the data back to the host or access the memory in a CUDA kernel.</p>
Training yolov8 throws an error - AttributeError: 'NoneType' object has no attribute '_free_weak_ref'

PedroC: <p>Hi</p>
<p>I’m facing an issue during training with yolov8. After each epoch the following errror appears several times.</p>
<p>lib\site-packages\torch\storage.py", line 520, in _free_weak_ref<br>
AttributeError: ‘NoneType’ object has no attribute ‘_free_weak_ref’<br>
Exception ignored in: &lt;function StorageWeakRef.<strong>del</strong> at 0x000002448E3708B0&gt;</p>
<p>Although I’m able to train and save yolov8 models, time to time, the training crashes, which is very frustrating taking into account a training takes 5 hours and after 3 hours of training the program crashes.</p>
<p>I’m using anaconda env and I installed the following packages</p>
<p>conda install pytorch==1.11.0 torchvision==0.12.0 torchaudio==0.11.0 cudatoolkit=11.3 -c pytorch</p>
<p>Everything worked fine with yolov5, but with yolov8 this issue happens.</p>
<p>Thanks</p>

ptrblck: <p>The issue seems to be related to <a href="https://github.com/pytorch/pytorch/issues/74016">this one</a> which is apparently fixed in <code>1.12.0+</code>, so you might want to update your PyTorch release.</p>

PedroC: <p>Interesting, I went to that page before. But I managed to solve the issue by not updating the pytorch. I mean I worked with yolov5 in pytorch 1.11 and no issue happened. In fact the most recent version of pytorch gave me problems that is why months ago I downgraded to 1.11<br>
But now I solved the issue by upgrading the version. Thank you very much!</p>
Understanding the QuantizeBase Implementation

Rami_Ismael: <pre><code class="lang-auto">class APoTFakeQuantize(FakeQuantizeBase):
    alpha: Tensor
    gamma: Tensor
    quantization_levels: Tensor
    level_indices: Tensor

    def __init__(self, observer=APoTObserver, **observer_kwargs):
        super().__init__()
        self.activation_post_process = observer(**observer_kwargs)
        self.dtype = self.activation_post_process.dtype

    def calculate_qparams(self, signed=False):  # type: ignore[override]
        return self.activation_post_process.calculate_qparams(signed=signed)

    def forward(self, X: torch.Tensor):  # type: ignore[override]
        if self.observer_enabled[0] == 1:
            self.activation_post_process.forward(X)
            result = self.activation_post_process.calculate_qparams(signed=False)
            self.alpha = result[0]
            self.gamma = result[1]
            self.quantization_levels = result[2]
            self.level_indices = result[3]

        if self.fake_quant_enabled[0] == 1:
            assert (self.alpha is not None
                    and self.gamma is not None
                    and self.quantization_levels is not None
                    and self.level_indices is not None), "Must set qparams for fake quant"

            X = fake_quantize_function.apply(X, self.alpha, self.gamma, self.quantization_levels, self.level_indices)

        return X
</code></pre>
<p>In PyTorch, QAT Implementation, will the observer method and fake quant always be on when training? Am I correct?</p>

Vasiliy_Kuznetsov: <p>You can control the <code>observer_enabled</code> and <code>fake_quant_enabled</code> flags as needed - the timing of when you turn them on or off can be considered hyperparameters of your training job.</p>
<p>There is an example here which works well for vision: <a href="https://github.com/pytorch/vision/blob/main/references/classification/train_quantization.py#L173" class="inline-onebox" rel="noopener nofollow ugc">vision/train_quantization.py at main · pytorch/vision · GitHub</a>, the --num-observer-update-epochs argument</p>
How could I potentially optimize specific Hyperparameters in CNN?

setsailforfail: <p>I am currently learning about hyperparameter optimization in Neural Networks and also learned about techniques like Hyperparamter Gradient Descent.</p>
<p>I am actually curious, if this method (Hyperparamter Gradient Descent) could be used to optimize things like padding and stride in a backward pass in PyTorch? If not, how do you alternatively tune these parameters if you are training a model using PyTroch?</p>
A trained Deep learning model giving same output for all images in the inference/test phase

aml.rhmn: <p>Hi!<br>
So I trained a U-Net model in adversarial setting with a Discriminator. The training and validation curves look fine. Just for context, I used L2 loss as the Generator loss and the BCE loss as Discriminator and the Adversarial loss.<br>
In the inference stage, I load this Pytorch model of the Generator and pass the (unseen) samples as input. Somehow, I am getting the same tensors as output against all samples. Am I missing something?<br>
[Screenshot of output tensor attached]<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/8/2896932966396cfc2e8e21414bbdc19b7555f830.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/2896932966396cfc2e8e21414bbdc19b7555f830" title="Screenshot 2023-01-13 at 4.41.25 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/8/2896932966396cfc2e8e21414bbdc19b7555f830_2_690x379.jpeg" alt="Screenshot 2023-01-13 at 4.41.25 PM" data-base62-sha1="5N3HDYkDTs8mWpXBKpntE78c1CU" width="690" height="379" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/8/2896932966396cfc2e8e21414bbdc19b7555f830_2_690x379.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/2/8/2896932966396cfc2e8e21414bbdc19b7555f830_2_1035x568.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/2/8/2896932966396cfc2e8e21414bbdc19b7555f830_2_1380x758.jpeg 2x" data-dominant-color="3E1831"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2023-01-13 at 4.41.25 PM</span><span class="informations">1920×1056 165 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Select layers in a CNN

Sumesh_Sankar: <p>how to select layers of a CNN by its name in pytorch</p>

srishti-git1110: <p>Hi, you could print your model to view all the layers present in it. And them access a particular layer using . like so -</p>
<pre><code class="lang-python">import torch
import torch.nn as nn

class LinearModel(nn.Module):
  def __init__(self):
      super(LinearModel, self).__init__()
      self.fc1 = nn.Linear(10, 20)
      self.fc2 = nn.Linear(20, 20)
      self.fc3 = nn.Linear(20, 20)


  def forward(self, x):
      pass
      

net = LinearModel()
print(net) # printing model to see the layers
print(net.fc1) # accessing particular layers
</code></pre>
<p>gives -</p>
<pre><code class="lang-auto">LinearModel(
  (fc1): Linear(in_features=10, out_features=20, bias=True)
  (fc2): Linear(in_features=20, out_features=20, bias=True)
  (fc3): Linear(in_features=20, out_features=20, bias=True)
)
Linear(in_features=10, out_features=20, bias=True)
</code></pre>
Indexing with in-place operation

huahuanZ: <p>I read the libtorch indexing API at <a href="https://pytorch.org/cppdocs/notes/tensor_indexing.html" class="inline-onebox" rel="noopener nofollow ugc">Tensor Indexing API — PyTorch master documentation</a><br>
and try to figure out how to do such operation in c++</p>
<pre><code class="lang-auto"># python
a[:3] += b
</code></pre>
<p>If not in-place,</p>
<pre><code class="lang-auto">// c++
auto c = a.index({torch::indexing::None, 3});
c += b;
a.index_put_({torch::indexing::None, 3}, c);
</code></pre>
Data loader multiprocessing architecture

Vinayaka_Hegde: <p>I wanted to deep-dive and understand the internal architecture of the data loader. So, I started off with the source code and tried to understand dataloader.py. I understood the type of datasets and the action of sampler based on these datasets. I also understood about the multprocessDataLoading and how the worker processes are created and how the indices are populated into the index queue and how the worker processes send data to the global data-queue. I wanted a more in-depth understanding on the multiprocessing data fetching part. I have questions like :</p>
<ol>
<li>How the worker processes consume from the index queue? Are the indices consumed in sequential order? random order?</li>
<li>How the worker processes write into the data queue? Are they written sequentially or parallely?</li>
<li>How the data loader fetches and outputs from the data queue? How the mini batches constructed ?</li>
</ol>
<p>Mainly, I wanted to get a detailed picture on the multiprocessingdata loading part and the workers interaction with data queue, index queue and scheduling, order in which data moves to/from worker to the queues. And how the data loader constructs the mini batches and outputs it ? Since all workers execute parallely.</p>
<p>I couldnt find any videos/blogs explaining the dataloader multiprocessing architecture (in-depth). Please point me to some resources or feel free to explain it here</p>

ptrblck: <p>Please don’t tag specific people as it might discourage others to answer and the tagged ones might not know all answers directly. In particular I cannot give detailed answers and would refer you to these <a href="https://github.com/pytorch/pytorch/blob/0bd3fa3d22db81cf960cc9618abf2c32602a94c2/torch/utils/data/dataloader.py#L685">docs</a> in case you haven’t seen them already (I guess you did).</p>
Reuse output of model

lightelf_forest: <pre><code class="lang-auto">def train(input):
    noises=noise_extractor(input)-input
    total_loss=0.0
    for i in range(batch_size):
        for noisys,_ in train_loader:
            noisys=noisys.to(device=device)
            optimizer.zero_grad()
            loss=restomer.loss(noisys+noises[i],noisys)
            loss.backward(retain_graph=True)
            optimizer.step()
            with torch.no_grad():
                total_loss+=loss.sum().detach()#返回tensor，要避免计算图不释放，导致内存溢出
            del loss
    return total_loss
def draw(mean_loss_noisy,mean_loss_clean,noisys,cleans,epoch):
    outs=restomer(noisys)
    writer.add_scalar('noisy_loss',mean_loss_noisy,epoch)
    writer.add_scalar('clean_loss',mean_loss_clean,epoch)
    writer.add_scalar('psnr',psnr(outs,cleans),epoch)
    writer.add_image('noisy',make_grid(convert_to_rgb255(noisys)),epoch)
    writer.add_image('clean',make_grid(convert_to_rgb255(cleans)),epoch)
    writer.add_image('out',make_grid(convert_to_rgb255(outs)),epoch)
def main(): 
    for epoch in tqdm(range(5000)):
        total_loss_noisy,total_loss_clean=0,0
        for noisys,cleans in train_loader:
            noisys=noisys.to(device=device)
            cleans=cleans.to(device=device)
            total_loss_noisy+=train(noisys)
            with torch.no_grad():
                total_loss_clean+=restomer.loss(noisys,cleans).sum().detach()
        with torch.no_grad():
            draw(total_loss_noisy,total_loss_clean,noisys,cleans,epoch)
main()
</code></pre>
<p>I want to reuse the output of noise_extractor,how can i achieve it?<br>
The code above hit an error:<br>
one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [3, 64, 3, 3]] is at version 2; expected version 1 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!</p>

ptrblck: <p>Could you describe your use case a bit more and especially how you would like to “reuse” this tensor, please?</p>
Why .cuda() always allocate GPU:0?

kangkang_li: <p>Only  b.to(device) can move b to GPU:1?</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/2X/4/40f370fc0a438d98fa5b88cc50f3ee773599afdd.png" alt="image" data-base62-sha1="9gA7Z3tLIxz9n2mG4OIlHQMQpCB" width="613" height="238"></p>

sai_tharun: <p>if u have 4 graphic cards,u can choose which one to use.<br>
By deault it will be cuda:0</p>

kangkang_li: <p>I have two,so is there a way to change the default gpu?</p>

sai_tharun: <p>you can try this<br>
<code>torch.cuda.set_device(1)</code></p>

JuanFMontesinos: <p>It’s easier to use the flag CUDA_VISIBLE_DEVICES=‘1’ rather than coding using set_device.</p>
<p>If you need that it means you are working with a single gpu.</p>
<p>You can call .cuda(idx) which allocates in the gpu you want if you use several gpus.</p>

Deepali: <p>CUDA_VISIBLE_DEVICES=“1” exposes only 1 GPU for the program. However, when torch uses the GPUS it starts ordering from 0. So <code>GPU 1</code> on system is mapped to <code>GPU 0</code> in the program.</p>

Dreamer_dev: <p>Your advice saved me a lot of time and energy <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=12" title=":smiley:" class="emoji" alt=":smiley:" loading="lazy" width="20" height="20"> Thanks, man! I used the command ‘set CUDA_VISIBLE_DEVICES=1’ before running the code to specify the GPU.</p>
QAT about ConvTranspose2d

geoffzhang: <p>please, can the pytorch1.2 suport the QAT of ConvTranspose2d?</p>

Vasiliy_Kuznetsov: <p>hi <a class="mention" href="/u/geoffzhang">@geoffzhang</a> , this is supported in PyTorch v1.13, hope that works for you!</p>
Difference between `torch.qint8` and `torch.int8`

ganler: <p>Just curious why do we need <code>qint8</code> when there is already <code>int8</code>? Is it because <code>qint8</code> has a different and more efficient binary layout than that of the int8? Thanks!</p>

Vasiliy_Kuznetsov: <ol>
<li>int8 is an integer type, it can be used for any operation which needs integers</li>
<li>qint8 is a quantized tensor type which represents a compressed floating point tensor, it has an underlying int8 data layer, a scale, a zero_point and a qscheme</li>
</ol>
<p>One could use torch.int8 as a component to build quantized int8 logic, that’s not how PyTorch does it today but we actually plan to converge towards this approach in the future.</p>
Reverse nn.Embedding

safin_salih: <p>how do you go back from nn.Embedding  to the original discrete values?</p>

ptrblck: <p>In case you have the feature vector, you could calculate the difference and select the index with a 0 (or close to 0) diff.</p>

safin_salih: <p>I’m not sure if I understand what you mean. Let’s say you have some discrete valued vector that you want to represent in the latent space, how would I get feature vector? Like the feature vectors from a given model, extracted and take differences?</p>

anantguptadbl: <pre><code class="lang-auto">a = torch.nn.Embedding(10, 50)
# Discreet values
a.weight

# Indexer
a(torch.LongTensor([1,1,0]))
</code></pre>

safin_salih: <p>that still gives a real-valued vector. Want to reverse this operation. Something akin to encoder decoder .</p>

ptrblck: <p>The feature vector would be the output of the embedding layer and you could calculate the difference afterwards to get the index back:</p>
<pre><code class="lang-python">emb = torch.nn.Embedding(10, 50)
x = torch.tensor([5])

out = emb(x)
out.shape
emb.weight.shape

rev = ((out - emb.weight).abs().sum(1) &lt; 1e-6).nonzero()
print(rev)
# &gt; tensor([[5]])
</code></pre>

anantguptadbl: <p><a class="mention" href="/u/safin_salih">@safin_salih</a> Understood</p>
<p>You can use this</p>
<pre><code class="lang-auto">a = torch.nn.Embedding(10, 50)
b = torch.LongTensor([2,8])
results = a(b)

def get_embedding_index(x):
    results = torch.where(torch.sum((a.weight==x), axis=1))
    if len(results[0])==len(x):
        return None
    else:
        return results[0][0]

indices = torch.Tensor(list(map(get_embedding_index, results)))
indices
</code></pre>
<pre><code class="lang-auto">tensor([2., 8.])
</code></pre>

J_Johnson: <p>UPDATE: I did a revamp on this to make it more memory efficient via <code>.expand()</code>, and consequently faster.</p>
<p>Just a slightly more robust definition that handles cases of <code>x.dim&gt;0</code>:</p>
<pre><code class="lang-auto">def emb2indices(output, emb_layer):
    # output is size: [batch, sequence, emb_length], emb_layer is size: [num_tokens, emb_length]
    emb_weights = emb_layer.weight

    # get indices from embeddings:
    emb_size = output.size(0), output.size(1), -1, -1
    out_size = -1, -1, emb_weights.size(0), -1
    out_indices = torch.argmin(torch.abs(output.unsqueeze(2).expand(out_size) -
                                    emb_weights.unsqueeze(0).unsqueeze(0).expand(emb_size)).sum(dim=3), dim=2)
    return out_indices
</code></pre>
<p>For a test:</p>
<pre><code class="lang-auto">emb=nn.Embedding(300000, 100)
test_indices=torch.randint(300000, (1, 100)) #batch 1, length 100
print(test_indices)
test_embed = emb(test_indices)
print(test_embed.size())

indices=emb2indices(test_embed, emb)
print(indices)
</code></pre>
<p>It’s more memory efficient than using stack. However, it can be modified to use less memory if iterating over the embeddings one by one.</p>
How to limit trained parameter range when applying LBFGS?

jie_yang: <p>I have a optimization problem, and I want to a LBFGS slover to limit the parameter’ range between 0 and 1.<br>
the code like this:</p>
<pre><code class="lang-auto">x = torch.zeros((len(model.raw_control_names)), device=args.device)
param = nn.Parameter(x)
optim = torch.optim.LBFGS([param])
for iter in tqdm(range(args.total_iters)):
        def closure():
            optim.zero_grad()
            out = model.getVertx(x)
            loss = F.mse_loss(out, gt, reduction='sum')
            loss.backward()
            print(f'loss :{loss:.6f}')
            return loss

        optim.step(closure)
</code></pre>
<p>I want to limit “param” in range of 0 and 1. Is there method to achieve it?</p>
Forward arguments are not updated in DDP

yzxhd: <p>I am using the example from <a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html#initialize-ddp-with-torch-distributed-run-torchrun" rel="noopener nofollow ugc">Initialize DDP with torch.distributed.run/torchrun</a>. I add a dict <code>state_info</code> as an additional input to the forward function, which will track the state of each forward call. However, the dict is not updated in DDP. The codes are:</p>
<pre><code class="lang-auto">import torch
import torch.distributed as dist
import torch.nn as nn
import torch.optim as optim

from torch.nn.parallel import DistributedDataParallel as DDP

class ToyModel(nn.Module):
    def __init__(self):
        super(ToyModel, self).__init__()
        self.net1 = nn.Linear(10, 10)
        self.relu = nn.ReLU()
        self.net2 = nn.Linear(10, 5)

    def forward(self, x, state_info):
        state_info['tmp'] = 10 # additional input that tracks the state
        return self.net2(self.relu(self.net1(x)))


def demo_basic():
    dist.init_process_group("nccl")
    rank = dist.get_rank()
    print(f"Start running basic DDP example on rank {rank}.")

    # create model and move it to GPU with id rank
    device_id = rank % torch.cuda.device_count()
    model = ToyModel().to(device_id)
    ddp_model = DDP(model, device_ids=[device_id])

    loss_fn = nn.MSELoss()
    optimizer = optim.SGD(ddp_model.parameters(), lr=0.001)

    optimizer.zero_grad()
    state_info = {'tmp': 0}
    print(state_info)
    outputs = ddp_model(torch.randn(20, 10), state_info)
    labels = torch.randn(20, 5).to(device_id)
    loss_fn(outputs, labels).backward()
    optimizer.step()
    print(state_info) # The dict is not updated here

if __name__ == "__main__":
    demo_basic()
</code></pre>
<p>For the second <code>print</code> function, <code>state_info</code> is not changed, i.e. not changed to 10. When DDP is not used, <code>state_info</code> is changed. Is it a feature of DDP? Is there any way to solve it?</p>

rvarm1: <p>This seems unexpected, would you mind filing an issue to PT with this repro so we can take a look? thanks!</p>

yzxhd: <p>Yes. Here is the link for the issue:<br>
<a href="https://github.com/pytorch/pytorch/issues/92128" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/issues/92128</a></p>
Is it possible to list all the tensors in a computational graph?

rinkujadhav2013: <p>I have a custom layer where I decorate the forward and backward functions with <code>@torch.inference_mode()</code>. The layer receives some tensors where it places the output in the forward pass and grads in another tensor in the backward pass. My suspicion is that it hangs on to some intermediate tensors (which should be deleted as soon as the function ends) because the memory keeps going up until OOM and switching the layer with another layer solves the issue.</p>
<p>Can I somehow verify that this, in fact, is the case? So, basically, is there a way to find out all the tensors that a computational graph is hanging on to?</p>
Torch.autograd.grad calculate grads of multiple scalars w.r.t the same input tensor in a batch manner

ZXYFrank: <p>Hi! I am new to manipulating over <code>torch.autograd</code> and the gradient flow. I encounter with a problem of <strong>Acceleration Gradient Calculation</strong></p>
<p>I have a YOLO net $\mathcal{N}$. It receives a video frame at every timestep and provide some outputs ${\mathbf{d}}_{i} = \mathcal{N}(\mathbf{I})$, where $\mathbf{I}$ is the image tensor of the frame and $\mathbf{d}$ is a detection output, i.e. <code>[x,y,w,h,...]</code></p>
<p>Assume that there are <strong>20</strong> detection boxes after NMS, i.e. len(${\mathbf{d}}$) = 20, now I need to calculate $\frac{\partial x_{i}}{\partial \mathbf{I}}$ for <code>i = 1:20</code>.</p>
<p>Since $\mathbf{d}<em>{i} = \mathcal{N}(\mathbf{I}),x</em>{i} = \mathbf{d}<em>{i}[0]$, $x</em>{i}$ is included in the computation graph containing $\mathbf{I}$, $\frac{\partial x_{i}}{\partial \mathbf{I}}$ can be obtained by <code>x_i.backward(retain_graph = True)</code> or <code>torch.autograd.grad(x_i, img_tensor, retain_graph = True)</code>(if I’m not mistaken.)</p>
<p>Now I calculate 20 gradients in a <strong>serial manner</strong>.</p>
<pre><code class="lang-python">res = []
for x_i in x_list:
    _grad = torch.autograd.grad(x_i, img_tensor, retain_graph=True) # bp pass
    res.append(_grad[0])
</code></pre>
<p>If forward pass of $\mathcal{N}$ takes about <code>T</code> seconds(~0.1s), the bp pass takes also approximately <code>T</code>s(~0.15s).</p>
<p>However the serial computations will take <code>20 x T</code> seconds! That’s terrible.</p>
<p>I wonder if there is a <strong>parallel way</strong>, utilizing the power of <strong>GPU</strong> I mean, to shorten <code>20 x T</code>, like</p>
<pre><code class="lang-python">res = torch.autograd.grad(x_list, [img_tensor] * len(reg_sum_list))

# _ress[0] is _ress[1]
# True
</code></pre>
<p>It does not work🤣</p>
Multi Class Classification with "Bad" Class

hadarshavit: <p>Hi,</p>
<p>I’m using multi class classification but for some instances choosing one class is a “bad” decision than other (e.g., there are weights per-instance-per class for “how bad” is this class).<br>
I’m currently using cross entropy loss, is there a better loss function for it?</p>
<p>Thanks</p>

KFrank: <p>Hi Hadar!</p>
<aside class="quote no-group" data-username="hadarshavit" data-post="1" data-topic="170149" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hadarshavit/48/47537_2.png" class="avatar"> hadarshavit:</div>
<blockquote>
<p>there are weights per-instance-per class for “how bad” is this class</p>
</blockquote>
</aside>
<p>If I understand your use case correctly, you might find some ideas about<br>
how to address your issue in this post:</p>
<aside class="quote quote-modified" data-post="4" data-topic="157369">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/create-a-variation-of-cross-entropy-loss-with-per-pair-of-classes-weights/157369/4">Create a Variation of Cross Entropy Loss with Per Pair of Classes Weights</a> 
  </div>
  <blockquote>
    Hi David! 

Cross entropy (by definition) doesn’t work this way.  But you can write 
some other loss function that does have “per-pair” penalties. 
I give a couple of possibilities in the example script, below.  misclassA() 
just weights the probability for an incorrect prediction by the per-pair 
weight given in your matrix D. 
My guess is that such a loss function won’t train very well.  In my experience, 
the logarithmic divergence for bad predictions in cross entropy seems to 
be very helpfu…
  </blockquote>
</aside>

<p>Best.</p>
<p>K. Frank</p>
Can't understand why built-in MNIST DataLoader uses less memory than custom

Sam_Lerman: <p>It looks like the built-in DataLoader for MNIST fully loads the dataset into RAM for each CPU worker, creating replicas. However, I tried a custom DataLoader that uses memory mapping and loads from hard disk and found that the memory usage is higher! I can’t figure out how that is. Maybe it’s the exact stats I’m collecting via Activity Monitor on MacBook.</p>
<p>Unfortunately I can’t provide a code snippet.</p>
How to make my label(y).shape = torch.Size([ ])

Taimoor-R: <p>Hi,<br>
My dataset frame and label sizes are below what i want to understand is how to make label(y)size = torch.Size([ ]) currently the code outputs Frame(x) shape :: torch.Size([32, 3, 224, 224]) Label(y) shape :: torch.Size([32])</p>
<p>This is my code:</p>
<p>def <strong>getitem</strong>(self, idx):</p>
<pre><code class="lang-auto">    if torch.is_tensor(idx):
        idx = idx.tolist()
    frames = []
    for fr in range(idx, idx+self.seq_len):  # frames from idx to idx+seq_len
        img_name = os.path.join(self.frames_list.iloc[fr, 0])  # path to image
        image = Image.open(img_name)
        image = image.resize((self.img_width, self.img_height))

        if self.transform:
            image = self.transform(image)
        frames.append(image)

    frames = torch.stack(frames)
    frames = frames.permute(0, 1, 2, 3)
    frames = torch.squeeze(frames, dim=1)
    frames = (frames-torch.mean(frames))/torch.std(frames)*255
    lab = np.array(self.frames_list.iloc[idx:idx + self.seq_len, 1])
    labels = torch.tensor(lab, dtype=torch.float)
    #labels = torch.empty(lab, dtype=torch.float)
    #labels = torch.zeros(lab, dtype=torch.float)
    sample = (frames, labels)
    return sample 
</code></pre>

ptrblck: <p>You can index the <code>label</code> tensor to create a scalar:</p>
<pre><code class="lang-python">label = torch.randint(0, 10, (32,))
print(label.shape)
# torch.Size([32])
print(label[0].shape)
# torch.Size([])
</code></pre>