akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Accuracy drop after prepare_qat_fx with no quantization

username: Chame_call
cooked: <p>I want to quantize only backbone from the whole model (simple timm feature extractor) and I get bad results (low metrics) during training so I decided to make sure that the model will have similar to original accuracy if I set qconfig for the backbone (the only quantized part) to None so that no quantization should be applied.</p>
<pre><code class="lang-auto">    qconfig_mapping = get_default_qat_qconfig_mapping("x86")
    qconfig_mapping \
        .set_module_name("backbone", None)
    ...
    model.backbone = prepare_qat_fx(model.backbone, qconfig_mapping, example_inputs)
</code></pre>
<p>but it has appeared that the model still shows bad accuracy (around half of original accuracy) and I’m wondering what’s wrong?<br>
I did check that my module_name is correct so that after calling <code>prepare_qat_fx</code> I don’t have any quantization parameters (like scale, zero_point etc) inside my backbone.<br>
Any thoughts?</p>

username: jcaip
cooked: <p>Hi <a class="mention" href="/u/andrew_holmes">@Andrew_Holmes</a>  can you share a link to a gist to reproduce this error?</p>

username: HDCharles
cooked: <p>you have a global_qconfig that is being used because backbone is not the name of any modules in your model</p>
<p>e.g. if your model was</p>
<pre><code class="lang-auto">root module
&gt; backbone
&gt;&gt;vertebrae
&gt; frontbone
&gt;&gt;sternum
</code></pre>
<p>if you did .set_module_name(“backbone”, None) but left the global qconfig as default,  and then applied quantization to the root model, it would not do quantization to the backbone module or its child (vertebrae), but would apply quantization to frontbone and its child sternum.</p>
<p>if you instead pass only model.backbone to the quantizer, your model now looks like this</p>
<pre><code class="lang-auto">root model
&gt;vertebrae
</code></pre>
<p>so since there’s no module named backbone, it does quantization to vertebrae.</p>
Share python dictionary across ranks

username: heeseokjung
cooked: <p>I have got a problem when I use pytorch lightning in ddp setting. My pl.LightningModule class is like as follows.</p>
<p>class MyClass(pl.LightningModule):<br>
def _<em>init_</em>(self):<br>
self.id2emb = {} # video id : tensor<br>
def validation_step(self):<br>
…<br>
self.id2emb[vid] = self(x)<br>
def validation_epoch_end(self):<br>
…<br>
for vid in video_list:<br>
emb = self.id2emb[vid]  # key error!</p>
<p>Since different ranks have their own dictionaries, they are not shared between ranks.<br>
How can I solve this problem?<br>
What I want to do is only rank 0 can access entire video id list and only rank 0 has self.id2emb, which is mapping to entire video id : each embedding.<br>
Please help me…</p>
Does torch::cuda::nccl::reduce has stream sync?

username: raninbowlalala
cooked: <p>I want to use torch::cuda::nccl::reduce(inputs, output, root, op, streams, user_comms); to do reduce, I pass the inputs, output, and root.<br>
I saw the code from torch/csrc/cuda/nccl.cpp, and I didn’t saw the cuStreamSynchronize(), but in nccl example there are need cuStreamSynchronize() after ncclReduce(). So if there do not need cuStreamSynchronize() how to make sure the data is reduce done?:<br>
void reduce(<br>
const std::vector<a>at::Tensor</a>&amp; inputs,<br>
at::Tensor&amp; output,<br>
int32_t root,<br>
int32_t op,<br>
const stream_list&amp; streams,<br>
const comm_list&amp; user_comms) {<br>
<span class="hashtag-raw">#ifdef</span> USE_NCCL<br>
using namespace torch::cuda::nccl::detail;<br>
TORCH_CHECK(<br>
root &gt;= 0 &amp;&amp; static_cast&lt;size_t&gt;(root) &lt; inputs.size(), “invalid root”);</p>
<p>check_inputs(inputs, output, root, 1, 1);<br>
const auto len = inputs.size();</p>
<p>auto data_type = to_nccl_data_type(inputs[0]);</p>
<p>const auto count = inputs[0].numel();<br>
auto comms_ref = user_comms.empty() ? get_communicators(inputs)<br>
: ArrayRef&lt;ncclComm_t&gt;(user_comms);</p>
<p>AutoNcclGroup nccl_group_guard;<br>
at::cuda::OptionalCUDAGuard device_guard;<br>
for (const auto i : c10::irange(len)) {<br>
int device = inputs[i].device().index();<br>
device_guard.set_index(device);<br>
// Default to the current stream<br>
const auto stream = (streams.empty() || !streams[i])<br>
? at::cuda::getCurrentCUDAStream(device).stream()<br>
: streams[i]-&gt;stream();</p>
<pre><code>ncclComm_t comm = comms_ref[i];
NCCL_CHECK(ncclReduce(
    inputs[i].data_ptr(),
    static_cast&lt;std::remove_cv_t&lt;decltype(i)&gt;&gt;(root) == i
        ? output.data_ptr()
        : nullptr,
    count,
    data_type,
    to_nccl_red_op(op),
    root,
    to_nccl_comm(comm),
    stream));
</code></pre>
<p>}<br>
<span class="hashtag-raw">#else</span><br>
AT_ERROR(“PyTorch built without NCCL support”);<br>
<span class="hashtag-raw">#endif</span><br>
}</p>

username: ptrblck
cooked: <p>From the <a href="https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/usage/streams.html">NCCL docs</a>:</p>
<blockquote>
<p>NCCL calls are associated to a stream and are passed as the last argument of the collective communication function. The NCCL call returns when the operation has been effectively enqueued to the given stream, or returns an error. The collective operation is then executed asynchronously on the CUDA device. The operation status can be queried using standard CUDA semantics, for example, calling cudaStreamSynchronize or using CUDA events.</p>
</blockquote>
<p>If you use the default <code>CUDAStream</code> you can enqueue more kernels as is done with any other kernel. However, if you explicitly use custom streams you are responsible for a proper synchronization.</p>
How to obtain the C++ operator of a function's underlying call

username: BiophiliaSWDA
cooked: <p>If I have a function,</p>
<pre><code class="lang-auto">import torch
layer = torch.nn.Conv2d(1, 6, 5)
</code></pre>
<p>When torch.nn.Conv2d runs, how to get the C++ file of this system call?</p>

username: ptrblck
cooked: <p>You could run the <code>python</code> executable via <code>gdb --args python script.py args</code> and use breakpoints to check the backend functions or directly step through the execution.</p>
Datapip TFRecordLoader is extremly slow

username: nima_rafiee
cooked: <p>Hi</p>
<p>I’m trying to use datapipe wit Dataloader2 to read from TFRecord files. When I increase the batch_size (e.g: to 32), the data loading process becomes extremely slow. Any suggestions how can I optimise the pipeline that works with larger batch sizes as well?</p>
<pre><code class="lang-auto">def build_datapipes(path):
    datapipe = FSSpecFileLister([path])
    datapipe = datapipe.filter(filter_fn=filter_tf_records)
    datapipe = datapipe.shuffle().sharding_filter()
    datapipe = datapipe.open_files_by_fsspec(mode="rb")
    datapipe = datapipe.prefetch(1024)
    datapipe = TFRecordLoader(datapipe=datapipe, spec=None) 
    datapipe = datapipe.batch(batch_size=32, drop_last=True)
    datapipe = Collator(datapipe)
    return datapipe


datapipe = build_datapipes(path_to_tfrecord_files)
ms = MultiProcessingReadingService(num_workers=8,
                                     worker_prefetch_cnt=8,
                                     main_prefetch_cnt=8,
                                     )
dl2 = DataLoader2(datapipe=datapipe, reading_service=ms)
start = default_timer()
count = 0
for i, x in enumerate(dl2):
    count += 1
end = default_timer()
print(end - start)
dl2.shutdown()

</code></pre>
Torchscript jit scripting & Faster rCNN issue

username: Stefano
cooked: <p>Hello community,</p>
<p>I hope you’re all doing well. For the past few weeks, I’ve encountered a persistent issue that has me truly stumped, and I’m reaching out in the hopes that someone might have encountered this before or can shed some light.</p>
<p>Here’s the situation: I’ve exported a TorchScript model using (torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True)<br>
):<br>
However, every time I attempt to infer with this model, the output I receive is an empty list: <code>[]</code>.</p>
<p>I was hopping to get a dictionary including boxes = instances, labels, scores.</p>
<p>Has anyone successfully read the output of a <code>fasterrcnn</code> model during TorchScript inference? I would be profoundly grateful for any guidance or suggestions.</p>
<p>Thank you in advance for considering my query. <img src="https://discuss.pytorch.org/images/emoji/apple/pray.png?v=12" title=":pray:" class="emoji" alt=":pray:" loading="lazy" width="20" height="20"></p>
<p>Stephano</p>
Models for Model Parallelism training

username: iamexperimentingnow
cooked: <p>Hi everyone,</p>
<p>Firstly I don’t know whether this question should be asked here or not. I’m sorry if my isn’t relevant in this forum.</p>
<p>I would like to practice training deep learning models in Model parallelism method or Tensor parallelism method. Previously, I worked more in Computer Vision domain. It would be helpful if you can recommend me any CV application/pretrained model or any relevant examples to practice model parallelism or Tensor parallelism.</p>
<p>During my google search, I noticed there are many models available to practice in NLP domain.</p>
<p><a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>Thanks</p>

username: ptrblck
cooked: <p>I don’t know if there are “stable” model sharding APIs for CNNs, but you could take a look at e.g. <a href="https://github.com/NVIDIA/apex/blob/9fc94b7d6db1b178adf9a6e92750f070dd9f825d/apex/contrib/bottleneck/bottleneck.py#L265">this experimental interface</a>. You are also right that the a lot of model sharding approaches are used for LLMs, e.g. in the <a href="https://github.com/NVIDIA/Megatron-LM">Megatron repository</a>.</p>
Pytroch on iOS in run time

username: Lokesh1
cooked: <p>I need to run the custom Python script locally on iOS and get the retune values. I am unable to find the right steps to integrate PyTorch into my iOS application.</p>

username: ptrblck
cooked: <p>You could check the <a href="https://pytorch.org/mobile/ios/">legacy workflow</a> or the new <a href="https://pytorch.org/executorch/stable/demo-apps-ios.html">ExecuTorch workflow</a> (which is still in development if I’m not mistaken).</p>

username: Lokesh1
cooked: <p>Thanks for your response. I am looking for .Framework which can be directly integrated into existing iOS application  <a class="mention" href="/u/ptrblck">@ptrblck</a></p>

username: kimishpatel
cooked: <p>Anthony, can you respond to this?</p>

username: shoumikhin
cooked: <p><a class="mention" href="/u/lokesh1">@Lokesh1</a>, as <a class="mention" href="/u/ptrblck">@ptrblck</a> mentioned above, for the legacy workflow, you can use Cocoapods or <a href="https://ossci-ios.s3.amazonaws.com/libtorch_ios_1.10.0.zip" rel="noopener nofollow ugc">download</a> a .zip archive with prebuilt libs and headers (and make a framework with them if you wish), and for ExecuTorch you can build a .xcframework locally using <code>build_apple_frameworks</code> script as described <a href="https://pytorch.org/executorch/stable/demo-apps-ios.html#build-runtime-and-backends" rel="noopener nofollow ugc">here</a></p>
Type casting problem when using torch in C++/CUDA

username: Liu_Altair
cooked: <p>Hello community, now I’ve encountered a problem related to type casting.<br>
In my python file, I want to call the function <code>func</code> with:</p>
<pre><code class="lang-auto">a = torch.rand(size).to(torch.float16).to("cuda:0")
b = torch.rand(size).to(torch.float16).to("cuda:0")
func(a, b)
</code></pre>
<p>Where <code>func</code> is defined in a .cu file as follow:</p>
<pre><code class="lang-auto">void func(const torch::Tensor input, torch::Tensor output);
</code></pre>
<p>Inside <code>func</code> function, I will call a CUDA kernel:</p>
<pre><code class="lang-auto">__global__ void kernel(const half* _input, half* _output);
</code></pre>
<p>So, to implement <code>func</code>, I need to do following things:</p>
<pre><code class="lang-auto">void func(const torch::Tensor input, torch::Tensor output) {
    // This half type is defined in cuda_fp16.h
    half* __input  = reinterpret_cast&lt;half*&gt;(input.data_ptr&lt;half&gt;());
    half* __output = reinterpret_cast&lt;half*&gt;(output.data_ptr&lt;half&gt;());
    kernel&lt;&lt;&lt;block, thread&gt;&gt;&gt;(__input, __output);
    cudaDeviceSynchronize();
}
</code></pre>
<p>But when I execute this, I encounter the error:</p>
<pre><code class="lang-auto">lilac.cpython-310-x86_64-linux-gnu.so: undefined symbol: _ZNK2at10TensorBase8data_ptrI6__halfEEPT_v
</code></pre>
<p>I guess this is caused by the type casting, and I wonder which data type should I use in the .cu file, or how to change the tensor datatype in the py file.</p>

username: ptrblck
cooked: <p>Try to use <code>at::Half</code> instead.</p>
Gradient of stacked forward passes?

username: mzw
cooked: <p>Hello</p>
<p>I’m trying to write a pytorch model that flies an aircraft to the point (0,0,0)</p>
<p>The loss function looks approximately like this:</p>
<p>aircraft_state = get_initial_state() ← position, velocity, etc.<br>
pilot_decision = get_initial_pilot_decision() ← constant thrust forward</p>
<p>for i in range(number_of_time_steps):<br>
pilot_decision = model(aircraft_state)<br>
aircraft_state = iterate_state(pilot_decision) ← moves the aircraft based on the pilot decision</p>
<p>loss = aircraft_state.get_position().get_norm()</p>
<p>After implementation, I found that this isn’t working. I’m not exactly sure what it does, but the loss changes very little when I start the training.<br>
Could it be that only the last forward pass in the for loop is used for the backpropagation?<br>
Is there a way to get pytorch to calculate the gradient based on all consecutive model calls?</p>
Training Data for Denoising Diffusion Probabilistic Model

username: 1alex
cooked: <p>Dear Community,</p>
<p>I am currently training a denoising diffusion probabilistic model (DDPM) but the results I obtain differ from the training data. Each training image is divided into four quadrants, where each quadrant depicts an independent image (each image has a white frame). In other words I arranged four images in a 2x2 grid.</p>
<p>The neural network has 250 million trainable parameters and I have 40 000 images for training (I can obtain more if I need to, but this is a time consuming task).</p>
<p>My questions:</p>
<ul>
<li>How many images do I need to train the DDPM with 250 million trainable parameters? (I somewhere read of a 10:1 or at least 1:1 ratio of training data to trainable parameters.)</li>
<li>How can I imagen the influence of training data to accuracy of the model? Does each training data point increase the accuracy of the model in the same way (let’s say 1 data set increases accuracy by 0.1%(just to name some random number)) or is the relationship between training data and accuracy non linear (e.g. an S-curve if I plot the accuracy over the training data points)?</li>
</ul>

username: J_Johnson
cooked: <p>Typical initial training of DDPMs take A LOT of compute for good results and tons of data, like LAION-5B dataset(5.85 billion images).</p>
<p>You might be better off starting with a pre-trained open source model downloaded fromHuggingface and then just train LoRA weights over top of it, for your use case.</p>
<p>Just by way of example, Stable Diffusion 2.0 took 79,000 A100 hours to train. That’s an 860 million parameter U-Net and 123 million param encoder.</p>
<p>But training LoRA weights overtop something is very quick and can be done on a 24GB GPU in less than a day.</p>
Py parallel execution of a model with many/few features

username: solocazzimiei
cooked: <p>Since I have unused GPU memory to fit during serial forecasting of several k model and features.<br>
I’d like to receive your help to implement smart a way of fit such memory (and speed up/ shorten the whole model swipe) by parallelization of picking up a bunch of pre-learned models specific for each feature, pick-up a bunch of different independent features at time and feed concurrently the Cuda GPU with such models to obtain the forecast results.<br>
I’m specifically referring to multivariate forecast time series.<br>
I’ve tried implement both threads,(but seams GIL is incompatible with GPU) and multiprocess but always without success.</p>
Replicating conv autoencoder for anomaly detection

username: Kennes
cooked: <p>Hi,</p>
<p>I’m trying to train an autoencoder on the hazelnut dataset of MVTec AD for reconstruction to detect anomalies.</p>
<p>I’m am trying to replicate the results of this study: <a href="https://arxiv.org/pdf/2008.12977.pdf" rel="noopener nofollow ugc">https://arxiv.org/pdf/2008.12977.pdf</a><br>
In particular, the simpler setup used in the study as reference, with features a simple AE model with no skip connection and no stain model. Just a plain 6 layer encoder and 6 layer decoder.<br>
The encoder layers have 16-32-64-128-256-512 filters, strided conv with kernel of 5, BatchNorm, and leakuRelu activation. The decoder have the same filter sizes, and have upsampling, conv (kernel=5), batchnorm, leakyRelu and sigmoid at the final layer.<br>
As in the paper, i’m using MSE as loss, Adam as optimizer (batch of 16) and i’m augmenting the input data with some cropping, rotations and flipping.</p>
<p>Although I feel like I have been doing what is described in the paper, my model is unable to learn any anything meaningful and only shows very rough and blurred white shadows for reconstruction:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/c/cc34604453d4589d7b60f01887cecb28b8861375.png" data-download-href="https://discuss.pytorch.org/uploads/default/cc34604453d4589d7b60f01887cecb28b8861375" title="hazelnuts"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/c/cc34604453d4589d7b60f01887cecb28b8861375_2_690x265.png" alt="hazelnuts" data-base62-sha1="t8tCnUTXSbxNHpA70PJmBdIVumh" width="690" height="265" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/c/cc34604453d4589d7b60f01887cecb28b8861375_2_690x265.png, https://discuss.pytorch.org/uploads/default/original/3X/c/c/cc34604453d4589d7b60f01887cecb28b8861375.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/c/c/cc34604453d4589d7b60f01887cecb28b8861375.png 2x" data-dominant-color="878787"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">hazelnuts</span><span class="informations">915×352 95.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The code and my notebook is available here : <a href="https://nbviewer.org/urls/gist.githubusercontent.com/KenN7/8e835334a669599b42c4144d3e68f617/raw/2a59b578fc9e5febc1e00f535eb75ec5e72b8f35/gistfile1.txt" class="inline-onebox" rel="noopener nofollow ugc">Jupyter Notebook Viewer</a></p>
<p>I don’t get what I am doing wrong here, I know that some things could improve the situation (e.g. using SSIM as loss), but I’m more interested in achieving the results of the paper ( 0.9 AUC% in the setup ). Do you have any idea what could be wrong ?</p>
Is there a way to find loop?

username: ruka
cooked: <p>I’m trying to migrate pytorch model into our own c++ framework. Currently I’m using torch.jit.trace to get the graph of the model, which is convenient, I am able to get most ops through the node name like ‘aten::add’, ‘aten::conv2d’, but for the following model:</p>
<blockquote>
<p>class A(torch.nn.Module):<br>
def <strong>init</strong>(self):<br>
super(A,self).<strong>init</strong>()<br>
self.layer1 = torch.nn.Linear(10,20)<br>
self.layer2 = torch.nn.Linear(20,10)</p>
<pre><code>def forward(self,xx):
    # xx=torch.randn(50,3,10)
    for i in range(xx.shape[0]):
        x = xx[i]
        x = self.layer1(x)
        x = self.layer2(x)
    return x
</code></pre>
</blockquote>
<p>I have trouble finding if the model is using the for loop, and which node(s) is within the for loop. Is there a way to find out the loop part?</p>
How acos/asin/atan in pytorch handle half input on gpu?

username: garfieldxs
cooked: <p>I see that acos for gpu has implementation here.</p><aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/08dbfecdbdf2af6f66b3226881c71d8977431197/aten/src/ATen/native/cuda/UnaryGeometricAcosKernel.cu#L50">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/08dbfecdbdf2af6f66b3226881c71d8977431197/aten/src/ATen/native/cuda/UnaryGeometricAcosKernel.cu#L50" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/08dbfecdbdf2af6f66b3226881c71d8977431197/aten/src/ATen/native/cuda/UnaryGeometricAcosKernel.cu#L50" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/08dbfecdbdf2af6f66b3226881c71d8977431197/aten/src/ATen/native/cuda/UnaryGeometricAcosKernel.cu#L50</a></h4>



    <pre class="onebox"><code class="lang-cu">
      <ol class="start lines" start="40" style="counter-reset: li-counter 39 ;">
          <li>        });</li>
          <li>#endif</li>
          <li>  } else {</li>
          <li>    AT_DISPATCH_FLOATING_TYPES_AND2(</li>
          <li>        ScalarType::Half,</li>
          <li>        ScalarType::BFloat16,</li>
          <li>        common_dtype,</li>
          <li>        "acos_cuda",</li>
          <li>        [&amp;]() {</li>
          <li>          gpu_kernel(iter, [] GPU_LAMBDA(scalar_t a) -&gt; scalar_t {</li>
          <li class="selected">            return ::acos(a);</li>
          <li>          });</li>
          <li>        });</li>
          <li>  }</li>
          <li>}</li>
          <li></li>
          <li>REGISTER_DISPATCH(acos_stub, &amp;acos_kernel_cuda);</li>
          <li></li>
          <li>} // namespace at::native</li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p>
But when I tried to compile the following code</p>
<pre><code class="lang-auto">// Type your code here, or load an example.
#include &lt;cuda.h&gt;
#include &lt;cuda_fp16.h&gt;
#include &lt;cuda_runtime.h&gt;

__global__ void asin_kernel(half *input, half *output, int size) {
    int tid = blockIdx.x * blockDim.x + threadIdx.x;
    if (tid &lt; size)  {
        output[tid] = ::acos(input[tid]);
    }
}

</code></pre>
<p>I got error</p>
<pre><code class="lang-auto">&lt;source&gt;(9): error: more than one instance of overloaded function "acos" matches the argument list:
            function "std::acos(long double)" (declared at line 91 of /opt/compiler-explorer/gcc-10.2.0/include/c++/10.2.0/cmath)
            function "std::acos(float)" (declared at line 87 of /opt/compiler-explorer/gcc-10.2.0/include/c++/10.2.0/cmath)
            argument types are: (half)
          output[tid] = ::acos(input[tid]);
                        ^

1 error detected in the compilation of "&lt;source&gt;".
Compiler returned: 2
</code></pre>
<p>how does pytorch get half input with acos work?</p>
Is it possible to use hook to measure latency of each blocks running on CUDA?

username: deJQK
cooked: <p>In this blog the author mentioned the correct way to measure latency on GPU using <code>torch.cuda.event</code> and proper synchronization. But this cannot give the latency profile for each block. Is it possible to register forward hook on blocks or layers to get each block’s latency? It seems that hook are not running along with the forward, but maybe before or after forward, so not sure if it can be useful for this. Thanks.</p>
TorchData: Sharding, Caching, and Prefetching

username: cinjon
cooked: <p>Right now, my DataPipe and DataLoader2 look like the following. They are reading data from R2 using an AWS-like bucket_path with an endpoint_url specification. This is surprisingly slow and I think I must be doing something wrong.</p>
<p>Some questions:</p>
<ol>
<li>If I was to do an on-disk cache, where should I put it in the pipeline? I tried putting an <code>on_disk_cache</code> before the <code>open_file_by_fsspec</code> and then the <code>end_caching</code> after it, but that stalled the pipeline completely.</li>
<li>The <code>prefetch(5)</code> doesn’t actually seem to do anything. Am I ussnig it wrong?</li>
<li>I want this pipeline to continue indefinitely. How do I use <code>repeat</code> with it? Do I put it … afterwards? That seems inefficient.</li>
</ol>
<pre><code class="lang-auto">dp = IterableWrapper([bucket_path]).list_files_by_fsspec()
dp = dp.sharding_filter()    
dp = dp.open_files_by_fsspec(mode="rb")
dp = dp.prefetch(5)
dp = dp.load_from_tar()  
dp = dp.groupby(groupby, group_size=2, guaranteed_group_size=2)
dp = dp.map(process)
dp = ExpandDataIPUPipe(dp, predict_ms=predict_ms)
dp = dp.shuffle(buffer_size=10)
dp = dp.batch(batch_size)
dp = dp.map(collate)
mp = MultiProcessingReadingService(num_workers=num_workers) # typically 12
dl = DataLoader2(dp, reading_service=mp)

def collate(batch):
  wavs = torch.cat([k[0].unsqueeze(0) for k in batch], axis=0)
  labels = [k[1] for k in batch]
  labels = torch.tensor(labels).to(torch.int64)
  if wavs.shape[1] == 2:
    wavs = wavs.mean(1)
  return wavs, labels

def groupby(row):
  return os.path.basename(row[0]).split(".")[0]

def process(row):
  if row[0][0].endswith('json'):
      stream_json_wrapper = row[0][1]
      stream_wav_wrapper = row[1][1]
  else:
      stream_json_wrapper = row[1][1]
      stream_wav_wrapper = row[0][1]
  labels = stream_json_wrapper.read()
  labels = json.loads(labels.decode('utf-8'))
  wav = io.BytesIO(stream_wav_wrapper.read())
  wav, _ = torchaudio.load(wav)    
  return wav, labels
</code></pre>
Train Bart model loaded from torch hub

username: dav_kor
cooked: <p>I am trying to train a Bart model downloaded from torch hub, but i got the following error when calling the train method on the model.</p>
<pre><code class="lang-auto">import torch
bart = torch.hub.load('pytorch/fairseq', 'bart.base')
model = bart.model
model.train()
</code></pre>
<p>I got the following error message</p>
<pre><code class="lang-auto">File "/home/david/.cache/torch/hub/pytorch_fairseq_main/fairseq/models/fairseq_model.py", line 220, in train
    raise RuntimeError("cannot train after make_generation_fast")
RuntimeError: cannot train after make_generation_fast
</code></pre>
Could not load library libcudnn_cnn_train.so.8 in new version

username: cinjon
cooked: <p>I have a model that uses torchaudio.transforms.MelSpectrogram and torchaudio.models.Conformer. It works in torch==2.0.0, torchaudio==2.0.1, and torchdata==0.6.0. However, it does not work in the latest packages - torch==2.1.0 torchaudio==2.1.0 torchdata==0.7.0. This is an issue for me as I need stuff from the later packages.</p>
<p>The problem that arises is this error:</p>
<pre><code class="lang-auto">Could not load library libcudnn_cnn_train.so.8. Error: /usr/local/cuda/lib64/libcudnn_cnn_train.so.8: undefined symbol: _ZN5cudnn3cnn5infer22queryClusterPropertiesERPhS3_, version libcudnn_cnn_infer.so.8
Traceback (most recent call last):
...
  File "/venvs/bipu2/lib/python3.10/site-packages/torch/_tensor.py", line 492, in backward
    torch.autograd.backward(
  File "/venvs/bipu2/lib/python3.10/site-packages/torch/autograd/__init__.py", line 251, in backward
    Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: GET was unable to find an engine to execute this computation
</code></pre>
<p>It comes up after running <code>loss.backward()</code> and so it’s only in train.</p>
<p>What’s going on?</p>

username: ptrblck
cooked: <p>PyTorch ships with its own CUDA dependencies (including cuDNN) and the error message points to a locally installed cuDNN version. Either uninstall it as a workaround or remove it from the <code>LD_LIBRARY_PATH</code> to allow PyTorch to use its own version.</p>

username: cinjon
cooked: <p>Cool, thanks, that seems to have worked. I appreciate the fast response.</p>
Is there a pytorch-1.2.0 for gpu cp38+cp38 distribution? Or other pytorch version can run in Nvidia GeForce 730 with cp38+cp38? If not, How can I solve the compilation error "error C3861: IsWindowsXPOrGreater: Identifier Not Found"?

username: Chuanwise
cooked: <p>I want to run pytorch on my GPU: Nvidia GeForce 730 which compute capability is sm_35.</p>
<p>I found that the latest pytorch version supports this compute capability is pytorch 1.2.0, but I can’t install it because my pip only allows products with tags:</p>
<pre><code class="lang-auto">Compatible tags: 30
  cp38-cp38-win_amd64
  cp38-abi3-win_amd64
  cp38-none-win_amd64
  cp37-abi3-win_amd64
  cp36-abi3-win_amd64
  cp35-abi3-win_amd64
  cp34-abi3-win_amd64
  cp33-abi3-win_amd64
  cp32-abi3-win_amd64
  py38-none-win_amd64
  py3-none-win_amd64
  py37-none-win_amd64
  py36-none-win_amd64
  py35-none-win_amd64
  py34-none-win_amd64
  py33-none-win_amd64
  py32-none-win_amd64
  py31-none-win_amd64
  py30-none-win_amd64
  cp38-none-any
  py38-none-any
  py3-none-any
  py37-none-any
  py36-none-any
  py35-none-any
  py34-none-any
  py33-none-any
  py32-none-any
  py31-none-any
  py30-none-any
</code></pre>
<p>And there is not a pytorch 1.2.0 or lower versions are compiling with any of them. So I decided to compile source codes, and I met this error (full output is too long, and can be seen at <a href="https://paste.ubuntu.com/p/nNPsYj5QdW/" rel="noopener nofollow ugc">Ubuntu Pastebin</a>):</p>
<pre><code class="lang-auto">error C3861: “IsWindowsXPOrGreater”: 找不到标识符
</code></pre>
<p>Which means “Identifier ‘IsWindowsXPOrGreater’ Not Found”. I try to run commands <code>build</code>, <code>develop</code> or <code>install</code> after run <code>python setup.py clean</code>, but it still occur.</p>
<p>How to solve it? Or Is there a pytorch-1.2.0 for gpu cp38+cp38 distribution? Or other pytorch version can run in Nvidia GeForce 730 with cp38+cp38? If not, How can I solve the compilation error “error C3861: IsWindowsXPOrGreater: Identifier Not Found”?</p>
Transformer Model for Language Modelling in NLP

username: Anirudh_Mukkamala
cooked: <p>In order to understand the LLM’s (concepts in deeper along with coding) I thought to code a small LLM from scratch.<br>
I have completed the coding part but I have some conceptual errors( I guess) due to which I was not able to achieve my desired task.</p>
<h1><a name="purpose-of-model-1" class="anchor" href="#purpose-of-model-1"></a>Purpose of Model</h1>
<p>The Model is attention based transformer model. It aim to create a low level small scale LLM (due to resource constraints I am happy with even a very basic LLM as I am intended to learn the concepts along with its code).</p>
<h1><a name="expected-working-2" class="anchor" href="#expected-working-2"></a>Expected Working</h1>
<p>The Model should be able to do atleast any of the following tasks,</p>
<ol>
<li>text completion</li>
<li>text generation (I.e. Generating the output which is much more relational to input)</li>
</ol>
<p>If everything goes fine then I am thinking about fine tuning it with the normal chat dataset. So I can get a chatbot of my own from scratch.</p>
<h1><a name="problem-3" class="anchor" href="#problem-3"></a>Problem</h1>
<p>Although I have coded the model with the help of GPT and some online resources but I think that I have missed something important (Conceptual Error) due to which I was not able to achieve the results expected.</p>
<h1><a name="model-4" class="anchor" href="#model-4"></a>Model</h1>
<p>My Model - <a href="https://www.kaggle.com/code/anirudhmukkamala/llmwithtransformer" rel="noopener nofollow ugc">llmwithtransformers</a></p>
Ignore_index available in CrossEntropyLoss but not in BCEWITHLOGITSLOSS

username: Jan_Pisl
cooked: <p>I have a segmentation problem where I feed my model images and want it to recognize only a single class. Therefore, I want to use the BCEWithLogitsLoss. However, since there are areas for which I don’t have annotation (but still want to include them in the dataset for context), I want to specify the ignore_index value so that model’s predictions in these areas are not used for computing gradients.</p>
<p>However, the ignore_index is only available in CrossEntropyLoss and not BCEWithLogitsLoss. I don’t understand why? Is there a reason for why it doesn’t make sense that I am missing ?</p>
<p>Also, is my understanding correct that I can accomplish identical behavior to ignore_index by using pos_weight?</p>
<p>Thank you very much!</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Jan_Pisl" data-post="1" data-topic="190945">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jan_pisl/48/16421_2.png" class="avatar"> Jan_Pisl:</div>
<blockquote>
<p>However, the ignore_index is only available in CrossEntropyLoss and not BCEWithLogitsLoss. I don’t understand why? Is there a reason for why it doesn’t make sense that I am missing ?</p>
</blockquote>
</aside>
<p><code>nn.BCEWithLogitsLoss</code> is used for a binary or multi-label classification and accepts “soft” targets. Assuming you work on a binary classification use case it would be uncommon to exclude one class as you would end up predicting the other one only. If you want to mask the loss of specific samples, use an unreduced loss and multiply the sample loss with zeros.</p>

username: Jan_Pisl
cooked: <p>Ok, thanks a lot for the quick response. I think I misunderstood how the parameter works but now it’s clearer <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
Trainable & non-trainable parameters in custom model

username: Matias_Bilkis
cooked: <p>Hello!</p>
<p>I want to create a <strong>custom model using Pytorch</strong>, where I need to multiply inputs with <strong>a matrix containing trainable and non-trainable parameters</strong> <em>(I’m looking to implement a trainable Kalman-filter, with free and fixed parameters)</em>. Furthermore, such matrix has the same parameter in more than one entry.</p>
<p>However I am struggling, (maybe too much!) in training this… any workaround?</p>
<pre><code>class CustomModel(torch.nn.Module):
    def __init__(self,w0):
        super(CustomModel, self).__init__()
        self.w = torch.nn.Parameter(data = torch.tensor([w0], dtype=torch.float32, requires_grad=True))
    
        #self.matrix = torch.tensor(data = [[self.w, -1.],[-self.w, -1.]], dtype=torch.float32, requires_grad=True    This computes \partial_matrix(COST) --&gt; BAD

        self.matrix_trainable = self.w*torch.tensor(data=[[0,1],[-1,0]], dtype=torch.float32,requires_grad=False)
        self.matrix = self.matrix_trainable - torch.eye(2)
    
    def forward(self, x):
        return self.matrix.matmul(x)

def loss(pred,y):
    return torch.mean((pred- y)**2)


my_model = CustomModel(w0=0.01)
optimizer = torch.optim.Adam(lr=0.01, params=my_model.parameters())

device = torch.device("cpu")
x = torch.ones(2).to(device)
y = torch.tensor(data=[2.,0.], dtype=torch.float32).to(device)


for k in range(10):

    optimizer.zero_grad()
    my_model.zero_grad()
    pred = my_model(x)
    cost = loss(pred,y)
    cost.backward()
    optimizer.step()
</code></pre>
<blockquote>
<p>RuntimeError: Trying to backward through the graph a second time (or directly access saved tensors after they have already been freed). Saved intermediate values of the graph are freed when you call .backward() or autograd.grad(). Specify retain_graph=True if you need to backward through the graph a second time or if you need to access saved tensors after calling backward.</p>
</blockquote>
<p><em>torch version 2.0.1</em></p>
<p><strong>Thanks a lot!!</strong></p>
<p>Matías</p>

username: ptrblck
cooked: <p>You are creating <code>self.matrix_trainable</code> in a differentiable way in <code>__init__</code>, which will cause the issue since the computation graph will stay alive.<br>
Create <code>matrix_trainable</code> in the <code>forward</code> for each new input and it should work:</p>
<pre><code class="lang-python">    def forward(self, x):
        matrix_trainable = self.w*torch.tensor(data=[[0,1],[-1,0]], dtype=torch.float32,requires_grad=False)
        matrix = matrix_trainable - torch.eye(2)
        return matrix.matmul(x)
</code></pre>

username: Matias_Bilkis
cooked: <p>thanks a lot!<br>
is this the most efficient solution available in pytorch ?</p>
Why is requires.grad not saved in model.state_dict()?

username: eturok
cooked: <p>Hello!</p>
<p>I found a strange quirk in the following code:</p>
<pre><code class="lang-py">import torchvision
model = torchvision.models.vgg19(pretrained=True)
</code></pre>
<p>By default all parameters in <code>model</code> have <code>requires.grad</code> set to <code>True</code>. Here we set <code>features.0.weight</code> and <code>features.0.bias</code> to <code>True</code> and all other parameters to <code>False</code>.</p>
<pre><code class="lang-py">learnable_params = ['features.0.weight', 'features.0.bias']
for name, param in model.named_parameters():
    param.requires_grad = name in learnable_params
    
    if int(name.split('.')[1]) &lt; 3 and name.startswith('features'):
        print(name, param.requires_grad)
</code></pre>
<p>Here are the print statements from the first 2 feature layers.</p>
<pre><code class="lang-auto">features.0.weight True
features.0.bias True
features.2.weight False
features.2.bias False
</code></pre>
<p>As we expected, <code>features.0.weight</code> and <code>features.0.bias</code> are <code>True</code> and all other parameters are <code>False</code>.</p>
<p>But then I investigated the <code>model.state_dict()</code>:</p>
<pre><code class="lang-py">for name, v in model.state_dict().items():
    if int(name.split('.')[1]) &lt; 3 and name.startswith('features'):
        print(name, v.requires_grad)
</code></pre>
<p>which outputted</p>
<pre><code class="lang-auto">features.0.weight False
features.0.bias False
features.2.weight False
features.2.bias False
</code></pre>
<p>Now, <code>features.0.weight</code> and <code>features.0.bias</code> are <code>False</code>.</p>
<p>Why?</p>
<p>When I updated <code>param.requires_grad</code> did this note save into the <code>model.state_dict()</code>? Why is this happening?</p>
<p>Any suggestions would be appreciated. Thanks!</p>

username: ptrblck
cooked: <p>Use <code>keep_vars=True</code> as described in the <a href="https://pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module.state_dict">docs</a> and the parameters won’t be detached.</p>

username: eturok
cooked: <p>Wow, thank you so much!</p>
<p>This perfectly fixed my problem.</p>
<p>The code</p>
<pre><code class="lang-py">for name, v in model.state_dict(keep_vars=True).items():
    if int(name.split('.')[1]) &lt; 3 and name.startswith('features'):
        print(name, v.requires_grad)
</code></pre>
<p>outputs</p>
<pre><code class="lang-auto">features.0.weight True
features.0.bias True
features.2.weight False
features.2.bias False
</code></pre>
<p>as expected.</p>
<p><a href="https://discuss.pytorch.org/u/ptrblck">ptrblck</a> you are the MVP! Thanks!</p>
When I want to get the input grad in mse_loss, what's the difference between mse_loss, backward and torch.ops.aten.mse_loss_backward?

username: yuer-cn
cooked: <p>I wanna get input grad in mse_loss with reduction=‘mean’. Regularly, the process is</p>
<pre><code class="lang-auto">import torch
x = torch.tensor(torch.tensor([1.,2.,3.,4.,5.]))
x.requires_grad_(True)
target = torch.tensor(torch.tensor(5.,4.,3.,2.,1.))
out = torch.nn.functional.mse_loss(x, target, reduction = 'mean')
out.backward(torch.tensor(float(2.)))
print(x.grad)
the outpu is tensor([-3.2, -1.6, 0.0, 1.6, 3.2])
besides, if I exec like this: out.backward(torch.tensor[2,3,4,5,6])), I will get a RuntimeError.
</code></pre>
<p>if I use mse_loss_backward to get x.grad, the process is</p>
<pre><code class="lang-auto">import torch
grad_out = torch.tensor(torch.tensor([2,3,4,5,6]))
x = torch.tensor(torch.tensor([1,2,3,4,5]))
target = torch.tensor(torch.tensor(5,4,3,2,1))
out = torch.ops.aten.mse_loss_backward(grad_output, x, target, 1)
print(out)
the out is tensor([-3.2, -2.4, 0.0, 4.0, 9.6])
</code></pre>
<p>I want to konw what’s the difference between the two methods, and Why can’t they have the same input?</p>

username: ptrblck
cooked: <p>Your code has a lot of issues and not only raises warnings in the unnecessary re-wrapping of tensors via <code>torch.tensor(torch.tensor())</code>, but also fails:</p>
<pre><code class="lang-python">out.backward(torch.tensor[2,3,4,5,6])
# TypeError: 'builtin_function_or_method' object is not subscriptable
...
NameError: name 'grad_output' is not defined
...
    target = torch.tensor(torch.tensor(5,4,3,2,1))

TypeError: tensor() takes 1 positional argument but 5 were given
</code></pre>
<p>After fixing these issues, the code works as expected. Your first example fails since <code>out</code> is reduced and thus a single scalar. The gradient has to match its shape as indicated by the error message.</p>
TypeError: conv2d() received an invalid combination of arguments for ResNet fine-tuning

username: Ajit
cooked: <p>Hi,<br>
I have added classification and regression head to resnet50 pretrained model. But while training after the fine-tuning getting below error message:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/1/314758b22d0544e88675fdd871de2d99dced7969.png" data-download-href="https://discuss.pytorch.org/uploads/default/314758b22d0544e88675fdd871de2d99dced7969" title="Screenshot from 2023-10-29 21-52-31"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/1/314758b22d0544e88675fdd871de2d99dced7969_2_625x500.png" alt="Screenshot from 2023-10-29 21-52-31" data-base62-sha1="71WfOOTtAse2vL4Zum0GmFUckaJ" width="625" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/1/314758b22d0544e88675fdd871de2d99dced7969_2_625x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/3/1/314758b22d0544e88675fdd871de2d99dced7969_2_937x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/3/1/314758b22d0544e88675fdd871de2d99dced7969_2_1250x1000.png 2x" data-dominant-color="222225"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2023-10-29 21-52-31</span><span class="informations">1432×1144 105 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>It seems the input is a <code>list</code> while a tensor is expected.<br>
Post a minimal and executable code snippet instead of screenshots for better debuggability.</p>

username: Ajit
cooked: <p>Thanks for the input. I converted input list to tensor. Its working now <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=12" title=":smiley:" class="emoji" alt=":smiley:" loading="lazy" width="20" height="20"></p>