akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Are torch.transform done in parralel?

Agar: <p>Hello,</p>
<p>First off, sorry if my question is probably more related to Python itself rather than PyTorch.<br>
I am unclear if torch.transform pipelines created with torch.compose are done in parrallel over the batch.</p>
<p>If I have a batch of 2 samples (fetched with a DataLoader, num_worker = 0).<br>
And a torch.compose having two transformations A &amp; B.</p>
<p>What really happens ?</p>
<p>(A&amp;B over sample 1) then (A&amp;B over sample 2).<br>
or<br>
(A over sample 1) then (A over sample 2 &amp; B over sample 1) then ( B over sample 1)</p>
<p>I have the same question regarding a forward pass in any sequential model.<br>
Are samples passed one by one over the entire network, or are they sent in a queue layer by layer ?</p>

ptrblck: <aside class="quote no-group" data-username="Agar" data-post="1" data-topic="197859">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/agar/48/66839_2.png" class="avatar"> Agar:</div>
<blockquote>
<p>(A&amp;B over sample 1) then (A&amp;B over sample 2).</p>
</blockquote>
</aside>
<p>Each sample will be loaded and processed in the <code>Dataset.__getitem__</code>.<br>
The common approach is to load and transform a single sample so the transformations would be performed sequentially in the main processes given you are using <code>num_workers=0</code>.<br>
<code>torchvision.transforms</code> also accept tensors and you could use a <a href="https://discuss.pytorch.org/t/how-to-use-batchsampler-with-getitem-dataset/78788/4"><code>BatchSampler</code></a> if you want to load and process multiple tensors at once.<br>
If you use <code>num_workers&gt;0</code> each process will process and create its own batch.</p>
<aside class="quote no-group" data-username="Agar" data-post="1" data-topic="197859">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/agar/48/66839_2.png" class="avatar"> Agar:</div>
<blockquote>
<p>Are samples passed one by one over the entire network, or are they sent in a queue layer by layer ?</p>
</blockquote>
</aside>
<p>Samples are passed as a batch to the model and all layers accept batched inputs thus processing all samples at once.</p>
BatchNorm2d making training extremely slow

shreyasr: <p>I’m implementing SuperPoint network and the backbone uses a VGG-style architecture. I am unable to obtain the same results in my implementation as the paper. There were some unofficial implementations [<a href="https://github.com/rpautrat/SuperPoint/tree/master/superpoint" rel="noopener nofollow ugc">1</a>] [<a href="https://github.com/shaofengzeng/SuperPoint-Pytorch/" rel="noopener nofollow ugc">2</a>] that have used batch normalization int he backbone. I tried implementing the same and started training on MS-COCO dataset. The training speed of model with BN is around 0.3 it/s whereas that without BN is over 10 it/s. I am using Pytorch Lightning to train with two A6000 Ada GPU’s . I’m not sure where I am going wrong or why BatchNorm2d is causing this delay.</p>
<p>Can anyone explain conceptually why the delay is caused?</p>

shreyasr: <p>Original: <a href="https://github.com/magicleap/SuperPointPretrainedNetwork/tree/master" class="inline-onebox" rel="noopener nofollow ugc">GitHub - magicleap/SuperPointPretrainedNetwork: PyTorch pre-trained model for real-time interest point detection, description, and sparse tracking (https://arxiv.org/abs/1712.07629)</a></p>
SpeechBrain 1.0 - A free and open-source AI toolkit for all things speech based on PyTorch

flexthink: <p>Bring your conversational AI ideas to life!</p>
<p><a href="https://speechbrain.github.io" class="onebox" target="_blank" rel="noopener nofollow ugc">https://speechbrain.github.io</a></p>
<p>Check out SpeechBrain 1.0:  an open-source AI toolkit for all things speech: speech recognition, speech synthesis, speech enhancement, emotion recognition and more! This milestone release comes with state-of-the art models for you to customize and use in your projects.</p>
<p>The toolkit features 200+ recipes and 100+ pretrained models on HuggingFace:</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://huggingface.co/speechbrain">
  <header class="source">

      <a href="https://huggingface.co/speechbrain" target="_blank" rel="noopener nofollow ugc">huggingface.co</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/372;"><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/speechbrain.png" class="thumbnail" width="690" height="372"></div>

<h3><a href="https://huggingface.co/speechbrain" target="_blank" rel="noopener nofollow ugc">speechbrain (SpeechBrain)</a></h3>

  <p>Deep Learning, Speech Technologies</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
What is the right approach to data augmentation when the dataset contains images of different sizes?

pravin382: <p>I am trying to carry out training for the task of segmentation. I have dataset containing images of size ranging from 192X183 to 3456X4000. And I am planing to train with input size (384, 512) as most of the images are of size (768, 1024).</p>
<p>My current strategy is to apply padding for smaller images so that they have same size to that of input size(384, 512). For larger image, I intend to first resize them to size (768, 1024) and then apply random cropping or maybe FiveCrop operation. I prefer to apply FiveCrop because it helps to increase data size.<br>
Questions:</p>
<ol>
<li>How would one combine these two operations(pad if small image and resize+fivecrop if large image) in dataloader? I think, this will cause batch size to vary.</li>
<li>Is it preferred to carry both operation before the training starts? I mean do I have to create new dataset with both operation first and then perform training?</li>
</ol>
<p>For 1, any tutorial or pseudo code would be very much appreciated.</p>
Applying my own filter using torchaudio on 1D signal

dthanoon: <p>I have a 1 D signal of size (151pts) sample at every 4ms =&gt; 250Hz. I want to use torch audio to apply my own filter . lets say a filter that is going from 0 to 1 (from 10Hz to 20Hz) then 1 from 20Hz to 50Hz, then 1 to 0 from 50Hz to 60Hz and then 0 for &gt;60Hz.<br>
How can I do this signal filtering by taking my signal, moving it to frequency domain, multiply the filter and then go back to time domain</p>
How to disable auto log when loop in Dataloader

Wanrat_Limprapaipong: <p>Hi, I’m new to Pytorch. When I use for loop in Dataloder, it always auto log lists like below.</p>
<pre><code class="lang-auto">for image, mask in trainloader:  
  break
  

print(f"batch image shape : {image.shape}")
print(f"batch in mask shape : {mask.shape}")
</code></pre>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/3/131deda1f249d632917190d197e8906f606a84b7.png" alt="image" data-base62-sha1="2J7bXB4cWx0672PCSCx9kVavuKj" width="572" height="313"></p>
<p>Is there a way to disable that log lists?</p>

ptrblck: <p>PyTorch does not show any logging info from the <code>Dataset</code> so I guess your (custom) dataset might print it and you should check the implementation.</p>
VAE not getting trained

sank: <p>Hi,<br>
I am trying to generate samples through VAE, but after training, I can see a blob at the center of the image. I tried mse loss, bce loss but still, the problem persists.<br>
The loss is shooting up and becomes nan values after some epoch.<br>
I tried with mse loss</p>
<pre><code class="lang-auto">recons_loss =F.mse_loss(recons, input,reduction='mean')
kld_loss = torch.mean(-0.5 * torch.sum(1 + log_var - mu ** 2 - log_var.exp(), dim = 1), dim = 0)
</code></pre>
<p>and also with bce loss</p>
<pre><code class="lang-auto">recons_loss = F.binary_cross_entropy_with_logits(recons, input, size_average=False)
        kld_loss = -0.5 * torch.sum(1 + log_var - mu.pow(2) - log_var.exp()).mean()
</code></pre>
<p>Please let me know what is the issue here?</p>
<p>kaggle notebook link: <a href="https://www.kaggle.com/sanketgandhi/pokemon" class="inline-onebox" rel="noopener nofollow ugc">Pokemon | Kaggle</a></p>

sluharex: <p>Hello!<br>
Looks like you have exploding gradient. I got same problem with my project - NaN instead of values. This is my loss</p>
<pre><code class="lang-auto">reconstruction_loss = self.loss(output, y)
kl_divergence = -0.5 * torch.sum(1 + logvar - mu.pow(2) - logvar.exp())
loss = reconstruction_loss + kl_divergence
</code></pre>
<p>Solution was to limit sigma (dispertion):</p>
<pre><code class="lang-auto">self.max_logvar = nn.Parameter(torch.ones(latent_size) * 0.5, requires_grad=True)
self.min_logvar = nn.Parameter(torch.ones(latent_size) * -0.5, requires_grad=True)

def reparameterize(self, mu, logvar):
        logvar = torch.clamp(logvar, min=self.min_logvar, max=self.max_logvar) ###
        std = torch.exp(0.5*logvar)
        eps = torch.randn_like(std)
        z = mu + eps*std
        return z
</code></pre>
About time series models and system identification

Mh_14: <p>Greetings to everyone!</p>
<p>I am a young data science researcher working on control engineering topics with DL and NN. I am also very new to DL and NN architectures. I am trying to predict the input and output sensor values of a dynamic and complex system, that is system identification. I want to get the predicted value of the temperature sensor according to the future fuel inputs of the system. There are 3 features with 300k lines of data including Train, Test and Validation datasets. Each line contains 1 minute sensor value timeseries. The minute sensor value is read. As a first step, I want to train a model like LSTM / GRU, I use the Darts library using the pytorch forecasting infrastructure. I even have some past covariates / time_varying_unknown_reals variables that I only know the past, I want to use the TFT model in the next stage. If I need to briefly describe the features of the PC I use, I have 1 GPU, tesla T4 16GB, 12 GB Ram and 5 CPU.  In my first attempts, the training took too long and the results were not encouraging. I can share loss graphs if you want. I think exactly the model does not fit properly, do you think I should work on a small data and overfit it or should I use all the data and wait for long periods of time? Does Optune make the model fit properly?</p>
<p>Also I can only consume 1 gb of the 16 GB GPU during these trainings, do you think this is correct as gpu utalisation? Is there so little consumption because there is only 3 Column efficiency. GPU duty cycle reaches 100/100s.</p>
Where is the C++ definition of Tensor located

PersonalFinance004: <p>I’m trying to find the definition of <code>Tensor</code> as used in the C++ parts of PyTorch, for example in <code>pytorch/aten/src/ATen/native/ReduceOps.cpp</code> in line 746 we have the function</p>
<pre><code class="lang-auto">Tensor prod(const Tensor&amp; self, int64_t dim, bool keepdim, c10::optional&lt;ScalarType&gt; opt_dtype) {
  ScalarType dtype = get_dtype_from_self(self, opt_dtype, true);
  Tensor result = create_reduction_result(self, dim, keepdim, dtype);
  native::prod_out_impl(result, self, dim, keepdim, dtype);
  return result;
}
</code></pre>
<p>but for some reason the IDE I’m using (CLion) can’t find the declaration of the type <code>Tensor</code>.</p>
<p>Could you kindly direct me to the declaration.</p>

tom: <p>This is declared in a generated header <code>include/ATen/core/TensorBody.h</code> which you find in LibTorch / PyTorch installations. The template is in <a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/templates/TensorBody.h#L734" rel="noopener nofollow ugc">aten/src/ATen/templates/TensorBody.h</a>.</p>

PersonalFinance004: <p>I asked this because I wanted to understand the following: I have a function</p>
<pre><code class="lang-auto">- func: function(Tensor self) -&gt; Tensor
  variants: method, function
  dispatch:
    CPU: function_cpu
    CUDA: function_cuda
</code></pre>
<p>whose implementation I want to change.</p>
<p>The current version’s return tensor is declared this way:</p>
<pre><code class="lang-auto">Tensor result;
result = at::empty({}, self.options().dtype(dtype));
</code></pre>
<p>I computed the result I want to return with <code>std::vectors</code> (probably not optimal, feel free to suggest something better). Now I’m wondering how I can pass this vector to the <code>result</code> tensor. I think I have to use following in some way, and was trying to find its implementation by asking about the location of the <code>Tensor</code> class:</p>
<pre><code class="lang-auto">self.data_ptr&lt;scalar_t&gt;()
</code></pre>
<p>I can’t seem to infer this from the information you gave me above. Can you explain to me to how I can properly give my data to the <code>return</code> tensor? Is there maybe an instructive function somewhere I could look at.</p>

shadowshadow: <p>Hello, I can’t find Tensorbody.h in the project. Can you tell me how to generate theses files?</p>
Computing to a sub-tensor portion of the output tensor?

Jong_Seok_Park: <p>Hello!</p>
<p>Is there a way to update the computation of a certain sub-tensor portion of an output tensor without re-computing the whole output tensor?</p>
<p>To be specific, I have a series of similar input matrices with very small patches of different values scattered around. I am trying to perform matrix multiplication on these matrices, but to reduce computation and memory usage, I would like to compute using only the changed portion of the input and update the output matrix in-place, after the initial computation.</p>
<p>I can locate the changes, and use slices of the input and weight matrix with the linear op (torch.nn.functional.linear) to compute only the patch that needs to be updated on the output. However, I still need to copy and scatter the output patch to the original output tensor to actually update it. I know that using BLAS libraries I can achieve <strong>in-place computation and update</strong> by setting the variable <em>m</em> to the <em>number of rows of the submatrix</em>, <em>ldc</em> to the <em>number of rows of the original matrix</em>, and <em>C</em> to the start of the submatrix, during the <a href="https://spec.oneapi.io/versions/latest/elements/oneMKL/source/domains/blas/gemm.html" rel="noopener nofollow ugc">GEMM function call</a>. (Assuming column-major matrices)</p>
<p>Is there a way to similarly specify the <em>storage offset</em>, <em>stride</em>, and <em>data_ptr</em> of the output tensors for the given PyTorch op (matrix multiplication, convolution, add, etc.)? Or will they always create a new contiguous tensor for output?</p>
<p>Thank you in advance for your help!</p>
Suggestions for backpropagating DSP code

LOGUNIVPM: <p>Hi, not sure this is the right category to post, but it is surely related to how autograd works. I need to better understand how requires_grad is set to False or True by the torch engine and why the loss.backward() and optimizer.step() sometimes complains that during backprop some tensor have grad False and sometimes not.</p>
<p>More specifically:<br>
I am developing a class for sound synthesis that takes a tensor of parameters and translate these into a time-domain (i.e. the sound). The signal must be generated by iterating a for loop (we use a physical model, so there is no way to avoid the iterations). Everything is done using torch operations, from the input tensor to the output, and this class indeed manages to get the output from a NN and synthesize sound. By inspecting the model parameters, however, they are all tensors with requires_grad=False. Indeed list(model.parameters())[0].grad is False, and the network weights stay the same at each iteration, but torch can compute the graph. Why?<br>
What are smart ways to build custom code that is differentiable?</p>
<p>BTW: in the past using keras we were able to conduct entire simulations of room impulse responses, convolutions and FFT without problems. But with keras you don’t know what happens under the hood.</p>

LOGUNIVPM: <p>Ping! Maybe the question is too specific?</p>

JuanFMontesinos: <p>Sounds like a bug with the code.<br>
Could you paste it?</p>

triakshunn: <p>I think we can help better with code to see where we can troubleshoot?</p>
LSTM quant to get the scale and zp of hidden input

kelsey: <p>Hi, if I quant the LSTM uing torch.ao.nn.quantized.LSTM and torch.ao.nn.quantizable.LSTM like <a href="https://github.com/pytorch/pytorch/blob/main/test/quantization/fx/test_quantize_fx.py#L4886" class="inline-onebox" rel="noopener nofollow ugc">pytorch/test/quantization/fx/test_quantize_fx.py at main · pytorch/pytorch · GitHub</a>, how to get the scale and zp of LSTM hidden input h0 and c0?  I want to quant the int8 model using long sequance data and inference it frame by frame, so I need to know the scale and zp of the hidden input of LSTM for every frame when inferencing.</p>
How to install Torch version for Cuda 10.2, GTX 980 ti?

Jonas_Adel: <p>Hello,</p>
<p>I’m using a GTX 980 TI GPU and I’m trying to make it work with Cuda toolkit 10.2, but I can’t install torch for the Cuda toolkit 10.2.</p>
<p>I want to use Torch with GTX 980 TI GPU for training Neural networks. currently, I can’t use Cuda 10.2 with torch so the Network only uses CPU instead of GPU…</p>
<p>Anyone have a clue what I can do?</p>

ptrblck: <p>Your GTX 980 Ti should be a Maxwell GPU with a compute capability of 5.2, which is still supported in our binary builds, so you should be able to install the latest PyTorch release with CUDA 11.8 or 12.1.</p>
Error when applying Opacus to TVAE

ledwher: <p>Hello! I am trying to apply DP to TVAE using Opacus.</p>
<p>Here is a code snippet of the training section:<br>
self.decoder, optimizer_decoder, loader = privacy_engine.make_private_with_epsilon(<br>
module=self.decoder,<br>
optimizer=optimizer_decoder,<br>
data_loader=loader,<br>
epochs=self.epochs,<br>
target_epsilon=target_epsilon,<br>
target_delta=target_delta,<br>
max_grad_norm=max_grad_norm,<br>
)</p>
<pre><code>    for i in range(self.epochs):
        for id_, data in enumerate(loader):
            
            optimizer_encoder.zero_grad()
            optimizer_decoder.zero_grad()
            real = data[0].to(self._device)
            mu, std, logvar = encoder(real)
            eps = torch.randn_like(std)
            emb = eps * std + mu
            rec, sigmas = self.decoder(emb)
            loss_1, loss_2 = _loss_function(
                rec, real, sigmas, mu, logvar,
                self.transformer.output_info_list, self.loss_factor
            )
            loss = loss_1 + loss_2
            print(loss)
            loss.backward()

            optimizer_encoder.step()
            optimizer_decoder.step()
            
            self.decoder.sigma.data.clamp_(0.01, 1.0)
</code></pre>
<p>I get the following error after loss.backward():<br>
TypeError: only integer tensors of a single element can be converted to an index</p>
<p>When I fake a training step with only the decoder, I get the loss but with the following error:<br>
decoder(noise)[0].sum()<br>
UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior.<br>
warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes "</p>
<p>Any help would be very appreciated, thank you!</p>
How to save/load a model with torch.compile

LuoXin-s: <p>I have a model compiled with <code>torch.compile</code>, and I found <code>torch.compile</code> will add a prefix ‘_orig_mod.’ to <code>state_dict()</code> of the model. However, I expect loading these weights to a non compiled model, so I have to remove this prefix manually.</p>
<p>My question is why adding this prefix? What is best practice playing with <code>torch.compile</code> when saving/loading models.</p>

marksaroufim: <p>For now save or load the uncompiled model - they share weights</p>
<p>longer answer here <a href="https://github.com/pytorch/pytorch/issues/101107#issuecomment-1542688089" class="inline-onebox" rel="noopener nofollow ugc">Make compiled models serializable · Issue #101107 · pytorch/pytorch · GitHub</a></p>

ecstayalive: <p>In fact, there is a general format that can solve this problem, which is to rewrite the <code>state_dict</code> and <code>load_state_dict</code> functions every time after writing network. Because the functions we write all inherit from <code>nn.Module</code>, custom functions can also be solved in this way, for example:</p>
<pre data-code-wrap="python"><code class="lang-python">from collections import OrderedDict
import torch.nn as nn


class CustomNet(nn.Module):
    def __init__(self):
        self.kernel: nn.Module = ...
    # override state_dict() method
    def state_dict(self, *args, destination=None, prefix="", keep_vars=False):
        if destination is None:
            destination = OrderedDict()
        prefix = ""  # remove prefix
        destination.update([('kernel', self.kernel.state_dict(destination=destination, prefix=prefix, keep_vars=keep_vars)])
        return destination
    # override load_state_dict()
    def load_state_dict(self, state_dict, ...):
        self.kernel.load_state_dict(state_dict)
</code></pre>
<p>The above code is just an example of the method and cannot be run. You should adjust the structure accordingly according to your own code.</p>
Quantized Resnet has no parameters()

rxd: <p>Hi there,<br>
i’m currently trying to perform a quantization-aware fine-tuning of a Resnet50 model on another dataset and experiencing some issues related to the initialization of the training optimizer.</p>
<pre><code class="lang-auto">import torch
from torchvision.models.quantization.resnet import resnet50, ResNet50_QuantizedWeights

model = resnet50(weights=ResNet50_QuantizedWeights.IMAGENET1K_FBGEMM_V1, quantize=True)
model.train()
model.qconfig = torch.quantization.get_default_qat_qconfig("qnnpack")
model_qat = torch.quantization.prepare_qat(model, inplace=False)
optimizer = torch.optim.SGD(model_qat.parameters(), lr=1e-4)
</code></pre>
<p>Running this script leads to the following error:  <code>ValueError: optimizer got an empty parameter list</code>.</p>
<p>I must admit that I am pretty inexperienced in quantization in Pytorch so I may be missing a crucial step here. Thanks!</p>

supriyar: <p><code>quantize = True</code> returns a quantized model with 8 bit weights. Quantized models only support inference and run on CPUs. So you cannot do QAT on that, since we don’t support training integer weights.</p>
<p>You can follow the tutorial <a href="https://pytorch.org/tutorials/advanced/static_quantization_tutorial.html#quantization-aware-training" rel="noopener nofollow ugc">here</a> for details on how to do QAT.</p>
<p>Note that for resnet you can also try the PT2 export based QAT, which doesn’t require any manual model changes. Please refer to the tutorials<br>
1 <a href="https://pytorch.org/tutorials/prototype/pt2e_quant_qat.html" class="inline-onebox" rel="noopener nofollow ugc">(prototype) PyTorch 2 Export Quantization-Aware Training (QAT) — PyTorch Tutorials 2.2.1+cu121 documentation</a><br>
2. <a href="https://pytorch.org/tutorials/prototype/pt2e_quant_x86_inductor.html#quantization-aware-training" class="inline-onebox" rel="noopener nofollow ugc">PyTorch 2 Export Quantization with X86 Backend through Inductor — PyTorch Tutorials 2.2.1+cu121 documentation</a></p>

rxd: <p>Thank you very much for your response. Does it means that once a model is trained in a quantized-aware manner, it can no longer be modified (say, in a continuous training setting)?</p>
Cross entropy loss multi target

april: <p>hello, I want to use one-hot encoder to do cross entropy loss</p>
<p>for example</p>
<p>input:<br>
[[0.1, 0.2, 0.8, 0, 0],<br>
[0,0, 2, 0,0,1]]</p>
<p>target is<br>
[[1,0,1,0,0]]<br>
[[1,1,1,0,0]]</p>
<p>I saw the discussion to do argmax of label to return index,<br>
but I have multiple 1s in one row, argmax will only return 1,</p>
<p>how do I solve this problem?</p>

ptrblck: <p>You cannot use <code>nn.CrossEntropyLoss</code> for a mulit-label classification and would need to use <code>nn.BCEWithLogtisLoss</code>.<br>
To do so you can keep the shape of the target tensor and transform it to a <code>FloatTensor</code> via <code>target = target.float()</code>.</p>

dimitriyrudenk: <p>Beware of copying it, proper spelling is torch.nn.BCEWithLogitsLoss</p>
Forward/Backward partially on cpu and partially on gpu?

metro: <p>Is it possible to forward a model on gpu but calculate the loss of the last layer on cpu?<br>
If so, how does pytorch know during backprop where the tensor is? Or is it expecting all tensors to lie consistently on one device?</p>
<p>If it is possible, is there a documentation article or other resource which explains this process?</p>
<p>Background: I calculate a loss with torch.pca_lowrank which is significantly faster on cpu in my usecase (more than 8x).</p>
<p>Cheers!</p>

ptrblck: <p>Yes, you can move around activations, as Autograd will track the <code>to()/cpu()/cuda()</code> operations.<br>
E.g.:</p>
<pre><code class="lang-python"># setup
x_cuda = torch.randn(1, 1).to('cuda')
lin_cuda = nn.Linear(1, 1).to('cuda')
lin_cpu = nn.Linear(1, 1)

# workload on the GPU
out = lin_cuda(x_cuda)
# transfer to CPU
out_cpu = out.to('cpu')
# workload on CPU
out_cpu = lin_cpu(out_cpu)

# loss calculation on the CPU
loss = (out_cpu ** 2).mean()

# backward
loss.backward()

# check grads
for name, param in lin_cuda.named_parameters():
    print(name, param.grad)
&gt; weight tensor([[-0.2126]], device='cuda:0')
  bias tensor([-0.4568], device='cuda:0')

for name, param in lin_cpu.named_parameters():
    print(name, param.grad)
&gt; weight tensor([[-0.7824]])
  bias tensor([0.8254])
</code></pre>
<p>will work.</p>

sadra-barikbin: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> , Does it make any difference to bring the loss factors to the same device as the model, in terms of performance and memory?</p>
<p>For example in a RL setup, The logprob in experience buffer which is normally stored on cpu, participates in loss computation while the model output logprob is on gpu. Is it better to move the former to gpu or the latter to cpu?</p>

ptrblck: <p>Assuming you want to keep these two outputs on different devices (and won’t be able to move everything to the GPU), you would need to compare the data transfer time against the loss calculation.<br>
If the loss calculation is “expensive” you might want to move the <code>CPUTensor</code> to the GPU before computing the loss. You can also transfer the <code>CPUTensor</code> asynchronously and could hide the transfer time if other operations can be executed before the loss calculation is needed.</p>
<p>However, if the loss computation is really cheap and if the <code>CUDATensor</code> is small you could transfer it to the CPU instead, but should definitely profile it first.</p>
Model.to(device) method doesn't work on models instantiate as list

mlkonopelski: <p>Let me know please if this is expected behaviour.</p>
<pre><code class="lang-auto">    class ModelB(nn.Module):
        def __init__(self) -&gt; None:
            super().__init__()
            self.layer = nn.Linear(1, 1)

        def forward(self, x):
            self.layer(x)
            return x

    class ModelA(nn.Module):
        def __init__(self) -&gt; None:
            super().__init__()
            self.model = ModelB()
            self.models = [ModelB()]
        def forward(self, x):
            x = self.model(x)
            for model in self.models:
                x = model(x)
            return x

        
    X = torch.rand(10, 1).to('mps')
    model_a = ModelA().to('mps')
    out = model_a(X)
    print(out)
</code></pre>
<p>the error is on this line: <code>x = model(x)</code> while being almost the same as correct: <code>x = self.model(x)</code></p>
<blockquote>
<p>RuntimeError: Placeholder storage has not been allocated on MPS device!</p>
</blockquote>
<p>when I try: <code>self.models = nn.ModuleList([ModelB()])</code> it again works correctly.</p>

ptrblck: <p>This is expected and you would need to use <a href="https://pytorch.org/docs/stable/generated/torch.nn.ModuleList.html#torch.nn.ModuleList"><code>nn.ModuleList</code></a>. The same limitations apply to parameters, too (where you would need to use <code>nn.ParameterList</code> instead).</p>
Local modifications of the backpropagation in Pytorch

zkob: <p>Hi,</p>
<p>On pytorch, I am looking for a flexible way to modify the computation of the backpropagation in a neural network with multiple layers.</p>
<p>More precisely, I want to create blocks of layers indexed by <code>k in {1,...,K}</code> such that :</p>
<ol>
<li>the output is computed by the formula <code>x_{k+1}  = f_k(g_k(x_k, theta_k))</code>, where <code>g_k</code> is the composition of multiple layers with learnable parameters <code>theta_k</code>, and <code>f_k</code> is some function that does not have learnable parameters.</li>
<li>during the backpropagation, the gradient with respect to <code>theta_k</code> is calculated using the above formula.</li>
<li>still during the backpropagation, the gradient with respect to <code>x_k</code> is computed as if the operation made was <code>x_{k+1}  = h_k(g_k(x_k, theta_k))</code>, where <code>h_k</code> is a different function from <code>f_k</code></li>
</ol>
<p>I tried using hooks but I could not figure out a simple way to do this. I asked ChatGPT4 multiple times but everytime the answer does not solve my problem and it takes a long time to check. Any help would be much appreciated.</p>

albanD: <p>Hey!</p>
<p>This is going to be a couple possible approaches here:</p>
<ul>
<li>You can override the whole block to do whatever you want with a custom Function (<a href="https://pytorch.org/docs/stable/notes/extending.html#extending-torch-autograd" class="inline-onebox">Extending PyTorch — PyTorch 2.2 documentation</a>). In this case, you might want to do “reentrant backward” where your custom backward call backward on other pieces. You will need to properly handle enable_grad to get the behavior you want (some related example is <a href="https://github.com/pytorch/pytorch/blob/d3839b624b5f6451a13bd9b5ecbbce4c2a9b1db6/torch/utils/checkpoint.py#L320" class="inline-onebox">pytorch/torch/utils/checkpoint.py at d3839b624b5f6451a13bd9b5ecbbce4c2a9b1db6 · pytorch/pytorch · GitHub</a>).</li>
<li>You can override only one smaller function (f_k) to have a custom backward behavior (should be simpler than the above), but the problem is that since you want gradients for each input to be computed with a different backward for f_k, you will have to do two calls to backward: one wrt x_k with the original f_k gradient and one wrt theta_k with the modified gradient.</li>
</ul>

zkob: <p>Thank you for the reply. I alerady heard about the first approach you are proposing, and I wanted to avoid it, but I think I will go for this one. I do not think the second one may work in my situation.</p>
Multiprocessing CPU under-used, only 2 cores are used

CXY1: <p>I’m having this weird issue where only 2,3 cpu cores are use by torch.multiprocessing. It only seems to happen on our new machine with i9-13900K CPU.<br>
It works fine on our older Xeon CPUs (100% on 50 cores as expected, in the data preprocessing stage).<br>
The code is the same.<br>
I’m using pytorch 2.0.1, both machine running Ubuntu 20.04<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/9/f996068a1487ee11bd90daca36ffe61dead9d2f8.png" data-download-href="https://discuss.pytorch.org/uploads/default/f996068a1487ee11bd90daca36ffe61dead9d2f8" title="Screenshot 2023-07-21 at 4.52.11 PM copy"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/9/f996068a1487ee11bd90daca36ffe61dead9d2f8_2_517x282.png" alt="Screenshot 2023-07-21 at 4.52.11 PM copy" data-base62-sha1="zBWjJSRmx0myFaSYxfMX04odFKo" width="517" height="282" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/9/f996068a1487ee11bd90daca36ffe61dead9d2f8_2_517x282.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/9/f996068a1487ee11bd90daca36ffe61dead9d2f8_2_775x423.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/f/9/f996068a1487ee11bd90daca36ffe61dead9d2f8_2_1034x564.png 2x" data-dominant-color="2A312E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2023-07-21 at 4.52.11 PM copy</span><span class="informations">3954×2160 1.43 MB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
How to implement custom batch sizes

ConnorW: <p>I am trying to do a Listwise LTR NN, where I am predicting a rank based on the QID.</p>
<p>I currently have this custom datset but now need to make sure each batch is only relevant to the QID and doesn’t have other QID in same batch</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/1/515aa2175ddf12a50af58592877c5a4a34eb71e1.png" alt="image" data-base62-sha1="bBGR3BgpqVp9XUq6IT7jSuBFKEN" width="590" height="342"></p>
<p>Any pointers would be great please</p>
GPU memory problem

amy2: <p>I am using pytorch to train my model on nvidia rtx 3050 for training everything is fine but when I tried to run the inference it stopped like this<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/8/d8b147e21c1f1d2160f8e0270b5cd33a466acc89.png" data-download-href="https://discuss.pytorch.org/uploads/default/d8b147e21c1f1d2160f8e0270b5cd33a466acc89" title="error"><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/8/d8b147e21c1f1d2160f8e0270b5cd33a466acc89.png" alt="error" data-base62-sha1="uUWXgJwezC6cfYt55I0SpYbcvsR" width="690" height="254" data-dominant-color="0D0D0D"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">error</span><span class="informations">1610×594 5.01 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
I checked the code and there is no errors ,<br>
and before it was running fine<br>
any suggestions !</p>

ptrblck: <p>What kind of memory problem are you seeing?<br>
Your script seems to hang, so I’m unsure how it’s related to the GPU and its memory.</p>

amy2: <p>when I used 112<em>112 sample size for training and inference everything worked fine and when I doubeled 224</em>224 at first it work fine but after the script start to hang like that when I tried again with 112*112 it worked .<br>
I assumed  that was caused by the memory<br>
.If its not what could be the problem<br>
and if I want to check the memory used or  free it all what should I do</p>
How to convert SparseVectors inside tabular data to a Tensor

ConnorW: <p>I have have a tabular data set that I am wanting to use for my features. It consists of numerical values and categorical variables that I have one hot encoded.</p>
<p>I want to but these features into a tensor but when i convert to a numpy array then try to place into a tensor it says an error due to the sparse vectors.</p>
<p>Array data:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/7/2/722cfc8b2fb8a65937b8fea29f56116ca59660c4.png" alt="image" data-base62-sha1="gi2OvWdiF5hA9IBFGp1sfZLn15O" width="530" height="62"></p>

ConnorW: <p>Is there another way to convert categorical variables instead of one hot encoding? I have seen people mention about nn.embeddings but not sure how that would work if my case if I am feeding in features with some numerical and some categorical</p>
How does gradient calculation in quantization-aware training? Straight through estimatior?

lisyuan: <p>Hi all,</p>
<p>I am confusing how the gradient calculation in QAT.<br>
In my understanding, for example, the QAT of 8-bit process is mainly like<br>
<code>weight fp32 -&gt; weight fake-quant 8-bit  -&gt; activation -&gt; activation fake-quant 8-bit</code>.<br>
Then choose the observers to observe the min/max value to calculate the scales &amp; zero points of weights/activation output.</p>
<p>So, is the fake-quant of weights/activation calculated in backward? if so, how does it caulcated?<br>
The most common methods I surveyed is about Straight-Through-Estimator (STE), is that right?</p>
<p>But there are some answers show that it doesn’t participate in backward calculation,<br>
<a href="https://discuss.pytorch.org/t/can-we-calculate-gradients-for-quantized-models/133575/4" class="inline-onebox">Can we calculate gradients for quantized models? - #4 by HDCharles</a>?<br>
<a href="https://discuss.pytorch.org/t/register-backward-hook-in-quantized-model/164820/2" class="inline-onebox">Register_backward_hook in quantized model - #2 by HDCharles</a>?</p>
<p>Please correct me if I get a wrong idea. I want to implement a STE method in QAT. Therefore, I want to know whether pytorch do it automatically or not. Any suggestion is welcome. Thanks in advance!</p>