akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to enforce the on the same batch dataloader that has multiple datasets will return the items from the same dataset

David_Sriker1: <p>I have the following MUXDataset which holds several instnaces of dataset objects:</p>
<pre><code class="lang-auto">class MUXDataset(Dataset):
    """
        Defines a dataset class that provides a way to read scenes and also visualization tools
    """

    def __init__(self, mux_dataset_params: MultiplexDatasetParams) -&gt; None:
        self._params = mux_dataset_params
        self.sampling_vec = (
            mux_dataset_params.sampling if isinstance(mux_dataset_params.sampling, list) else self.init_sampling()
        )
        assert len(self.sampling_vec) == len(self._params.datasets), (
            'The datasets sampling weights are not equal to the amount of datasets'
        )

        if self._params.debug_flag:
            self.mux_debug_visualization()
            self.mux_save_datasets_info()
        return

    def init_sampling(self) -&gt; List[float]:
        if self._params.sampling == 'uniform':
            num_datasets = len(self._params.datasets)
            sampling = [1 / num_datasets for _ in self._params.datasets]
        elif self._params.sampling == 'proportional':
            size_vec = [len(_d) for _d in self._params.datasets]
            sampling = [1 - _s / sum(size_vec) for _s in size_vec]
            sampling = [_s / sum(sampling) for _s in sampling]
        else:
            raise ValueError(f'{self._params.sampling} is not supported')
        return sampling

    def __len__(self) -&gt; int:
        return len(self._params.datasets)

    def __getitem__(self, idx) -&gt; DatasetOutput:
        # endless iterator that will sample the datasets based on the sampling vector
        _ds = self._params.datasets[
            random.choices(range(len(self._params.datasets)), weights=self.sampling_vec, k=1)[0]
        ]
        return _ds[random.choices(range(len(_ds)), weights=_ds.scenes_w, k=1)[0]]
</code></pre>
<p>But when I use dataloader it give me instance from several dataset, is there a way to enforce it to use the same dataset for the current batch?</p>

ptrblck: <p>You could use a <code>BatchSampler</code> and load the entire batch in your <code>__getitem__</code> method as described <a href="https://discuss.pytorch.org/t/how-to-use-batchsampler-with-getitem-dataset/78788/4">here</a>.</p>
How can I use LAPACK's ssteqr with torch.linalg module

ibkvictor: <p>Hi<br>
I would like to implement <code>ssteqr</code> and <code>ssytrd</code> in the linalg backend representing implicit ql algorithm using MRRR and its symmetric matrix tri-diagonalization respectively. However, I cannot locate where to include this modifications.</p>
<pre data-code-wrap="ssteqr"><code class="lang-plaintext">subroutine ssteqr	(	character 	COMPZ,
integer 	N,
real, dimension( * ) 	D,
real, dimension( * ) 	E,
real, dimension( ldz, * ) 	Z,
integer 	LDZ,
real, dimension( * ) 	WORK,
integer 	INFO 
)	
</code></pre>
<pre data-code-wrap="ssytrd"><code class="lang-plaintext">subroutine ssytrd	(	character 	UPLO,
integer 	N,
real, dimension( lda, * ) 	A,
integer 	LDA,
real, dimension( * ) 	D,
real, dimension( * ) 	E,
real, dimension( * ) 	TAU,
real, dimension( * ) 	WORK,
integer 	LWORK,
integer 	INFO 
)	
</code></pre>

ibkvictor: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> could you help me check this out, please?</p>
<p>Thanks in advance.</p>

ptrblck: <p>I’m not an expert in the CPU/LAPACK backend, but would assume you could define new methods in <a href="https://github.com/pytorch/pytorch/blob/d72db37c4a6513c0f67f6f69870c9c45bf4880e6/aten/src/ATen/native/BatchLinearAlgebra.cpp#L779"><code>BatchLinearAlgebra.cpp</code></a> and the corresponding header.</p>

ibkvictor: <p>Thanks a lot sir <a class="mention" href="/u/ptrblck">@ptrblck</a> . Would I need to rebuild the <code>torch</code> package to implement because I plan to implement the modification on a Jetson Nano device.</p>

ptrblck: <p>Yes, if you manipulate these files you would need to rebuild PyTorch from source.<br>
Alternatively, you could try to write a custom extension as described <a href="https://pytorch.org/tutorials/advanced/cpp_extension.html">here</a>.</p>
How is my model modified when preparing for deployment

ahmadhatahet: <p>Hi everyone,</p>
<p>Let’s say I finished my model training, and now ready for deployment.<br>
While training my model I used Batch Normalization, Dropout, and L2.<br>
My question is: how do these regularizations deploy?<br>
Do they get dismissed, like delete the layers associated with Batch Normalization and Dropout?<br>
And because the model will be in evaluation mode, so the L2 effect on the loss function will be not computed?</p>
<p>Thank you</p>

eqy: <p>For batch normalization, it is a useful transformation even during inference as it is shifting/scaling the outputs of a given layer and the next layer will have learned to depend on this scaling during training. The main difference is that the stats such as mean, variance, etc., do not need to be computed/updated during inference in a deployment setting. Dropout becomes an identity function during deployment inference as none of the hidden units will be dropped during inference. Finally, L2 regularization, as it is computed in the loss function as you note, will not be computed as you do not need to compute the loss function during inference.</p>
<p>Dropout docs describing identity behavior: <a href="https://pytorch.org/docs/stable/generated/torch.nn.Dropout.html" class="inline-onebox" rel="noopener nofollow ugc">Dropout — PyTorch 1.13 documentation</a></p>
<p>The BatchNorm and Dropout changes will happen automatically if you use the builtin implementations in <code>torch.nn</code> and use e.g., <code>model.eval()</code> for deployment.</p>
Why is it required to set seeds separately for each library?

shrbrh: <p>I have set the seed for torch in my code as:</p>
<pre><code class="lang-auto">torch.manual_seed(3)
</code></pre>
<p>Now, since I am importing numpy and tensorflow as well, is it required set the seeds for each library separately? If yes, why?</p>

ptrblck: <p>Yes, this is required since each library uses their own pseudorandom number generator, which needs to be seeded.</p>

shrbrh: <p>Thank you for the clarification.<br>
So does the number I use matter? For example, since I used 3 for torch, should I use 3 for the other libraries also or it can be any random number?</p>

ptrblck: <p>No, I don’t think it matters since different PRNG implementations could be used as already seen in this small example:</p>
<pre><code class="lang-python">torch.manual_seed(2809)
torch.randn(1)
# tensor([-2.0748])

np.random.seed(2809)
np.random.randn(1)
# array([0.23747478])
</code></pre>
<p>Also note that not all libraries support the same backends (e.g. numpy doesn’t support the GPU) so a parity isn’t even possible for all use cases.</p>
Adjust AutoAugment transforms?

supagu: <p>I’m trying to apply some random augmentations to my images and found there is transforms.AutoAugment<br>
which can help.</p>
<p>However, my data needs to eliminate rotations from the augment, or at least make it so only very minor rotations can be applied.</p>
<p>Can the AutoAugment somehow be modified to do what I require?</p>

eqy: <p>You could do something like subclass the existing the existing <code>AutoAugment</code> class and add an if statement to the <code>forward</code> method to skip the rotation augmentation based on the <code>op_name</code> in the for loop.</p>
<p><a href="https://pytorch.org/vision/main/_modules/torchvision/transforms/autoaugment.html#AutoAugmentPolicy" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/vision/main/_modules/torchvision/transforms/autoaugment.html#AutoAugmentPolicy</a></p>
Crash using customized cuda kernel in DDP

yytang220: <p>I try to using cuda kernel with customized torch.autograd.function for both foward and backward (some naive implementation, nothing to do with cuda stream). After it is plugined into DDP pipeline,  crash will happen. I can only see  “NCCL Error 1: unhandled cuda error”.  Is there any sample for this pipeline, Or any idea how to debug these crashes?</p>

kumpera: <p>Did you verify whether it’s your kernel is causing those errors?</p>
<p>One common mistake is to not call cudaGetLastError() after launching a kernel using CUDA syntax.</p>
<p>Another option is to force a stream sync prior to launching a nccl collective to isolate the issue.</p>

yytang220: <p>I found out why. Simply forget to add CUDAGaurd before each kernel launch. Silly mistake</p>
Genetic algorithm for SVM

John4: <p>Hello! I used the sklearn genetic algorithm on random forest to find the best parameters:</p>
<pre><code>genPar = {'num_estimators': Integer(100, 500), 'max_depth': Integer(8, 128),
          'max_samples': Continuous(0.25, 0.9)}
</code></pre>
<p>It took about one day to complete and it was great; but when I apply it on Support Vector Machines it takes much more time:</p>
<pre><code>genPar = {'kernel': Categorical(['linear', 'rbf']), 'C': Integer(1, 10),
          'gamma': Integer(1, 10)}
</code></pre>
<p>Do you have any suggestion to speed-up the process?</p>

ptrblck: <p>This question seems to be scikit-learn - specific so you might want to post it into their discussion board or GitHub repository.</p>
RuntimeError when using load_in_8bit=True for AutoModelForSeq2SeqLM

winterknight: <p>I’m trying to train the google T5-FLAN-XL model but due to resource constraint I am having to lower the precision by setting <strong>load_in_8bit</strong> = True</p>
<p><code>model = AutoModelForSeq2SeqLM.from_pretrained(MODEL_NAME, device_map="auto", load_in_8bit=True, return_dict=True)</code></p>
<p>I’m able to fine tune the model, but at inference time I run into the error :</p>
<pre><code class="lang-auto">     dim = _get_softmax_dim("log_softmax", input.dim(), _stacklevel)
if dtype is None:
     ret = input.log_softmax(dim)
else:
     ret = input.log_softmax(dim, dtype=dtype)

RuntimeError: "log_softmax_lastdim_kernel_impl" not implemented for 'Half'
</code></pre>
<p>I checked and it is because softmax expects float type tensors. I don’t know how to cast the tensors because the softmax layer in <strong>AutoModelForSeq2SeqLM</strong> is abstracted.  Is there a workaround for how I could get the model trained with 8 bit lowered precision, to generate sequences.</p>
<p>Below is my generate function :</p>
<pre><code class="lang-auto">def generate(qamodel, passage, question):
  source_encoding = tokenizer(
        question,
        passage,
        max_length=1024,
        padding="max_length",
        truncation="only_second",
        return_attention_mask=True,
        add_special_tokens=True,
        return_tensors="pt"
    )
  
  generated_token_ids = qamodel.model.generate(
      input_ids=source_encoding["input_ids"],
      attention_mask=source_encoding["attention_mask"],
      max_length=64,
      num_beams=5,
      repetition_penalty=2.5,
      length_penalty=1.0,
      early_stopping=True
    )
  
  preds = [
      tokenizer.decode(token_id, skip_special_tokens=True, clean_up_tokenization_spaces=True)
      for token_id in generated_token_ids
  ]

  return "".join(preds)
</code></pre>

ptrblck: <p>The error is raised in the <code>nn.LogSoftmax</code> layer as it’s not implemented for <code>float16</code> on the CPU. Use the GPU or <code>float32</code> on the CPU and it should work:</p>
<pre><code class="lang-python">sm = nn.LogSoftmax(1)

x = torch.randn(1, 10, dtype=torch.half)
out = sm(x)
# RuntimeError: "log_softmax_lastdim_kernel_impl" not implemented for 'Half'

x = x.to("cuda")
out = sm(x)
print(out.shape)
# torch.Size([1, 10])
</code></pre>
How to create 3d model of HRnet w48 in pytorch

Aram_Salehi: <p>Hi I am new to pytorch and I am wondering if there is a way to create HRnet for 3d image data for image reconstruction?</p>

ptrblck: <p><a href="https://discuss.pytorch.org/t/converting-alexnet-to-3d/71752">This topic</a> migth be interesting as it discusses a similar use case.<br>
However, I don’t know what the author ended up using.</p>
ValueError: Expected input batch_size (4) to match target batch_size (1)

Mert_Arda_Asar: <p>Hello, I am trying to implement a simple attention module to classify images. But when I implemented that module I am getting batch size error. How can I solve that?</p>
<p>Here how I load my dataset:</p>
<pre><code class="lang-auto">X_train, X_test = train_test_split(dataset ,test_size=0.4, random_state=42)

train_loader = DataLoader(X_train)
test_loader = DataLoader(X_test)
</code></pre>
<p>My attention module:</p>
<pre><code class="lang-auto">class AttentionModule(nn.Module):
    def __init__(self, input_dim):
        super(AttentionModule, self).__init__()
        self.fc1 = nn.Linear(input_dim, 128)
        self.fc2 = nn.Linear(128, 64)
        self.fc3 = nn.Linear(64, 1)
        
    def forward(self, x):
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = torch.sigmoid(self.fc3(x))
        
        return x.view(x.size(0), -1)
</code></pre>
<p>My classifier:</p>
<pre><code class="lang-auto">class ImageClassifier(nn.Module):
    def __init__(self):
        super(ImageClassifier, self).__init__()
        self.conv1 = nn.Conv2d(3, 32, kernel_size=3, stride=1, padding=1)
        self.conv2 = nn.Conv2d(32, 64, kernel_size=3, stride=1, padding=1)
        self.fc1 = nn.Linear(64 * 32 * 32, 128)
        self.fc2 = nn.Linear(128, 64)
        self.fc3 = nn.Linear(64, 10)
        self.attention = AttentionModule(input_dim=64 * 32 * 32)
        
    def forward(self, x):
        x = F.relu(self.conv1(x))
        x = F.relu(self.conv2(x))
        x = x.view(-1, 64 * 32 * 32)
        attention_weights = self.attention(x)
        x = x * attention_weights
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x
</code></pre>
<p>Error:</p>
<blockquote>
<p>ValueError: Expected input batch_size (4) to match target batch_size (1)</p>
</blockquote>

ptrblck: <p>Replace:</p>
<pre><code class="lang-python">x = x.view(-1, 64 * 32 * 32)
</code></pre>
<p>with:</p>
<pre><code class="lang-python">x = x.view(x.size(0), -1)
</code></pre>
<p>to keep the batch size equal and fix shape mismatch errors in the next layers, if needed.</p>
Access to the weight of an intermediate layer in a network based on a specific input

S_M: <p>Hi how can I access to the weight matrix based on a specific input image?<br>
my model is like follows structure and I want to access to the fourth feature layer based on an specific image X.<br>
Network(<br>
(features): Sequential(<br>
(0): Conv2d(3, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))<br>
(1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
(2): ReLU(inplace=True)<br>
(3): MaxPool2d(kernel_size=5, stride=5, padding=0, dilation=1, ceil_mode=False)<br>
(4): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))<br>
(5): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
(6): ReLU(inplace=True)<br>
(7): MaxPool2d(kernel_size=5, stride=5, padding=0, dilation=1, ceil_mode=False)<br>
(8): AvgPool2d(kernel_size=1, stride=1, padding=0)<br>
)<br>
(classifier): Linear(in_features=512, out_features=2, bias=True)<br>
)<br>
Any help would be very appreciated.</p>

eqy: <p>Can you do something like <code>Network[4].weight</code> as the <code>nn.Sequential</code> module implements the <code>__getitem__</code> function?</p>

S_M: <p>Hi <a class="mention" href="/u/eqy">@eqy</a> I tried this;</p>
<blockquote>
<p>Network.features[4].weight</p>
</blockquote>
<p>But how can I pass X to this structure?</p>

eqy: <p>I’m not understanding the question. Are you trying to collect the intermediate output of this layer? Are you trying to skip all other layers and just run the layer on the input directly?</p>
<p>For the former I would check out a forward hook example e.g., <a href="https://discuss.pytorch.org/t/how-can-l-load-my-best-model-as-a-feature-extractor-evaluator/17254/6" class="inline-onebox">How can l load my best model as a feature extractor/evaluator? - #6 by ptrblck</a></p>
<p>For the latter, provided that the convolution’s expected input channels matches that of the input, you can create a new convolution layer and clone the weights of the existing layer into it.</p>

S_M: <p>I want to do something like that Network(X).features[4].weight but it dose not correct in Pytorch.<br>
Passing input X to the network and then get the weight of feature layer 4.</p>

eqy: <p>In that case please check the link above and <a href="https://pytorch.org/docs/stable/generated/torch.nn.modules.module.register_module_forward_hook.html" class="inline-onebox" rel="noopener nofollow ugc">torch.nn.modules.module.register_module_forward_hook — PyTorch 1.13 documentation</a> for more information on how to retrieve the intermediate output.</p>
<p>I assume you mean the intermediate output rather than the weight; <code>.weight</code> refers to the parameters of the model which are not affected by <code>X</code> unless a backward pass is performed after. Their values would not change before/during the forward pass with X.</p>

S_M: <p>Thank you very much:).</p>
WeightedRandomSampler - Imbalanced classes Multiclass classification

Er_Hall: <p>When we use WeightedRandomSampler for a mutliclass problem, we set reduction of the loss to mean? (always?) and what this ‘mean’ means?</p>

ptrblck: <p>The usage of a <code>WeightedRandomSampler</code> is independent of the used loss function.<br>
For e.g. <code>nn.CrossEntropyLoss</code>, the <code>reduction='mean'</code> uses</p>
<blockquote>
<p>The losses are averaged across observations for each minibatch.</p>
</blockquote>
<p>as explained in the <a href="https://pytorch.org/docs/stable/nn.html?highlight=crossentropyloss#torch.nn.CrossEntropyLoss">docs</a>.</p>

Er_Hall: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="80514">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>WeightedRandomSampler</p>
</blockquote>
</aside>
<p>Thanks,</p>
<p>Could you explain me briefly what’s the purpose of WeightedRandomSampler? I found an answer you gave here few years ago and I used it, everything ok but I don’t know how exactly I did for my imbalanced data.</p>
<p>Also is it ok that I used both WeightedRandomSampler with reduction = mean?</p>

ptrblck: <p><code>WeightedRandomSampler</code> is used to provide weights to each sample, which is used during the sampling process of selecting the data samples for each batch.<br>
The higher the weight assigned to a particular index, the more likely this data sample will be used in a batch.<br>
To created batches of balanced classes, you would therefore assign a higher weight to samples from the minority class, and a lower weight to samples from the majority class.</p>
<aside class="quote no-group" data-username="Er_Hall" data-post="3" data-topic="80514">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/er_hall/48/16234_2.png" class="avatar"> Er_Hall:</div>
<blockquote>
<p>Also is it ok that I used both WeightedRandomSampler with reduction = mean?</p>
</blockquote>
</aside>
<p>Yes, the reduction is independent from the sampling strategy.</p>

ankita.buntolia: <p>so according to you, if we use <code>WeightedRandomSampler</code> then we do not need to use weights in the cross-entropy loss function, right? If your answer is yes then is there any benefit of using <code>WeightedRandomSampler</code> over weights in cross-entropy loss?</p>

ptrblck: <p>You could use either, as both approaches have the same goal: to counter issues while training on an imbalanced dataset (i.e. overfitting the majority class(es)). Users have different experience with both approaches and I’m sure some might prefer one over the other. I haven’t seen use cases where both methods were used simultaneously.</p>
How to directly replace a certain layer in the pretrained ConvertModel in pytorch

YUYI_ZHONG: <p>Hi,</p>
<p>I have a pytorch model that is converted from onnx model.</p>
<pre><code class="lang-auto">net = copy.deepcopy(model_ori)
print(net)
</code></pre>
<p>And print(net) returns as follows:</p>
<pre><code class="lang-auto">ConvertModel(
  (Constant_9): Constant(constant=tensor([[[[0.13070001]]]], device='cuda:0'))
  (Sub_10): sub()
  (Constant_11): Constant(constant=tensor([[[[0.30810001]]]], device='cuda:0'))
  (Div_12): Div()
  (Conv_13): Conv2d(1, 16, kernel_size=(4, 4), stride=(2, 2), padding=(1, 1))
  (Relu_14): ReLU(inplace=True)
  (Conv_15): Conv2d(16, 32, kernel_size=(4, 4), stride=(2, 2), padding=(1, 1))
  (Relu_16): ReLU(inplace=True)
  (Flatten_17): Flatten()
  (Gemm_18): Linear(in_features=1568, out_features=1000, bias=True)
  (Relu_19): ReLU(inplace=True)
  (Gemm_output): Linear(in_features=1000, out_features=10, bias=True)
)
</code></pre>
<p>I wonder how I can directly replace one layer with another layer (eg: Conv_13 to another self-defined layer) and <code>net</code> is updated as well?<br>
I appreciate any suggestions. Thank you!</p>

J_Johnson: <p>Two considerations:</p>
<ol>
<li>Sizing issues - if you plan to replace the conv2d layer with one of a different size or layer type, that is probably not going to work, unless you make sure the input and output of this intermediate layer(s) are of both the same size and dimensions. In such a case, you’ll probably be better off defining the new structure with a new forward pass and then load your pretrained weights to each layer as appropriate, if transfer learning is involved.</li>
<li>If you’re only changing the weights of <code>Conv_13</code>, you can just use <code>with torch.no_grad():</code> and set the weights and/or bias equal to whatever pretrained tensor you have.</li>
</ol>
<p>For example:</p>
<pre><code class="lang-auto">with torch.no_grad():
    net.Conv_13.weight=new_weights
    net.Conv_13.bias=new_bias
</code></pre>

YUYI_ZHONG: <p>Thank you for your heads-up!<br>
I used <code>net._modules['Conv_13'] = new_layer</code> to do the update</p>
Parallelize Pytorch for loop evaluation?

Goldname: <p>I was wondering if it’s possible to parallelize this model evaluation within a for loop using a gpu here?<br>
Should I be doing this in PyTorch or use multiprocessing module in Python? Here’s an example of one of the things I want to do:</p>
<pre><code class="lang-auto">model.eval()
with torch.no_grad(): 
  for param in model.parameters():
    for j in param.flatten():
      for i in range(0,3):
        j = torch.tensor(i)
        correct = 0
        for batch, label in tqdm(evalloader):
            batch = batch.to(device)
            label = label.to(device)
            pred = model(batch)
            print(pred)
            correct += (torch.argmax(pred,dim=1)==label).sum().item()
            print(correct)
            break
        accuracy = correct/len(evalloader.dataset)
</code></pre>
<p>In the above code I am basically trying to map out what the accuracy landscape looks like by doing a grid search. I manually changing the weights to 0,1,2 and evaluating the model for every single weight/bias of the neural network.</p>
<p>How can I use the GPU to evaluate the model in parallel a bunch of times?</p>
<p>PS: In fact, why doesn’t parallel evaluation of a model happen even for simple cases? Sometimes a testing dataset can be quite large.</p>

eqy: <p>The approach depends on where the bottleneck is. If your GPU(s) are already fully utilized, parallelizing the evaluation loop wouldn’t yield a speedup. On the other hand, if it isn’t fully utilized, I would take a look at increasing the batch size first as that can be a simpler way to add parallelism/increase utilization without jumping through multiprocessing hoops.</p>

Goldname: <p>Do you mean that the batch size is large enough such that the GPU is already fully utilized?</p>

eqy: <p>Yes, I would consider increasing the batch size and see if that is another way of increasing performance. I wouldn’t expect any benefit of parallelizing the for loop over the current implementation as it would just increase contention on the GPU(s).</p>
Using One Model for Training Another Model

Ibrahim_Shaer: <p>Hi everyone. I am using already-trained pytorch models ‘ae_model’ and ‘classifier’ to train another pytorch model. However, the resulting loss function (BCELoss) is not updating the weights of the new model because the prediction grad is <strong>None</strong>.  The requires_grad of all intermediate and final outputs is <strong>True</strong>. How can I mitigate this issue? Thank you for your help.<br>
Should a better solution consider creating a Master class that combines these models altogether?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/e/cef9690d48eaff50000a16d0a4d33d2fc56ce642.png" data-download-href="https://discuss.pytorch.org/uploads/default/cef9690d48eaff50000a16d0a4d33d2fc56ce642" title="pytroch_forums_discriminator"><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/e/cef9690d48eaff50000a16d0a4d33d2fc56ce642.png" alt="pytroch_forums_discriminator" data-base62-sha1="twYIwxnxZSNUEIosaf42i847XIC" width="690" height="288" data-dominant-color="222424"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">pytroch_forums_discriminator</span><span class="informations">1316×551 32 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Namespace Conflict with JUCE

JLenz: <p>When attemping to build libtorch into an audio plugin with JUCE (<a href="https://github.com/juce-framework/JUCE/tree/develop" class="inline-onebox" rel="noopener nofollow ugc">GitHub - juce-framework/JUCE at develop</a>) there is a conflict between nullopt in the two libraries. I see this has been posted on multiple forums but have yet to find a definitive solution. Here is my specific issue:</p>
<pre><code class="lang-auto">In file included from /usr/local/include/torch/script.h:3:
In file included from /usr/local/include/torch/csrc/api/include/torch/types.h:3:
In file included from /usr/local/include/ATen/ATen.h:9:
/usr/local/include/ATen/DeviceGuard.h:27:49: error: reference to 'nullopt' is ambiguous
  return t.has_value() ? device_of(t.value()) : nullopt;
                                                ^
/usr/local/include/c10/util/Optional.h:163:21: note: candidate found by name lookup is 'c10::nullopt'
constexpr nullopt_t nullopt{0};
                    ^
plugin/JUCE/modules/juce_core/containers/juce_Optional.h:27:16: note: candidate found by name lookup is 'juce::nullopt'
constexpr auto nullopt = std::nullopt;
</code></pre>
<p>Is there a version of libtorch with this fixed, or something I can do other than manually changing torch files?</p>

ptrblck: <p>You could add the missing <code>c10</code> namespace into <a href="https://github.com/pytorch/pytorch/blob/f2aee8b8d526a04747a9c2f7b68e2caca611e5e8/aten/src/ATen/DeviceGuard.h#L27">this line of code</a> to fix the issue.<br>
Would you also be interested in submitting the fix directly to PyTorch?</p>
Model runs slower when adding more workers

Messiah: <p>Hi,</p>
<p>In my model, there is a step where I am extracting features for each timestep of a sequence (line 2) that are then passed to my model in a next step (line 6):</p>
<pre><code class="lang-auto">&gt;    1                           state_vector, local_costmap, global_costmap = \     
&gt;    2                                 extract_features(grid, curr_pos, start_pos,     
&gt;    3                                 goal_pos, prev_pos, node_list, batch_size, i,   
&gt;    4                                 device, normalize_inputs)                       
&gt;    5                                                                                 
&gt;    6                             pred, hidden_state = model(state_vector,            
&gt;    7                                                  local_costmap,                 
&gt;    8                                                  global_costmap,                
&gt;    9                                                  use_full_teacher_forcing,      
&gt;   10                                                  hidden=hidden_state)
</code></pre>
<p>The extract_features() function performs a variety of operations (including a for loop over the batch size) to construct the inputs of the model for line 6. What I observe is that when I use a num_workers value of 0, the extract features method runs fast, but whenever I use any multiprocessing (num_workers&gt;0), the extract features method runs slower. I am guessing that my extract_features() method is not conducive to parallel processing (it wasn’t designed for that either).</p>
<p>Therefore I am wondering if I can disable multiprocessing temporarily during the extract_features() step, and resume it once it is finished?</p>

eqy: <p>Is <code>extract_features</code> running on the CPU or another device? If it is on the CPU and using too many threads, you might check if e.g., using the <code>OMP_NUM_THREADS=1</code> environment variable setting or some other small number of threads might help. If it is on another device (e.g., GPU) but is the last step in your data loading/input construction, I would consider simply moving it “downstream” e.g., as part of your model but with something like <code>with torch.no_grad():</code> so that it is not affected by the <code>num_workers</code> parameter.</p>

Messiah: <p>How can I know what device <code>extract_features</code> is running on? I would like to leverage multiprocessing with <code>num_workers &gt; 1</code> to load data faster, therefore I don’t think I want to set  <code>OMP_NUM_THREADS=1</code>. I tried moving the  <code>extract_features</code> inside of the <code>forward()</code> while also using <code>with torch.no_grad()</code>, if that’s what you meant by “moving it downstream”, however I still observe the same behavior as before.</p>

eqy: <p><code>OMP_NUM_THREADS</code> doesn’t limit the number of processes (e.g., what <code>num_workers</code> does) but rather limits the amount of parallelism each worker uses, possibly reducing contention when more workers are used.</p>
What is the use of 1D filter?

Mah_Neh: <p>I have seen a tutorial on CNNs and it seems to me that 1D filters can not possibly be useful to detect edges etc as you’d see in the standard handwritten digit problem.</p>
<p>Is there any reason to use 1D filter instead of a matrix ?</p>

J_Johnson: <p>CNNs for vision tasks typically use 2d kernels. I’ve not seen a model based on 1d kernels. Usually those get used in sequence data like audio or nlp tasks.</p>
<p>Suppose we have an image with:</p>
<pre><code class="lang-auto">[[1, 1, 1],
[0, 0, 1],
[0, 0, 1]]
</code></pre>
<p>And let’s suppose we convolve it with the following kernel with a stride of 1 and no padding:</p>
<pre><code class="lang-auto">[[1, 1],
[0, 1]]
</code></pre>
<p>The output will be:</p>
<pre><code class="lang-auto">[[2, 3],
[0, 2]]
</code></pre>
<p>The pixel where there is a 3 indicates a left facing right angle. In this way, kernels can detect edges. Hope that helps.</p>

Jamie_Donnelly: <p>It might not be very useful when dealing with 2D array inputs like grayscale images, but it can be a useful tool for modelling with vector-based inputs over fully-connected networks.</p>
<p>For instance, if you have a vector-valued input where each item in the vector is indexed in time, a convolution in 1D over the input can act as a ‘smoother’ and recover moving averages, etc.</p>
Get Total Memory Usage by Dataloader Workers & Parent Process

nyxynyx: <p>Is it possible to determine the total amount of CPU RAM memorry being used by the parent PyTorch processs and all the DataLoader worker child processes?</p>
<p>Will like to do this on the bash command line in Ubuntu. This can help me track PyTorch memory usage especially when there is a variable number of other processes running on the machine.</p>
<p>Thanks!</p>

Lucas_Eduardo: <p>You can get the accumulated resource usage if you run your stuff in a container. You can get the info from docker inspect or docker stats.</p>
Does DDP support multiple forward/backward

mangoxb: <p>I want to train the model with multiple branch.<br>
When DDP model is trained with multiple backward (using retain_graph), it still fails to update weights.<br>
(its gradients are exploaded.)</p>
<p>For example, I wrote the code below.<br>
scaler.scale(loss1).backward(retain_graph=True)<br>
scaler.scale(loss2).backward()</p>
<p>If DDP support multiple forward/backward, could you suggest correct usage?</p>
Operation re-naming under onnx exporting

anguoyang: <p>hi guys,<br>
I converted my pytorch model into onnx, and then converted it to SNPE(qualcomm runtime SDK), but met a serious problem:<br>
<code>Mul_281 is not supported in SNPE</code>,<br>
I opened the onnx model with neutron, yes, there is a Mul_281 node indeed, but, I don’t know which operation in my pytorch model responsible for this Mul_281 operation, I guess this Mul_281 is a matrix multiplication in my pytorch model, but renamed during pytorch—&gt;onnx conversion,<br>
My questions is: how could I locate the corresponding operation for Mul_281 in my pytorch model? could I debug inside pytorch-onnx exporting? thank you.<br>
<a class="mention" href="/u/ptrblck">@ptrblck</a></p>
ONNX export AND having input be custom class instance

joakimjohnander: <p>Hey,</p>
<p>As models grow more complex I feel a need to create classes that group multiple inputs. For instance, in DETR-based models adapted to 3D object detection, we might have queries comprise</p>
<ul>
<li>content as (N, B, D) tensor</li>
<li>3D positional encodings as (N, B, D) tensor</li>
<li>2D positional encodings, where the 3D positions have been projected into some camera and then encoded, as (N, B, D) tensor</li>
</ul>
<p>. To that end, I designed a dataclass that comprises these three attributes and that has some utilities (deep copies, pytorch-like indexing, …). This also enables substantially more informative type-hints. However, I cannot export such models to ONNX. For ONNX, each module can take only Tensors, Tuples, Lists, and Dicts as input. Dicts are even advised against. Would it be possible to combine the best of these worlds: (i) nice code for complex models; and (ii) possibility to export to ONNX?</p>
<p>Any advice would be much appreciated!</p>
Meaning of keypoints_scores (Keypoint R-CNN)

Francesco_Pochetti: <p>Hi all,<br>
I am training a <a href="https://pytorch.org/docs/stable/torchvision/models.html#object-detection-instance-segmentation-and-person-keypoint-detection" rel="noopener nofollow ugc">Keypoint R-CNN</a> model and, at inference, I get, among others, <code>keypoints_scores</code>.<br>
I had assumed those were the “confidence scores” for each keypoint, same as <code>scores</code> for bboxes.<br>
I was surprised to see, though, that <code>keypoints_scores</code> can equal any value <code>&gt; 0</code> (e.g. I am getting 10 or 20) and are not constrained in the range 0-1, as bboxes <code>scores</code> are instead.<br>
I could of course normalize those via a sigmoid, but I just want to make sure I understand what those are before doing anything stupid.<br>
Given I don’t see any relevant documentation on <code>keypoints_scores</code>, can anyone please shed some light?<br>
Thanks a lot</p>

Guillermo_Alvarez: <p>Same doubt here, does anyone knows?</p>
Repeat-interleave-mps

r0otx: <p>Cant build repeat-interleave-mps, I have MacBook Pro 16 m1<br>
In conda I’m create env with python 3.10.9<br>
Make<br>
<code>git clone -b repeat-interleave-mps https://github.com/jazzysoggy/pytorch.git</code><br>
After</p>
<pre><code class="lang-auto">cd pytorch &gt; conda install cmake ninja &gt; pip install -r requirements.txt &gt; export CMAKE_PREFIX_PATH=${CONDA_PREFIX:-"$(dirname $(which conda))/../"}
python setup.py develop
</code></pre>
<p>After any time I’m see this error in terminal:</p>
<pre><code class="lang-auto">/Volumes/Developers/Python/pytorch/aten/src/ATen/native/mps/operations/Repeat.mm:199:5: error: use of undeclared identifier 'target_size'
    target_size[idx] = padded_size[idx] * repeats[idx];
    ^
/Volumes/Developers/Python/pytorch/aten/src/ATen/native/mps/operations/Repeat.mm:199:17: error: use of undeclared identifier 'idx'
    target_size[idx] = padded_size[idx] * repeats[idx];
                ^
/Volumes/Developers/Python/pytorch/aten/src/ATen/native/mps/operations/Repeat.mm:199:24: error: use of undeclared identifier 'padded_size'
    target_size[idx] = padded_size[idx] * repeats[idx];
                       ^
/Volumes/Developers/Python/pytorch/aten/src/ATen/native/mps/operations/Repeat.mm:199:36: error: use of undeclared identifier 'idx'
    target_size[idx] = padded_size[idx] * repeats[idx];
                                   ^
/Volumes/Developers/Python/pytorch/aten/src/ATen/native/mps/operations/Repeat.mm:199:51: error: use of undeclared identifier 'idx'
    target_size[idx] = padded_size[idx] * repeats[idx];
                                                  ^
5 errors generated.
[433/1046] Building CXX object caffe2/CMakeFiles/torch_cpu.dir/__/torch/csrc/jit/serialization/unpickler.cpp.o
ninja: build stopped: subcommand failed.
</code></pre>
<p>Why I can’t build this branch? I try install night release but he cant support MPS. Whisper can describe only cpu mode.</p>

r0otx: <p>This problem solved developer, after update, but whisper still work incorrect with mps. <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12" title=":frowning:" class="emoji" alt=":frowning:" loading="lazy" width="20" height="20"></p>
Running multiple instances of same program in single GPU

sathvik_udupa: <p>I have a program with a simple dense layer model, which takes around 2GB/24GB of GPU memory (3090). I want to run multiple instance of the same program , with differet config files, with the same GPU.</p>
<p>I used to be able to do it on other machines, but getting the following error</p>
<blockquote>
<p>RuntimeError: CUDA error: all CUDA-capable devices are busy or unavailable. CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.</p>
</blockquote>
<p>Should I make any changes so that it is not running in compute exclusive mode? there is 20gb of free GPU memory but unable to access it.</p>
<p>Edit: The GPU compute mode was set to exclusive process, changing it back to default did the trick.</p>

siavash307: <p>Did you find any solution for this idea ?</p>
AttributeError: 'ConvNet' object has no attribute 'conv1'

Jacky1: <p>I was trying to learn HiddenLayer visualization, but there was an error in the title when the code was running properly, and I found that my code seemed to be fine.</p>
<pre><code class="lang-auto">
class ConvNet(nn.Module):
    def __init__(self):
        super(ConvNet,self).__init__()
        
        self.convl=nn.Sequential(
            nn.Conv2d(
                in_channels=1,
                out_channels=16,
                kernel_size=3,
                stride=1,
                padding=1,
                ),
            nn.ReLU(),
            nn.AvgPool2d(
                kernel_size=2,
                stride=2,
                ),
            )
        
        self.conv2=nn.Sequential(
            nn.Conv2d(16,32,3,1,1),
            nn.ReLU(),
            nn.MaxPool2d(2,2)
        )
        
        self.fc=nn.Sequential(
            nn.Linear(
                in_features=32*7*7,
                out_features=128,
            ),
            nn.ReLU(),
            nn.Linear(128,64),
            nn.ReLU()
        )
        self.out=nn.Linear(64,10)
    def forward(self,x):
        x=self.conv1(x)
        x=self.conv2(x)
        x=x.view(x.size(0),-1)
        x=self.fc(x)
        output=self.out(x)
        return output
Myconvnet=ConvNet()

import hiddenlayer as hl

hl_graph=hl.build_graph(Myconvnet,torch.zeros([1,1,28,28]))
hl_graph.theme=hl.graph.THEMES["blue"].copy()
hl_graph.save("./deep_learning_data/MyConvnet_h1.png",format="png")
</code></pre>
<p>then</p>
<pre><code class="lang-auto">~\anaconda3\lib\site-packages\torch\nn\modules\module.py in __getattr__(self, name)
   1206                 return modules[name]
   1207         raise AttributeError("'{}' object has no attribute '{}'".format(
-&gt; 1208             type(self).__name__, name))
   1209 
   1210     def __setattr__(self, name: str, value: Union[Tensor, 'Module']) -&gt; None:

AttributeError: 'ConvNet' object has no attribute 'conv1'
</code></pre>

khushi-411: <p>Hey, there!<br>
In the <code>__init__</code> class, you have called using <code>self.convl</code> instead of <code>self.conv1</code>. Seems like a minor typo. Thanks!</p>
Loss not decreasing with QAT applied

a_sigaran: <p>Hello everyone, I wanted to ask for help regarding to QAT. I am using an open source repository for pose estimation, CenterNet <a href="https://github.com/xingyizhou/CenterNet" class="inline-onebox" rel="noopener nofollow ugc">GitHub - xingyizhou/CenterNet: Object detection, 3D detection, and pose estimation using center point detection:</a>.<br>
I am using essentially all their structure except that I created my own ResNet18 architecture and trained a model with that architecture. The model without QAT is working but when I try to add the QAT my loss is not decreasing and I am not sure where is the correct place to convert it.<br>
At the beginning I thought that the “convert” step was my problem, because when I trained the whole model my accuracy dropped to 0 and the loss was oscillating in the same values. but after I created an unit test for my loss calculation with QAT applied the loss and shapes are correct. Essentially I am doing the same in my unit test as in the main, except that instead of using the original dataset I generated a dataset with random numbers(in my unit test). In both options I have my QAT configuration applied without the “convert” step. Meaning that there is something in the middle that is making my unit test work correctly and  CenterNet model not, but I cannot find it. You will notice that the QAT is being apply just in my fist layers before deconvolution.</p>
<p>Could you guys help me trying to figured out? I will add in a private colab(unfortunately I cannot publish it) my modified main with the QAT configuration and my architecture. For the moment I just added <a class="mention" href="/u/jerryzh168">@jerryzh168</a>  as collaborator, but if someone else could help me too, I will be grateful and I can add them too.<a href="https://colab.research.google.com/drive/1AD0XYxYEFzJlXAE5FpKrXFjszXqVjSZg?usp=sharing" class="inline-onebox" rel="noopener nofollow ugc">Google Colab</a></p>
<p>Thank you very much in advance</p>
Am I initializing the hidden state of the RNN correctly? Why is my validation accuracy not improving

peony: <p>Hello, here is the code to my MLP-GRU model:</p>
<pre><code class="lang-auto">
class MLP(nn.Module):
  
  def __init__(self, num_classes, rnn_layers, hidden_size, fc_size):
    super().__init__()
    self.hidden_size = hidden_size
    self.num_classes = num_classes
    self.fc_size = fc_size
    

    self.fc_pre= nn.Sequential(nn.Linear(201, 120),
                               nn.Dropout(p=0.5),
                           nn.ReLU(),
                           nn.Linear(120, 75),
                           nn.Dropout(p=0.5),
                           nn.ReLU(),
                           nn.Linear(75, fc_size),
                           nn.Dropout(p=0.5),
                           nn.ReLU())
    #Defines the number of features that define each element (time-stamp) of the input sequence
    self.rnn = nn.GRU(input_size = fc_size,
                hidden_size = hidden_size,
                num_layers = rnn_layers,
                batch_first = True)
    self.fc = nn.Linear(hidden_size, num_classes)
    #feed the previous layer output to softmax and do softmax on the class dimension
    #self.softmax = nn.Softmax(dim=-1)
  
  #initialize hidden state for RNN  
  def init_hidden(self, num_layers, batch_size):
         return (torch.zeros(num_layers, batch_size, self.hidden_size).cuda(),
                 torch.zeros(num_layers, batch_size, self.hidden_size).cuda())

  def forward(self, inputs, hidden=None, steps=0):
        length = len(inputs)
        # print('inputs size')
        # print(inputs[3].size())
        #Input data: RNN should have 3 dimensions. (Batch Size, Sequence Length and Input Dimension (the number of expected features which is 201)
        fs = torch.zeros(inputs[0].size(0), length, self.rnn.input_size).cuda()

        for i in range(length):
            f = inputs[i]
           
            #flattens the tensor
            f = f.view(f.size(0), -1)
            #print(f.size())
            f = self.fc_pre(f)

            # stores the tensor f in the shape [fs.size(0), fs.size(2)] to each “row” in fs to prepare for learning the sequence in RNN
            fs[:, i, :] = f
            # print('fs')
            # print(fs.size())
       
            
        outputs, hidden = self.rnn(fs, hidden)
        #print(outputs.size())
        
        #the training code applies crossentropyloss as criterion which also applies softmax to the output so I don't have to use softmax here
        outputs = self.fc(outputs)
        #outputs = self.softmax(self.fc(outputs))
        #print(outputs)
        #print(outputs.size())
        
        return outputs
</code></pre>
<p>the fc_size is equivalent to the input to my MLP which is 201. I wish to know if I am initializing the RNN correctly. Thank you in advance</p>
Multi-thread and cudaMalloc

oracal: <p>I conducted  a simple experiment on  DL inference as follows:</p>
<h1>
<a name="warm-up-1" class="anchor" href="#warm-up-1"></a>warm up</h1>
<p>model.cuda()<br>
img = img.cuda()<br>
with torch.no_grad:<br>
for _ in range(100):<br>
y =  model(img)<br>
torch.cuda.synchronize()</p>
<p><span class="mention">@torch.no_grad</span>()<br>
def test():<br>
for _ in range(10):<br>
out = model(img)<br>
torch.cuda.synchronize()</p>
<p><em><strong>case 1</strong></em><br>
test()<br>
<em><strong>case 2</strong></em><br>
child_thread = Threading.Thread(target=test, …)<br>
child_thread.start()<br>
child_thread.join()</p>
<p>After conducting the analysis, I found that the inference time for Case 2 surpasses that of Case 1 significantly .  Further profiling of  the program using <em><strong>Nsight System</strong></em> revealed that there is no  <em><strong>cudaMalloc</strong></em> function call in Case 1 during inference,  whereas a substantial amount of  <em><strong>cudaMalloc</strong></em>  call are present in Case 2. This prompts the inquiry into the relationship between multi-threading and memory management in PyTorch. Could you give some suggestions? Thanks so much !</p>

eqy: <p>Just a hunch, but I would check if this is because the CUDACachingAllocator doesn’t mix allocations across different threads/streams. Do you observe the same behavior when also doing the warmup on the child thread?</p>

oracal: <p>You are right! When I doing the warmup on the child thread, there is no cudaMalloc call during inference</p>

oracal: <p>I have mix allocation across different streams for my development. I wonder how can I mix allocation across different threads?</p>

eqy: <p>In general I think this is a tricky use-case. I would take a look at the <code>record_stream</code> and <code>wait_stream</code> functions for managing tensor lifetypes and ordering respectively:<br>
<a href="https://pytorch.org/docs/stable/generated/torch.Tensor.record_stream.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.Tensor.record_stream.html</a><br>
<a href="https://pytorch.org/docs/stable/generated/torch.cuda.Stream.html#torch.cuda.Stream.wait_stream" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.cuda.Stream.html#torch.cuda.Stream.wait_stream</a></p>

oracal: <p>Thanks, I have modified some code in CUDACachingAllocator to mix allocations across different streams successfully. Now , I want to achieve that across different threads. How can I do that</p>