akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Ai model integration

pritam_sengupta13: <p>TypeError: empty() received an invalid combination of arguments - got (tuple, dtype=NoneType, device=NoneType), but expected one of:</p>
<ul>
<li>(tuple of ints size, *, tuple of names names, torch.memory_format memory_format, torch.dtype dtype, torch.layout layout, torch.device device, bool pin_memory, bool requires_grad)</li>
<li>(tuple of ints size, *, torch.memory_format memory_format, Tensor out, torch.dtype dtype, torch.layout layout, torch.device device, bool pin_memory, bool requires_grad)<br>
i m getting this error for the code<br>
class IntentModel(nn.Module):<br>
def <strong>init</strong>(self, input_size, output_size):<br>
super(IntentModel, self).<strong>init</strong>()<br>
self.fc = nn.Linear(input_size, output_size)<br>
self.weight = torch.empty((output_size, input_size))  # Corrected initialization<br>
def forward(self, x):<br>
x = self.fc(x)<br>
return x</li>
</ul>

ptrblck: <p>Could you post how <code>input/output_size</code> are defined?</p>
What should collator do exactly?

kuraga: <p>Suppose we have an audio classification task (AudioMNIST).</p>
<p>My pipeline and other pipelines I’ve seen consist of the next steps:</p>
<ol>
<li>Read the dataset (the data samples).</li>
<li>Do the base transforms (merge the audio channels, change the bitrate, etc).</li>
<li>Split the dataset into the train one, the test one, etc.</li>
<li>Do the main transforms (different for the train and the test) such as the augmentation.</li>
<li>Batch (along with the sampling).</li>
<li>Pad/Truncate the batch samples.</li>
<li>Do the forward pass with the batch.</li>
<li>&lt;…&gt;</li>
</ol>
<p>I saw the scheme:</p>
<ul>
<li>Dataset or a subclass - pp. 1., 2., 3., 4.</li>
<li>Collator - p. 6.</li>
</ul>
<p>Either:</p>
<ul>
<li>Dataset or a subclass - p. 1.</li>
<li>somebody else - pp. 2., 3., 4.</li>
<li>Collator - p. 6.</li>
</ul>
<p>Either:</p>
<ul>
<li>Dataset or a subclass - p. 1.</li>
<li>somebody else - p. 3.</li>
<li>Collator - pp. 2., 4., 6.</li>
</ul>
<p>What should the collator do and what shouldn’t? (The main question.)</p>
<p>What is the correct scheme? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>Thanks!</p>

kuraga: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> , would appreciate your answer <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

ptrblck: <p>The <code>collate</code> function is defined in the <code>DataLoader</code> and usually responsible to create the actual batch using the returned samples from the <code>Dataset.__getitem__</code>. The <code>__getitem__</code> would load, transform, and return a single samples while the <code>collate_fn</code> would then optionally pad the samples and stack them to a single batch.<br>
Let me know if this fits your understanding or if more details are needed.</p>

kuraga: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> , thanks! But if I want to populate samples? So, one dataset sample goes to five augmented ones. Who should do it?</p>

ptrblck: <p>If you want to load a single sample and transform it multiple times, you could add these operations into the <code>__getitem__</code> and return all samples at once. This would of course increase your actual batch size since by default a single sample is expected in the <code>collate_fn</code>. Let me know if I misunderstand your use case or question.</p>

kuraga: <p>Well, to learn the <code>DataLoader</code> subclass to call <code>dataset.get_augmented_items(n: int) -&gt; list[T_co]</code> and… rearrange batches… Understand.</p>
<p>The collator doesn’t change its behavior meanwhile.</p>
"Error when running model: TypeError: 'generator' object is not callable"

Pandian_p: <pre><code class="lang-auto">---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
&lt;ipython-input-53-58ca628450f4&gt; in &lt;cell line: 56&gt;()
     54 dict_dataloader_test = DataLoader(dict_dataset_test, batch_size=train_batch_size, pin_memory=True, drop_last=False)
     55 
---&gt; 56 for it, (images, captions) in enumerate(dataloader_train):
     57     print(images)

2 frames
/usr/local/lib/python3.10/dist-packages/torch/utils/data/dataloader.py in __next__(self)
    629                 # TODO(https://github.com/pytorch/pytorch/issues/76750)
    630                 self._reset()  # type: ignore[call-arg]
--&gt; 631             data = self._next_data()
    632             self._num_yielded += 1
    633             if self._dataset_kind == _DatasetKind.Iterable and \

/usr/local/lib/python3.10/dist-packages/torch/utils/data/dataloader.py in _next_data(self)
    673     def _next_data(self):
    674         index = self._next_index()  # may raise StopIteration
--&gt; 675         data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
    676         if self._pin_memory:
    677             data = _utils.pin_memory.pin_memory(data, self._pin_memory_device)

/usr/local/lib/python3.10/dist-packages/torch/utils/data/_utils/fetch.py in fetch(self, possibly_batched_index)
     47         if self.auto_collation:
     48             if hasattr(self.dataset, "__getitems__") and self.dataset.__getitems__:
---&gt; 49                 data = self.dataset.__getitems__(possibly_batched_index)
     50             else:
     51                 data = [self.dataset[idx] for idx in possibly_batched_index]

TypeError: 'generator' object is not callable




This error got when i when i implement the code https://github.com/weimingboya/DFT/
</code></pre>
<p>“I attempted to execute the code as provided on the GitHub page, but encountered an issue. I’m seeking assistance to resolve this problem as I’m unsure why it occurred.”</p>
<p>sir <a href="https://colab.research.google.com/drive/1xMJUmr6KhZkoglnvcpxYQTTUM1V4qeD4" class="inline-onebox" rel="noopener nofollow ugc">Google Colab</a></p>
<p>can you access this google colab link for accessing my code</p>
Warning of Pandas when customizing a dataset

dmerinodel: <p>Hi all <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>I’m coding a custom (but simple) dataset for training an MLP with PyTorch. Basically, my data are numeric vectors of two classes so the whole dataset is a pandas.DataFrame whose first column is 0 or 1 (representing both classes) and the rest of columns are the numeric component of the vectors. Since each row represents a data, I consider that a natural PyTorch’s dataset class would be:</p>
<blockquote>
<pre><code>class MatrixDataset(Dataset):
    def __init__(self, data: pd.DataFrame, device):
        self.data = data
        self.device = device

    def __len__(self):
        return self.data.shape[0]

    def __getitem__(self, ind):
        row = self.data.iloc[ind]
        x = torch.tensor(row.iloc[1:], dtype=torch.float32, device=self.device)
        y = torch.tensor(row.iloc[0], dtype=torch.float32, device=self.device)
        return x, y
</code></pre>
</blockquote>
<p>Actually, this Dataset works fine, but I receive the following warning</p>
<blockquote>
<p>/path_to_my_script/script.py: FutureWarning: Series.<strong>getitem</strong> treating keys as positions is deprecated. In a future version, integer keys will always be treated as labels (consistent with DataFrame behavior). To access a value by position, use <code>ser.iloc[pos]</code><br>
x = torch.tensor(row.iloc[1:], dtype=torch.float32, device=self.device)</p>
</blockquote>
<p>After searching about this warning, I still don’t get why it triggers because as many people say the warning is self-explanatory: ‘use .iloc method’ but I’m already using it. Furthermore, I try to replicate the error using Python’s interpreter in terminal and it doesn’t trigger.</p>
<p>Any help would be appreciated it, thanks in advance!</p>

Deneme13: <p>did you take that warning from all iloc’s. or only with x= part. is it about iloc[1:]</p>

dmerinodel: <p>It is only from that part. I ‘patched’ it by passing .values instead of the actual serie <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
Custom Neural Network Environment

ptrem: <p>Hi,<br>
I’m new in the field of reinforcement learning and currently searching for a suitable library to implement my deep reinforcement learning research project. It should be a Python library and use PyTorch for the deep learning integration. I would like to run multiple environments in parallel on (multiple) GPU. As my environment will contain a Neural Network it should be able to directly run on the GPU and work with tensors. I would like to hear your opinion and about your experience if TorchRL is the right (a suitable) way to go.</p>
<p>Thanks</p>
How to access the last layer of the give model

Nandani_sharma: <p>class PneumoniaCnnModel(BinaryClassificationBase):<br>
def <strong>init</strong>(self):<br>
super(PneumoniaCnnModel, self).<strong>init</strong>()<br>
# Load pre-trained DenseNet-121 model<br>
self.model = models.densenet121(pretrained=True)</p>
<pre><code>    # Freeze all layers except the last convolutional layer
    for param in self.model.parameters():
        param.requires_grad = False

    # Unfreeze the parameters of the last convolutional layer
    for name, param in self.model.named_parameters():
        if 'denseblock4' in name:
            param.requires_grad = True
            self.last_convolution_layer = name

    # Replace classifier with a new linear layer
    num_features = self.model.classifier.in_features
    self.model.classifier = nn.Sequential(
        nn.Linear(num_features, 512),  # Example new layer
        nn.ReLU(inplace=True),
        nn.Dropout(p=0.5),
        nn.Linear(512, 1)  # Output 1 value for binary classification
    )

def forward(self, x):
    # Pass through the model
    x = self.model(x)
    # Apply sigmoid activation function to the output
    x = torch.sigmoid(x)
    return x

def last_layer(self):
    return self.last_convolution_layer
</code></pre>

eqy: <p>The sequential layer should be indexable</p>
<pre><code class="lang-auto">def last_layer(self):
    return self.model.classifier[-1]
</code></pre>
Static Symbolic Method Operator

sfy: <p>I found a piece of <a href="https://github.com/triple-Mu/YOLOv8-TensorRT/blob/main/models/common.py#L27" rel="noopener nofollow ugc">code</a> that adds <code>TRT::EfficientNMS_TRT</code> to the YOLOv8 network and exports it to ONNX. The <code>TRT::EfficientNMS_TRT</code> used in it seems to be similar to <a href="https://github.com/NVIDIA/TensorRT/tree/main/plugin/efficientNMSPlugin" rel="noopener nofollow ugc">efficientNMSPlugin</a>, but its parameters are slightly different from those given in the documentation, with the addition of suffixes(e.g. <code>_f</code>) representing data types. In this code, the forward function only returns random tensors. I would like to know the specific requirements or limitations of the forward function and symbolic function (are the input parameters the same?). The PyTorch documentation only provides information about TorchScript operators and ONNX standard operators, but I want to know how TensorRT plugins are supported in <code>g.op()</code>. Where can I find tutorials or documentation related to this?</p>
Emulate distributed training setup with 1 GPU

justinliu: <p>I’m writing code for distributed training on multiple GPUs. However, my local devbox has only 1 GPU. Is it possible to emulate a multi-GPU setup on a 1-GPU devbox to test the code (esp. the parts that have collective communications) locally?</p>

H-Huang: <p>Hey <a class="mention" href="/u/justinliu">@justinliu</a>, you can use <code>gloo</code> which is a cpu backend (<a href="https://pytorch.org/docs/master/distributed.html?highlight=distributed#:~:text=Backend-,gloo,-mpi" class="inline-onebox" rel="noopener nofollow ugc">Distributed communication package - torch.distributed — PyTorch master documentation</a>) which supports a lot of the same collectives as nccl. Does that work?</p>
<p>There is currently no way to virtualize multiple gpus from 1 gpu in pytorch. Some nvidia gpus support MIG (<a href="https://www.nvidia.com/en-us/technologies/multi-instance-gpu/" class="inline-onebox" rel="noopener nofollow ugc">NVIDIA Multi-Instance GPU(MIG)</a>) however the collective communications in nccl do not support this</p>

justinliu: <p>Got it. This makes sense. Thank you!</p>
Applying torchvision Resize transform to ndarrays

lime: <p>In torchvision version 0.16.2+cu121 resizing a numpy array won’t resize and doesn’t give back any errors/warnings for not resizing due to the input type. I don’t know if this is intended but it might cause some confusion.</p>
<pre><code class="lang-auto">from torchvision.transforms.v2 import Resize
import numpy as np

img = np.random.randint(255,size=(1024,2048))
img_size = (256,512)
trans = Resize(img_size, antialias=True)
transimg=trans(npimg)
(npimg==transimg).all()
transimg.shape
</code></pre>
Example of using quantized.Embedding?

HuangLED: <p>I’ve read the quantization doc here: <a href="https://pytorch.org/tutorials/advanced/static_quantization_tutorial.html#quantization-aware-training" class="inline-onebox" rel="noopener nofollow ugc">(beta) Static Quantization with Eager Mode in PyTorch — PyTorch Tutorials 2.2.1+cu121 documentation</a></p>
<p>but I couldn’t find an example for an embedding model to work with quantized.Embedding. Is it documented somewhere?</p>
<p>Simply replace the embedding module, lead to error of empty parameter list error.</p>
<p>Thanks!</p>

HDCharles: <p>see the table above: <a href="https://pytorch.org/docs/stable/quantization.html#eager-mode-quantization" class="inline-onebox" rel="noopener nofollow ugc">Quantization — PyTorch 2.2 documentation</a></p>
<p>if you apply either the fx or eager quantization flows to a model with embeddings, they should get quantized.</p>
Where is torch.ops.quantized.conv2d defined?

James_Diffenderfer: <p>Hello,</p>
<p>I am trying to understand <em>exactly</em> what the function on line 468 of this code does: <a href="https://github.com/pytorch/pytorch/blob/eba28a6f911ad02515be0f22dc814d37b122f1cd/torch/ao/nn/quantized/modules/conv.py#L468" class="inline-onebox" rel="noopener nofollow ugc">pytorch/torch/ao/nn/quantized/modules/conv.py at eba28a6f911ad02515be0f22dc814d37b122f1cd · pytorch/pytorch · GitHub</a> .</p>
<p>I have tried tracing back to find the source code for this function (<code>torch.ops.quantized.conv2d</code>) but have been unable to find it yet. I have already quantized models successfully in PyTorch and I am trying to better understand the exact processes/steps in the quantized Conv2D module. Ideally I would prefer a link to the source code instead of getting an explanation as to what it does.</p>
<p>Thanks!</p>

HDCharles: <p>best way to do this is in general is to look in native_functions.yaml or for quantized ops, library.cpp <a href="https://github.com/pytorch/pytorch/blob/main/aten/src/ATen/native/quantized/library.cpp" class="inline-onebox" rel="noopener nofollow ugc">pytorch/aten/src/ATen/native/quantized/library.cpp at main · pytorch/pytorch · GitHub</a></p>
<p>from there you can search for quantized::conv2d.new you’ll find the file: <a href="https://github.com/pytorch/pytorch/blob/773ae817f7bd1d428bc1a0bc83ca56f84e0d6a87/aten/src/ATen/native/quantized/cpu/qconv.cpp#L1931" class="inline-onebox" rel="noopener nofollow ugc">pytorch/aten/src/ATen/native/quantized/cpu/qconv.cpp at 773ae817f7bd1d428bc1a0bc83ca56f84e0d6a87 · pytorch/pytorch · GitHub</a></p>
Information about variables that connected to graph

mY_NAme: <p>Hello! Does anybody know how can I get information about variable’s connection to graph? For example, I have a tensor A with shape [3, 5, 10] and required_grad=True, Let’s say I just want to create new tensor in for loop that will be connected to the same graph. And for example, if I will create just an empty tensor B like torch.empty(3, 5, 10) and then insert rows from A to B like B[I] = A[I] in for loop, B will not be connected to the graph of A, but if I will use B = torch.tensor(<span class="chcklst-box fa fa-square-o fa-fw"></span>) and then torch.cat((B[I], A)) in for loop - graph will be the same. Here is a question: how can I get information about torch operations that will use the same graph with new variable.</p>
How to implement VAE loss (Cosine similarity and KLD)

Gabriel_CRETIN: <p>I am unsure about how to implement correctly the KLD loss of my Variational Auto Encoder.</p>
<p>I work with input tensors of shape (batch_size, 256, 768), and at the bottleneck/latent dim of the VAE the tensors are of shape (batch_size, 32, 8) which are flattened by the FC layers for mu and log_var calculations to (batch_size, 256).<br>
The ending of input tensors are padded by zeros because the input tensors have different sequence lengths.<br>
My reconstruction loss is a cosine similarity loss calculated between the input tensors and the reconstructed tensors defined as:</p>
<pre><code class="lang-python">def masked_cosine_similarity_loss(output: torch.Tensor, target: torch.Tensor, masks:torch.Tensor):
    masks = masks[:, :, 0]
    cos = nn.CosineSimilarity(dim=2)
    cos_loss = cos(output, target)
    cos_loss = 1.0 - cos_loss
    cos_loss *= masks
    cos_loss = torch.sum(cos_loss) / torch.sum(masks)
    return cos_loss
</code></pre>
<p>And my KLD loss is calculated on the latent dim vectors of shape (batch_size, 256), defined as:</p>
<pre><code class="lang-python">KLD_loss = -0.5 * torch.sum(1 + log_var - mu.pow(2) - log_var.exp())
</code></pre>
<p>Are the lossed calculated correctly ? Knowing that the sequence dimension is 32 and features dim is 8 (in the latent dim), should I reshape the mu and log_var tensors from (batch, 256) to (batch, 32, 8) before calculating the KLD ? And if yes, should it be calculated on a specific dimension and normalized somehow ?<br>
Also, the cosine similarity is calculated on padded tensors, even though I multiply by the masking just after, I guess it is not really the best way to calculate it but I didn’t find anything on the  net that does this ?</p>

banikr: <p>did you get to make it work?</p>
Addition of the new IDGI framework to the list of explainability methods

backslash: <p>IDGI is a relatively new attribution method described in <a href="https://arxiv.org/abs/2303.14242" rel="noopener nofollow ugc">this paper in 2023</a></p>
<p>IDGI is a framework that can be applied to any integrated gradients method (BlurIG, GIG etc.) however I propose implementing it for the vanilla integrated gradients method. The implementation of this would exist as a separate class and would not be dependant on the IntegratedGradients class, which would keep the code implementation clean.<br>
The mathematical implementation of IDGI is simple and it is not very compute-intensive</p>
<p>I have opened an issue for the same on <a href="https://github.com/pytorch/captum/issues/1246" rel="noopener nofollow ugc">Github</a><br>
I would like to work on the implementation of this framework. Should I go ahead?</p>
Remapping of KITTI dataset

Deneme13: <p>I was search most of the sources i think, but can not find how to do. I use KITTI dataset from pytorch <code>training_set = torchvision.datasets.Kitti(root, train=True, download=True,transform=transform)</code> with <code>datasets.wrap_dataset_for_transforms_v2</code> so that wrapper makes <code>0:car 1:van 2:truck ... to 8:DontCare</code> I think,<br>
want to use retinanet and faster r-cnn, when I look up their source code there is warning as <code>__ground__</code> is default 0 so add to your classes +1 but i don’t know is my dataset labels add. or my model try to <code>0:__ground__  and 0:car</code> same time? i did not get error while training or draw it but classification of objects are terrible but bboxes are pretty good. could you help me? thanks in advance for even read.</p>
Speed up model transformation DistributedDataParallel

Johan_Adler: <p>Hi, I have a model defined as the following:</p>
<pre><code class="lang-auto">Doc2VecModel(
  (word_embeddings): Embedding(23026, 400)
  (doc_embeddings): Embedding(23026, 400)
  (linear): Linear(in_features=400, out_features=23026, bias=True)
  (linear2): Linear(in_features=800, out_features=400, bias=True)
  (log_softmax): LogSoftmax(dim=1)
)
</code></pre>
<p>The program takes a long time to execute, wrapping the model in DDP ie:</p>
<p><code>DDP(model, device_ids=[gpu_id])</code></p>
<p>However, if I reduce the size of the model to something like this:</p>
<pre><code class="lang-auto">Doc2VecModel(
  (word_embeddings): Embedding(400, 400)
  (doc_embeddings): Embedding(400, 400)
  (linear): Linear(in_features=400, out_features=400, bias=True)
  (linear2): Linear(in_features=400, out_features=400, bias=True)
  (log_softmax): LogSoftmax(dim=1)
)
</code></pre>
<p>then it executes almost instantly. Is there a way to speed up the model transformation for the first model?</p>

H-Huang: <p>In DDP initialization the model parameters are broadcasted to all ranks in the DDP group, so there is not much way around that. The initialization is a 1 time cost so if the training over multiple hours the initialization time ratio will be minimal</p>
<p>the first model has 23026 num_embeddings, out_features which is 58 times larger compared to model 2 with 400, so it makes sense that 2nd model is much faster. It would be like trying to initialize model 2, but 58 times.</p>
Data Partition to GPU Mapping

donparker1: <p>Currently, in deepspeed, it seems data ranks are interleaved for a specific number of pipeline stages on a single node for different cards.  I was wondering if anyone could provide any suggestions on how I can map data ranks to the same card for any number of pipeline stages</p>

H-Huang: <p>Sorry, I’m unsure what does “data rank” mean in this context? For pipeline parallelism data should only be going into the first stage of the pipeline group right? In interleaved 1f1b (<a href="https://arxiv.org/pdf/2104.04473.pdf" rel="noopener nofollow ugc">https://arxiv.org/pdf/2104.04473.pdf</a>) one gpu (rank) will have multiple pipeline stages, you can do by setting that model chunk to that specific gpu.</p>
<p>FYI PyTorch has an official pipeline parallelism package (<a href="https://github.com/pytorch/PiPPy" class="inline-onebox" rel="noopener nofollow ugc">GitHub - pytorch/PiPPy: Pipeline Parallelism for PyTorch</a>) that is going to upstreamed into the PyTorch repository into the coming months (you will be able to use these APIs from just <code>import torch</code>). The package provides support for the most popular schedules (gpipe, 1f1b, interleaved 1f1b, BFS) (<a href="https://github.com/pytorch/PiPPy/blob/main/pippy/PipelineSchedule.py" class="inline-onebox" rel="noopener nofollow ugc">PiPPy/pippy/PipelineSchedule.py at main · pytorch/PiPPy · GitHub</a>)</p>
Multi CPU parallel calculation

Giuseppe_SatSriAkal: <p>Hi everybody,<br>
I’m trying to run a script on multiple CPU/multiple core system. I’ve never used it. I’ve tried to do it searching in the web, but without success. Could you give me instructions?</p>
<p>My system:<br>
PC Core i-7 - Quad Core model: Intel Core i7-10510U bits: 64<br>
Linux Mint 20<br>
Anaconda - Spyder editor<br>
Application: script with pytorch for machine learning LSTM algorithm</p>
<p>I can run the same code on a remote computer with more CPU’s.</p>
<p>Where should I start in order to configure the script in the right way?</p>
<p>Best regards<br>
Beppe</p>

H-Huang: <p>You will need to update the script to use some distributed training APIs, please see this tutorial: <a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Getting Started with Distributed Data Parallel — PyTorch Tutorials 2.2.1+cu121 documentation</a></p>
<p>You can use the <code>gloo</code> backend which is a CPU backend</p>
Vectorize nn.ModuleDict

gbartyzel: <p>Hi,</p>
<p>For the past few days, I’ve been trying to figure out how can I speed up my multi-modal model (a feature extract for reinforcement learning that takes as input images and robot states). I know that <code>torch.func</code> provides a function <code>stack_module_states</code>, however, the models in the list have to have the same architecture. I wonder if there is a way to improve the model represented by the <code>nn.ModuleDict</code> to receive a dictionary of tensors, run all models in parallel and then collect the results. Thanks in advance for any advice!</p>
<p>Best,<br>
Greg.</p>
Disable gradient for one layer operation

alaria: <p>To calculate my final loss, a trained layer is used to pre-process the ground-truth labels for later comparison against another model result. I only want that layer to learn from the first input (actual input data) but not from the second input (ground truth). But I need the same operation performed on both so future steps remain comparable. Is there a way to temporarily disable “learning”?</p>
<p>Pseudo-code:</p>
<blockquote>
<p>self.enconding_layer = torch.nn.linear()<br>
self.loss_func = torch.nn.L1Loss()<br>
…<br>
encoded_input = self.encoding_layer(input)<br>
encoded_ground_truth = self.encoding_layer(ground_truth_label)<br>
…<br>
loss = self.loss_func(encoded_input, encoded_ground_truth</p>
</blockquote>

ptrblck: <p>I’m not sure if I understand your question completely, but as long as you don’t call <code>.backward()</code> on a loss, no gradients will be calculated and thus this loss won’t have any influence on the training.</p>

alaria: <p>I’m training a transformer and I want to perform positional encoding the ground truth labels of future values of a sequence before comparing them to the transformer outputs. I’m not sure how this is commonly done, directly comparing them without encoding one of them seems wrong. So effectively both encodings become part of the loss which is used for the gradients, but I thought maybe the inputs should be considered.</p>
Pytorch tensorrt compile

Snehashis_Chatterjee: <p>Tried to convert torch script model, got below error<br>
“WARNING: [Torch-TensorRT] - Input 0 of engine _run_on_acc_0_engine was found to be on cuda:1 but should be on cuda:0. This tensor is being moved by the runtime but for performance considerations, ensure your inputs are all on GPU and open an issue here (<a href="https://github.com/pytorch/TensorRT/issues" class="inline-onebox" rel="noopener nofollow ugc">Issues · pytorch/TensorRT · GitHub</a>) if this warning persists.”</p>
<p><code>self.__network = torch.load(weight_file, map_location=self.__device) self.__network1 = torch_tensorrt.compile(...ir = 'dynamo) </code></p>
<p>I have two gpu in my machine</p>

ptrblck: <p>Have you checked the <code>device</code> of the input and the model, as the warning points to a device mismatch?</p>

Snehashis_Chatterjee: <p>Yes, model convert in gpu 0, try to load weights file in GPU 1, during loading input also GPU 1, so input and model in same device or GPU id,<br>
I converted model in GPU 0, but tried to compile and infer in gpu 1, the same kind of error I am getting for torch torch-scripted model.</p>

ptrblck: <p>I’m unsure if the issue is caused by <code>TorchScript</code>, but could you move the model to the CPU before saving and move it back to the desired GPU after loading?</p>
DiffusionInst training with publaynet dataset

raushankm: <p>I am trying to run an Image segmentation training on the publaynet dataset with this github codebase. <a href="https://github.com/chenhaoxing/DiffusionInst/tree/main" class="inline-onebox" rel="noopener nofollow ugc">GitHub - chenhaoxing/DiffusionInst: This repo is the code of paper "DiffusionInst: Diffusion Model for Instance Segmentation" (ICASSP'24).</a><br>
It is trying to implement the Diffusion model on top of detectron2 for the instance segmentation. It is trained on coco dataset. However when I am training it on the publaynet dataset it is not segmenting well. The loss is decreasing to 2.5 after 250000 iterations. Also before feeding the input I checked the inputs publaynet docs and bounding boxes it seems to fit well. But still, The problem that is happening the trained model is giving the output for the whole image as one segment. It is not segmenting the objects individually. What could be the possible issue here?</p>
<p>Also another question this whole training the model is taking at least 3 days for me. So, to check any result whether it’s trained or not properly I am able to see only after 3 days. How do I debug it without wasting so much time on the training with whole data set? Any pointer will be hlpful.</p>
Stuck at Downloading Pytorch

Gimyung: <p>Hello,i’m downloading Pytorch,but it is stuck like this.How can I solve this problem?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/4/e4f4b6fd4674dfa2b125025359ecf0305d2adef6.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/e4f4b6fd4674dfa2b125025359ecf0305d2adef6" title="IMG_20230717_120431"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/4/e4f4b6fd4674dfa2b125025359ecf0305d2adef6_2_690x348.jpeg" alt="IMG_20230717_120431" data-base62-sha1="wFra3UxwC7JZQH6mVB3qckZKsnk" width="690" height="348" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/4/e4f4b6fd4674dfa2b125025359ecf0305d2adef6_2_690x348.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/e/4/e4f4b6fd4674dfa2b125025359ecf0305d2adef6_2_1035x522.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/e/4/e4f4b6fd4674dfa2b125025359ecf0305d2adef6_2_1380x696.jpeg 2x" data-dominant-color="30373E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">IMG_20230717_120431</span><span class="informations">1920×971 90.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>Your system might be unpacking the wheel, which might take some time. In case it’s really slow check if you might be running out of space and if the system is swapping memory.</p>

Gimyung: <p>Ok,thanks for the solution.i’ll wait a couple minutes then.</p>

v12hero: <p>did it work, I’m stuck at the same place for a few minutes</p>
Sharing a plugin to increase accuracy and/or decrease network size

PerforatedAI: <p>Hey all!</p>
<p>I’m startup founder and neuroscience PhD who has been working on a bio-inspired PyTorch plugin.  It  is based on a novel adjustment to the backpropagation algorithm that allows models to continue improving their accuracy after being fully trained.  In some cases this allows you to start with a smaller network and recover your original accuracy while maintaining a significantly reduced parameter count.  I currently have <a href="https://www.perforatedai.com/test-results" rel="noopener nofollow ugc">4 results</a> where I have pushed a state-of-the-art system with it, though for transparency it doesn’t work in all settings.  I’m looking for collaborators who would like to try it out on their systems to validate the results in additional projects.  You will have to run training again, but the actual coding adjustments will only take 5 minutes to an hour depending on the complexity of your network.  Please comment or DM me if you’d like to try it out!</p>
<p>Thanks!</p>
Stacking a list of tensors whose dimensions are unequal

HaziqRazali: <p>Hello, I have a simple problem where I am trying to stack a list of 2D tensors that have unequal number of rows. For example <code>data</code> is a list of 2D tensors and</p>
<p><code>data[0].size() = (15,2)</code><br>
<code>data[1].size() = (14,2)</code><br>
<code>data[2].size() = (7,2)</code></p>
<p>I tried using <code>torch.stack(data, dim=0)</code> and <code>torch.stack(data, dim=1)</code> but they both gave me an error. Basically what I am trying to achieve is to stack them in the first dimension, giving me</p>
<pre><code class="lang-auto">stacked_data.size() = (36,2)
</code></pre>

HaziqRazali: <p>Ok the solution was to simply use <code>torch.cat(data,dim=0)</code>. Doh.</p>

AusafMo: <p>a bit late, but is torch.cat really stacking ? as in final tensor being a tensor of individual tensors given each of them are same size,<br>
torch.cat will concatenate tensors resulting in a Single tensor with all the values of original tensors.<br>
stacking results = (3, 15, 2)<br>
concat  results = (36,2)<br>
am i missing something ?</p>
<p><strong>edit</strong> : only way that i found to <strong>Stack</strong> tensors of <em>unequal</em> sizes is padding,</p>
<pre><code class="lang-auto">  data = [
      torch.randn(15, 2),
      torch.randn(14, 2),
      torch.randn(7, 2)
  ]
  
  # maximum number of rows among the tensors
  max_rows = max(tensor.size(0) for tensor in data)
  
  # Padding to make sizes compatible
  padded_data = [torch.nn.functional.pad(tensor, (0, 0, 0, max_rows - tensor.size(0))) for tensor in data]
  
  # Stack padded tensors
  stacked_data = torch.stack(padded_data, dim=0)
  print(stacked_data.size()) 

&gt;&gt; torch.Size([3, 15, 2])

</code></pre>
I am getting 0 loss and accuracy in thousands

solar_system7: <p>here is my architecture:<br>
I am trying to learn transfer learning by using mobilenet pretrained model as encoder and custom decoder for segmentation problem, I have ISIC 2016 dataset.</p>
<p>the problem am facing is that my loss is always 0 and the accuracy came in thousands.</p>
<p>class SegmentationModel(torch.nn.Module):<br>
def <strong>init</strong>(self, dropout_prob=0.5):<br>
super(SegmentationModel, self).<strong>init</strong>()<br>
self.encoder = mobilenet_v2(weights=‘DEFAULT’, progress=True)<br>
self.encoder_features = list(self.encoder.features.children())<br>
self.encoder_features = self.encoder_features[:-1]</p>
<pre><code>    for params in self.encoder.parameters():
        params.requires_grad = False
    
    self.upconv1 = torch.nn.ConvTranspose2d(320, 160, kernel_size=2, stride=2)
    self.conv1 = torch.nn.Conv2d(256, 256, kernel_size=3, padding=1)
    self.bn1 = torch.nn.BatchNorm2d(256)  
    self.dropout1 = torch.nn.Dropout2d(dropout_prob)
    self.upconv2 = torch.nn.ConvTranspose2d(256, 96, kernel_size=2, stride=2)
    self.conv2 = torch.nn.Conv2d(128, 64, kernel_size=3, padding=1)
    self.bn2 = torch.nn.BatchNorm2d(64)  
    self.dropout2 = torch.nn.Dropout2d(dropout_prob)
    self.upconv3 = torch.nn.ConvTranspose2d(64, 32, kernel_size=2, stride=2)
    self.conv3 = torch.nn.Conv2d(56, 28, kernel_size=3, padding=1)
    self.bn3 = torch.nn.BatchNorm2d(28)  
    self.dropout3 = torch.nn.Dropout2d(dropout_prob)
    self.upconv4 = torch.nn.ConvTranspose2d(28, 16, kernel_size=2, stride=2)
    self.conv4 = torch.nn.Conv2d(32, 8, kernel_size=3, padding=1)
    self.bn4 = torch.nn.BatchNorm2d(8)  
    self.dropout4 = torch.nn.Dropout2d(dropout_prob)
    self.upconv5 = torch.nn.ConvTranspose2d(8, 1, kernel_size=2, stride=2)
    
    self._init_weights()
  
def _init_weights(self):
    for m in self.modules():
        if isinstance(m, torch.nn.Conv2d) or isinstance(m, torch.nn.ConvTranspose2d):
            torch.nn.init.kaiming_normal_(m.weight)
            if m.bias is not None:
                torch.nn.init.constant_(m.bias, 0)

def forward(self, x):
    outputs = []
    for layer in self.encoder_features:
        x = layer(x)
        outputs.append(x)
    
    x = self.upconv1(x)
    x = torch.cat([x, outputs[-5]], dim=1)
    x = torch.nn.functional.relu(self.conv1(x))
    x = self.bn1(x)  
    x = self.dropout1(x)
    
    x = self.upconv2(x)
    x = torch.cat([x, outputs[-12]], dim=1)
    x = torch.nn.functional.relu(self.conv2(x))
    x = self.bn2(x)  
    x = self.dropout2(x)
    
    x = self.upconv3(x)
    x = torch.cat([x, outputs[-15]], dim=1)
    x = torch.nn.functional.relu(self.conv3(x))
    x = self.bn3(x)  
    x = self.dropout3(x)
    
    x = self.upconv4(x)
    x = torch.cat([x, outputs[-17]], dim=1)
    x = torch.nn.functional.relu(self.conv4(x))
    x = self.bn4(x)  
    x = self.dropout4(x)
    
    x = self.upconv5(x)
    
    return x
</code></pre>
<p>is there any problem in this??</p>
<p>Thank you.</p>
Cuda 12.0 pytorch

yashika1: <p>I have ubuntu server containing python 3.10.3<br>
Cuda 12.0<br>
Nvidia driver 525.105.17<br>
Torch 2.2.1+cu121<br>
Torch.version.cuda 12.1<br>
Still getting runtime error: cuda error : opration not supported<br>
If anyone know solution kindly share</p>

ptrblck: <p>Could you post a minimal and executable code snippet reproducing the issue, please?</p>

yashika1: <p>[quote=“yashika1, post:1, topic:198803”]<br>
Cuda<br>
[/quote<br>
Thank you for respone,i am really stuck,I am running a detectron2 script to get detection  and there i m getting the error<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/b/5b925092ee1e4b04dd3b83944e00d92ee7ae5b48.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/5b925092ee1e4b04dd3b83944e00d92ee7ae5b48" title="17103358806523282537783790264184"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/b/5b925092ee1e4b04dd3b83944e00d92ee7ae5b48_2_690x388.jpeg" alt="17103358806523282537783790264184" data-base62-sha1="d44Vw4PZYBnMs9YHCZXbiag12as" width="690" height="388" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/b/5b925092ee1e4b04dd3b83944e00d92ee7ae5b48_2_690x388.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/5/b/5b925092ee1e4b04dd3b83944e00d92ee7ae5b48_2_1035x582.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/5/b/5b925092ee1e4b04dd3b83944e00d92ee7ae5b48_2_1380x776.jpeg 2x" data-dominant-color="9B9896"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">17103358806523282537783790264184</span><span class="informations">1920×1080 148 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>A screenshot unfortunately doesn’t allow us to copy/paste the code and execute it, so let me know if you can post the minimal and executable code snippet.</p>

yashika1: <p>Import torch<br>
device = torch.device(“cuda”)<br>
x=torch.randint(0,10,(3,3),dtype=torch.int.to(device)</p>

ptrblck: <p>This points to a broken installation and you might want to reinstall PyTorch, the NVIDIA driver, etc.</p>

yashika1: <p>Did reinstalled pytorch,but any other way than reinstalling nvidia driver</p>

Karls: <p>Are you sure you installed PyTorch compiled to use CUDA? Can you show us the output of “torch.cuda.is_available()” command?</p>
Forcing an object to be regarded as a Parameter

Max_Unhold: <p>Assume, I have created an object as is described in<br>
<a href="https://pytorch.org/docs/stable/notes/extending.html" rel="noopener nofollow ugc">Extending AutoGrad</a> - <strong>Extending <a href="https://pytorch.org/docs/stable/torch.html#module-torch" rel="noopener nofollow ugc"><code>torch</code></a> with a <code>Tensor</code> wrapper type</strong></p>
<p>I do <strong>not</strong> need any checks, I just need it to be passed to a custom AutogradFunction that calculates the backward (again without any checks of the type or other things)<br>
and be recognized by a custom optimizer as a parameter. The <code>nn.Parameter</code>does not work and <code>self.register_parameter</code> also requires parameters.</p>
<p>Is there any way, to brute-force write the tensor as a parameter such that it appears in the <code>model.parameter()</code> list and gets the update signal by the optimizer?</p>

JuanFMontesinos: <p>You don’t need a tensor to be a nn.parameter. You can pass any tensor to the optimizer and it will work.</p>
RuntimeError: CUDA error: device-side assert triggered\nCompile with `TORCH_USE_CUDA_DSA` to enable device-side assertions

Chieng_Frank: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/e/7ef045f46369c85535d66e896194865fb0f9877e.png" data-download-href="https://discuss.pytorch.org/uploads/default/7ef045f46369c85535d66e896194865fb0f9877e" title="1"><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/e/7ef045f46369c85535d66e896194865fb0f9877e.png" alt="1" data-base62-sha1="i6WWy84098o1k6KJm1CnyNWmNga" width="690" height="193" data-dominant-color="38414A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">1</span><span class="informations">1357×381 30.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>i run kohya lora trainining code and encounter CUDA index out of bounds problem,anyone can help me with this issue,thanks</p>

ptrblck: <p>Rerun the code with blocking launches and check the stacktrace to isolate the failing indexing operation. Once you know which op is failing, check the min./max. values of the index tensor as well as the shape of the tenors being indexed.</p>

Chieng_Frank: <p>thank you, after i rerun the code,the training process succeed in the end,btw,how to monitor checking the stacktrace andisolating the failing indexing operation?i have no idea how to breakpoint the beginning of this issue,i just curious if the latent bucketing previous of lora training cause this problem</p>

ptrblck: <p>Run the code via <code>CUDA_LAUNCH_BLOCKING=1 python script.py args</code> and check the stacktrace shown in your terminal to see which operation failed.</p>

Deeps: <p>Hi, the same issue happens while activating flash_attention in MBart. Any help here?</p>
Dynamic Graph Representation Puzzle

rijndael: <p>I’ve run into an interesting design problem in my application. I need to support graphs (in the machine learning context, i.e. with associated node/edge features) that have variable size (in terms of number of nodes/edges).</p>
<p>My <code>torch.nn.Module</code> has already been adapted to handle variable sized graphs, and works great, but now I’m turning to the data representation.</p>
<p>Specifically, to take the batch size 1 case, we start with an “empty” graph. At each step of a sampling loop, we either terminate sampling of the graph, or sample a new node type from a given vocabulary and identify which existing nodes it ought to be connected to, and then repeat the sampling loop, until we sample the termination token.</p>
<p>Independent of how I represent a graph, or a batch of graphs, my design objectives are:</p>
<ol>
<li>Ability to operate on <em>batches</em> of variably-sized graphs, not just one at a time.</li>
<li>The representation must be amenable to the addition of nodes/edges during the sampling procedure.</li>
<li>Throughput: I would like to minimize irregularity in how each graph in the batch is manipulated, perhaps at the cost of some masking logic.</li>
</ol>
<p>I’ve considered some fairly complicated solutions to this problem, but I’m curious if any data ninjas in the community might offer some advice. I imagine it’s fairly unusual (graphs are commonplace, but typically they have a known static size since they are given and not sampled), but would love any input.</p>
Incrementing tensor by values of another tensor using indices

jchwenger: <p>Hi there,</p>
<p>This might be relatively basic, and I’m sorry if there’s already a solution to this I could not find.</p>
<p>I’m trying to get rid of a for loop when gathering data in one tensor, and adding the sum of those values to another tensor (some of the rows in that last one will be updated multiple times, others may not be at all).</p>
<p>If you look at the toy example here:</p>
<pre data-code-wrap="python"><code class="lang-python">a = torch.zeros(2,2)
b = a.clone()
c = torch.tensor(
  [[1,1],
   [-1,-1]]
)

# only the last operation seems to have an effect
# try with one or the other to see the result change
# indc = [0,0,1]
indc = [1,0,0]

# sadly this does not work
a[indc] += c[indc]
print(a)

# works as epxected
for i,r in zip(indc, c[indc]):
  b[i] += r
print(b)
</code></pre>
<p>Is there a slick way of doing this that anyone knows of?</p>
<p>Many thanks in advance!</p>

ptrblck: <p><code>tensor.index_put_(..., accumulate=True)</code> should work:</p>
<pre data-code-wrap="python"><code class="lang-python">a = torch.zeros(2,2)
b = a.clone()
c = torch.tensor(
  [[1,1],
   [-1,-1]]
).float()

# only the last operation seems to have an effect
# try with one or the other to see the result change
# indc = [0,0,1]
indc = [1,0,0]

# sadly this does not work
a[indc] += c[indc]
print(a)
# tensor([[ 1.,  1.],
#         [-1., -1.]])

# works as epxected
for i,r in zip(indc, c[indc]):
  b[i] += r
print(b)
# tensor([[ 2.,  2.],
#         [-1., -1.]])

# also works
a = torch.zeros(2,2)
a.index_put_((torch.tensor(indc),), c[indc], accumulate=True)
print(a)
# tensor([[ 2.,  2.],
#         [-1., -1.]])
</code></pre>

jchwenger: <p>Torchtastic, <a class="mention" href="/u/ptrblck">@ptrblck</a>, thank you!</p>
<p>This was the last piece I needed to complete a fun little challenge found in Andrej Karpathy’s NN series, <a href="https://github.com/karpathy/nn-zero-to-hero/issues/46" rel="noopener nofollow ugc">I credit your help in this issue</a>.</p>
What does the term "peak memory" should be referring to, max allocated vs. reserved?

YimingDong: <p>I know that PyTorch uses a caching memory allocator to save the expensive cuda memory allocation calls, which in result produces two peak values, <code>max_memory_allocated</code> and <code>max_memory_reserved</code>, respectively. The allocated memory is the memory actually used by tensors, while the reserved memory also contains the cached ones made by pytorch memory allocator.<br>
My problem is that, <strong>when I want to know the peak memory stats, which number should I use?</strong> Moreover, which value determines the OOM cases? For instance, My deep learning task takes up to 9GB allocated memory and 11GB reserved on a 20GB GPU. Will I encounter OOM if I run this task on a 10GB GPU?</p>