akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Roadmap of docker version

GuWei007: <p><a href="https://hub.docker.com/r/pytorch/pytorch/tags" class="onebox" target="_blank" rel="noopener nofollow ugc">https://hub.docker.com/r/pytorch/pytorch/tags</a></p>
<p>The latest docker released is ubuntu20.04. Where can see the roadmap of docker version? When will ubuntu22.04 be released and why upgrade to ubuntu22.04 ？</p>
How can I convert detecron2 (.pth) custom trained model to onnx format or .pb format(tflite) for robot deployment

Kamal_Moha: <p>I am using detectron2 model for instant segmentation &amp; object detection .My goal is to deploy this model onto a robot using <a href="https://www.viam.com/" rel="noopener nofollow ugc">Viam robotics</a> and they(Viam) only accept a .tflite model when uploading.</p>
<p>I need a script (written in Python) that would take a trained PyTorch model file (.pth extension) and export it to TensorFlow format (.pb). [frozen graph] or .onnx so that I can be able to upload it into the robot</p>
<p>The model was trained using the Facebook’s DETECTRON2 (the pre-trained model was “COCO-InstanceSegmentation/mask_rcnn_X_101_32x8d_FPN_3x.yaml”)</p>
<p>I already have the output “pth” file. The script should take this file as a parameter and return single ‘.pb’ file.</p>
<p>I checked this <a href="https://github.com/tensorpack/tensorpack/tree/master/examples/FasterRCNN/convert_d2" rel="noopener nofollow ugc">doc</a> doing tensorflow conversions and it uses the below code for conversion;</p>
<pre><code class="lang-auto"># 1. Download the corresponding model from detectron2 model zoo
# 2. Convert:

$ python convert_d2.py --d2-config detectron2/configs/COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x.yaml --d2-pkl model_final_f10217.pkl --output R50FPN-d2-converted.npz
# the script will print tensorpack configs
'MODE_MASK=True' 'MODE_FPN=True' 'BACKBONE.STRIDE_1X1=True' 'PREPROC.PIXEL_MEAN=[123.675,116.28,103.53]' 'PREPROC.PIXEL_STD=[1.0,1.0,1.0]'

# 3. Use the above configs to verify the conversion is correct:
$ ./predict.py --evaluate out.json --load R50FPN-d2-converted.npz  --config DATA.BASEDIR=~/data/coco 'MODE_MASK=True' 'MODE_FPN=True' 'BACKBONE.STRIDE_1X1=True' 'PREPROC.PIXEL_MEAN=[123.675,116.28,103.53]' 'PREPROC.PIXEL_STD=[1.0,1.0,1.0]'

# 4. Naively convert the model to a frozen pb file:
$ ./predict.py --output-pb out.pb --load R50FPN-d2-converted.npz  --config DATA.BASEDIR=~/data/coco 'MODE_MASK=True' 'MODE_FPN=True' 'BACKBONE.STRIDE_1X1=True' 'PREPROC.PIXEL_MEAN=[123.675,116.28,103.53]' 'PREPROC.PIXEL_STD=[1.0,1.0,1.0]'
</code></pre>
<p>But I would like to know what <code>--d2-pkl</code> stands for and how can I use my custom trained model(‘.pth’) for this conversion. I would like to get your help on this, please.</p>
Changing backbone of MaskRCNN to ResNet152 does not train

Remco_S: <p>Hi, I’m doing a backbone comparison for comparing smaller backbones and their results on a dataset, however, when changing the backbone I’m probably doing something wrong, since this results in the network not being trained and having a weird output compared to the standard MaskRCNN with the ResNet50 output. Below is the code of what I’m doing, my images are 400x400, num_classes=9 and if there is a comprehensive guide on swapping backbones please point me to it.</p>
<pre><code class="lang-auto">      resnet152_backbone = torch.load('path/to/resnet152')
      model1 = torchvision.models.resnet152(pretrained=False)
      model1.load_state_dict(resnet152_backbone)
      modules = list(model1.children())[:-1]
      backbone = torch.nn.Sequential(*modules)
      backbone.out_channels = 2048

      # Make rpn
      anchor_generator = AnchorGenerator(
          sizes=((32,64,128,256,512),),
          aspect_ratios=((0.5,1.0,2.0),)
      )

      roi_pooler = torchvision.ops.MultiScaleRoIAlign(
          featmap_names=['0'],
          output_size=7,
          sampling_ratio=2,
      )
      mask_roi_pooler = torchvision.ops.MultiScaleRoIAlign(featmap_names=['0'],
                                                            output_size=14,
                                                            sampling_ratio=2)

      # put the pieces together inside a Faster-RCNN model
      model = MaskRCNN(
          backbone,
          num_classes=num_classes,
          rpn_anchor_generator=anchor_generator,
          box_roi_pool=roi_pooler,
           mask_roi_pool=mask_roi_pooler,
      )
</code></pre>
Android Error with D2Go FPN Model: Mismatch in Output Channel Size

Rachel_Bergman: <p>I recently encountered an issue while deploying a D2Go FPN model on an Android application. Initially, I had a model trained and tested with one class using the “faster_rcnn_fbnetv3a_C4.yaml” configuration, and it functioned flawlessly on Android. This model was configured with pre-loaded weights from the “faster_rcnn_fbnetv3a_C4.yaml” checkpoint and the configuration was merged appropriately.</p>
<p>Due to the need to detect smaller objects, I switched to a D2Go FPN model (both model and config were updated accordingly). However, when attempting to run this new model on Android, I consistently encounter a fatal exception error. The error message indicates an issue related to the mismatch in the output channel size of weight and bias. Below is the error log for reference:<br>
FATAL EXCEPTION: Thread-2<br>
Process: org.pytorch.demo.objectdetection, PID: 4275<br>
com.facebook.jni.CppException: Output channel size of weight and bias must match.</p>
<pre><code>                                                                                                  Debug info for handle(s): debug_handles:{-1}, was not found.
                                                                                                  
                                                                                                Exception raised from apply_impl at /home/agunapal/pytorch/aten/src/ATen/native/quantized/cpu/qconv.cpp:899 (most recent call first):
                                                                                                (no backtrace available)
                                                                                                	at org.pytorch.LiteNativePeer.forward(Native Method)
                                                                                                	at org.pytorch.Module.forward(Module.java:52)
                                                                                                	at org.pytorch.demo.objectdetection.MainActivity.run(MainActivity.java:332)
</code></pre>
<p>I am seeking suggestions or solutions on how to address this issue. Any insights into what might be causing this error and how to resolve it would be greatly appreciated.</p>
Constant val_loss in TimeSeriesDataSet using categorical_encoders

Alex_Pearson: <p>Hello everyone, I’m encountering a peculiar issue with my TimeSeriesDataSet in PyTorch Forecasting. Normally, my model trains well without categorical_encoders, showing a reduction in validation loss from around 9.5 to 3.3 after 100 epochs (this is for chunk 1 for example sake). However, when I introduce categorical_encoders for my group_ids, the validation loss frustratingly sticks at 36.67 for all epochs. This is a major concern as I need these encoders to handle ‘cold start’ issues in real-world data. Here’s a snippet of my implementation for reference:</p>
<p>chunk_size = 200000<br>
total_rows = 184902668<br>
while start_row &lt; total_rows:<br>
pl.seed_everything(415)<br>
max_prediction_length = 36<br>
min_encoder_length = 36<br>
max_encoder_length = 1240<br>
training_cutoff = data[“time_idx”].max() - max_prediction_length<br>
training = TimeSeriesDataSet(<br>
data[lambda x: x.time_idx &lt;= training_cutoff],<br>
group_ids=[“cat_col_1”, “cat_col_2”, “cat_col_3”, “cat_col_4”],<br>
categorical_encoders={<br>
‘cat_col_1’: pytorch_forecasting.data.encoders.NaNLabelEncoder(add_nan=True),<br>
‘cat_col_2’: pytorch_forecasting.data.encoders.NaNLabelEncoder(add_nan=True),<br>
‘cat_col_3’: pytorch_forecasting.data.encoders.NaNLabelEncoder(add_nan=True),<br>
‘cat_col_4’: pytorch_forecasting.data.encoders.NaNLabelEncoder(add_nan=True),<br>
},<br>
)</p>
<p>Has anyone faced a similar challenge or have insights on what might be causing this constant validation loss? Any advice or solutions would be greatly appreciated!</p>
Pytorch Mobile Benchmarking Binary

hello-fri-end: <p>According to the <a href="https://pytorch.org/tutorials/recipes/mobile_perf.html#android-benchmarking-setup" rel="noopener nofollow ugc">documentation</a>, the mobile benchmarking binary only runs on the mobile cpu and doesn’t using hardware accelerators like DSPs and GPUs. Is there a way of benchmarking pytorch models on a mobile phone which is able to leverage these hardware accelerators?</p>
<p>Also, another question, I’ve been successfully able to run the <a href="https://sites.google.com/view/repnet" rel="noopener nofollow ugc">RepNet</a> model on my phone using this binary but as I understand <code>qnnpack</code> and <code>xnnpack</code> do not support Conv3D or attention layers, is there a fallback in the pytorch mobile runtime that is used when the model contains unsupported layers because the model runs without any issues?</p>
Torch Dataset setting tensor to 0

Raspirat: <p>I have the following Dataset:</p>
<pre><code class="lang-auto">class MidiDataset(torch.utils.data.Dataset):
    def __init__(self, ndata, tokenizer, batch_size, block_size):
        self.ndata = ndata
        self.tokenizer = tokenizer
        self.ndata_enc = tokenizer.encode(ndata.type(torch.float32), device)
        print("-------NDATAENC0-----")
        print(self.ndata_enc[0])
        self.batch_size = batch_size
        self.block_size = block_size

    def __len__(self):
        return len(self.ndata_enc)

    def __getitem__(self, index):
        print("-------GETITEM_NDATAENC-----")
        print(self.ndata_enc)
        return self.ndata_enc[index]
</code></pre>
<p>The <strong>init</strong> works just fine and outputs</p>
<p>-------NDATAENC0-----<br>
tensor([4752., 4752., 4779., 4779., 4807., 4817., 4831., 4846., 4855., 4872.,<br>
4884., 4884., 4912., 4912., 4912., 4912., 4884., 4985., 4998., 4998.,<br>
5025., 5037., 5057., 5057., 5082., 5082., 5110., 5110., 5057., 5057.,<br>
5057., 5057., 5200., 5213., 5229., 5244., 5261., 5274., 5291., 4872.,<br>
4831., 5343., 4884., 5382., 5401., 5401., 5401., 5401., 4752., 4752.,<br>
5510., 5510., 4779., 4779.,   26.,   37.,   49.,   49., 4884., 5382.,<br>
5401., 5401., 5401., 5401.], device=‘cuda:0’)</p>
<p>as it should be. But in the <strong>getitem</strong> method every value in self.ndata_enc is set to 0:</p>
<p>------GETITEM_NDATAENC-----<br>
tensor([[0., 0., 0.,  …, 0., 0., 0.],<br>
[0., 0., 0.,  …, 0., 0., 0.],<br>
[0., 0., 0.,  …, 0., 0., 0.],<br>
…,<br>
[0., 0., 0.,  …, 0., 0., 0.],<br>
[0., 0., 0.,  …, 0., 0., 0.],<br>
[0., 0., 0.,  …, 0., 0., 0.]], device=‘cuda:0’)</p>
<p>Does anyone know why this is the case and how to fix it?<br>
Thank you in advance. Any help is appreciated!</p>

Raspirat: <p>I found out that this only happens on device cuda. Why could this be?</p>
How to adjust Linear Layers to Shapes of Adjacency Matrices?

vietngth: <p>Hi,</p>
<p>My task is to predict a label matrix that has the shape of the adjacency matrix, where each entry is predicted by computing a pair-wise dot product of node embeddings. I’m using <code>torch_geometric</code> library for such purpose. The problem is, I don’t know how to set the appropriate Linear output dimensions for the prediction task:</p>
<pre><code class="lang-auto">class PMIPredictor(pl.LightningModule):
    def __init__(self, **kargs):
        super().__init__()
        ...

        self.encoder = GSequential("x, edge_index, edge_attr", [
            (GATConv(feat_size, emb_size, heads=heads, edge_dim=edge_features),
             "x, edge_index, edge_attr -&gt; x"),
            ReLU(),
            (GATConv(emb_size * heads, emb_size, heads=heads, edge_dim=edge_features),
             "x, edge_index, edge_attr -&gt; x"),
            ReLU(),
            (GATConv(emb_size * heads, latent_size, heads=1, edge_dim=edge_features),
             "x, edge_index, edge_attr -&gt; x"),
            ReLU()
        ])

        self.predictor = Sequential(
            Linear(latent_size * 2, decoder_size),
            ReLU(),
            # What is the appropriate size here?
            Linear(decoder_size, ?)
        )

    def encode(self, x, edge_attr, edge_index):
        return self.encoder(x, edge_attr, edge_index)

    def predict(self, z, label, batch_vector):
        z, mask = to_dense_batch(z, batch_vector)
        z_t = z.transpose(1, 2)
        # pair-wise node embeddings of size [batch, max_node, max_node], max_node is taken from
        # biggest graph of the batch!
        pairwise_emb = z @ z_t
        # I want to keep the size [batch, max_node, max_node] here
        return logits, mask

    def training_step(self, batch, batch_index):
        z = self.encode(batch.x,  batch.edge_attr, batch.edge_index)
        # labels are already of size [batch, max_node, max_node]
        labels = batch.y
        logits, masks = self.predict(z, batch.batch)
        loss = self.bce(logits[mask], labels[mask])
        ...
</code></pre>
<p>How could I keep the size <code>[batch, max_node, max_node]</code> with an appropriate size of the final Linear layer? Thank you.</p>
Can you do backward propagation with nearest neighbor interpolation?

Lelouch1: <p>Hi,</p>
<p>I saw some pytorch models were using nearest neighbor interpolation in their upsampling layers (in convolutional decoders for instance). But to my knowledge, the nearest interpolation is not differentiable. Shouldnt the model not being able to learn anything because of that ?</p>
<p>Or does the backward propagation works similarly to a pooling layer in that case ?</p>

KFrank: <p>Hi Lelouch!</p>
<aside class="quote no-group" data-username="Lelouch1" data-post="1" data-topic="193421" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/lelouch1/48/38438_2.png" class="avatar"> Lelouch1:</div>
<blockquote>
<p>nearest interpolation is not differentiable.</p>
</blockquote>
</aside>
<p>Nearest-neighbor interpolation is differentiable and you can backpropagate<br>
through it as shown in this example:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'2.1.1'
&gt;&gt;&gt; s = torch.arange (4.0).reshape (1, 1, 4).requires_grad_()
&gt;&gt;&gt; s
tensor([[[0., 1., 2., 3.]]], requires_grad=True)
&gt;&gt;&gt; t = torch.nn.functional.interpolate (s, size = 8, mode = 'nearest')
&gt;&gt;&gt; t
tensor([[[0., 0., 1., 1., 2., 2., 3., 3.]]],
       grad_fn=&lt;UpsampleNearest1DBackward0&gt;)
&gt;&gt;&gt; t.sum().backward()
&gt;&gt;&gt; s.grad
tensor([[[2., 2., 2., 2.]]])
</code></pre>
<aside class="quote no-group">
<blockquote>
<p>Or does the backward propagation works similarly to a pooling layer in that case ?</p>
</blockquote>
</aside>
<p>I’m not quite sure what you are saying here, but I think that this description<br>
is correct.</p>
<p>Best.</p>
<p>K. Frank</p>
Is Noam scheduling widely used for training transformer-based models?

MyungHa_Kwon: <p>As in title, I wonder how widely it is used.<br>
Sometimes, it seems to have big impact on training transformer-based models.<br>
But I can’t find it implemented in torch, huggingface transformers, tensorflow.<br>
It’s only implemented in allennlp or opennmt.<br>
It’s little bit wierd to me.</p>

wen_he: <p>I am also interested in this. It seems that by googling, few papers use this scheduling policy.</p>

hiyyg: <p>Do you mean this <a href="https://fairseq.readthedocs.io/en/latest/_modules/fairseq/optim/lr_scheduler/inverse_square_root_schedule.html" class="inline-onebox" rel="noopener nofollow ugc">fairseq.optim.lr_scheduler.inverse_square_root_schedule — fairseq 0.12.2 documentation</a></p>
Libtorch 2.1.1 crashing when predicting from DLL

DejNea: <p>I’m using Libtorch 2.1.1 with CUDA 11.8 and GPU NVIDIA RTX A2000 12GB. To build code I’m using cmake with Visual studio code.</p>
<p>When I build app in debug or release version in Visual studio it works fine, but when I put the same code in DLL to use it in C# or any other language it crashes in the middle of prediction. I also use try catch to return error so it wouldn’t crash, but it still freeze the program and crashes. I’m using same DLL on different PCs with different GPUs and there the DLL works fine. Right now I’m out of ides on what to do. Models are used for image segmentation. I also used different models if the size of them would be problem, but still the same error.</p>
<p>In C# works only in Release.</p>
<p>Is it possible that the problem is GPU?</p>

DejNea: <p>I fixed the problem.<br>
In NVIDIA control panel I:</p>
<ul>
<li>Enabled ECC state for the GPU</li>
<li>Set power management mode to Prefer maximum performance</li>
<li>Set CUDA – Sysmem Fallback Policy to Prefer Sysmem Fallback</li>
<li>Set Adjust image settings with preview to Use the advanced 3D image settings</li>
<li>For video output (LCD) I used internal (Intel) GPU output.</li>
</ul>
Floating point precision differences when using torch.matmul on matrix transpose

ssakpal: <p>Hi,</p>
<p>Can someone please explain why there might be differences in floating point precision for matrix multiplication when using the transpose of a matrix vs not using transpose.</p>
<p>For example:</p>
<pre><code class="lang-auto">A = torch.tensor([[11., 25.,  3.,  5., 15.,  6., 12., 80.,  1.]], dtype=torch.float32)
B = torch.tensor([[-0.204977914690971375,  0.077419161796569824,  0.189963847398757935,
        -0.241283535957336426, -0.303015649318695068,  0.186268478631973267,
        -0.329424560070037842, -0.248794123530387878,  0.144485712051391602],
      [-0.111751005053520203,  0.235284119844436646, -0.072023868560791016,
        0.053652375936508179, -0.184143990278244019,  0.065914064645767212,
        -0.307873427867889404, -0.253490984439849854, -0.256091356277465820]], dtype=torch.float32)
C = torch.tensor([[-0.204977914690971375, -0.111751005053520203],
    [ 0.077419161796569824,  0.235284119844436646],
    [ 0.189963847398757935, -0.072023868560791016],
    [-0.241283535957336426,  0.053652375936508179],
    [-0.303015649318695068, -0.184143990278244019],
    [ 0.186268478631973267,  0.065914064645767212],
    [-0.329424560070037842, -0.307873427867889404],
    [-0.248794123530387878, -0.253490984439849854],
    [ 0.144485712051391602, -0.256091356277465820]], dtype=torch.float32)

print(torch.matmul(A, B.T))
print(torch.matmul(A, C))
</code></pre>
<p>Result:</p>
<pre><code class="lang-auto">tensor([[-28.095565795898, -21.891492843628]])
tensor([[-28.095567703247, -21.891494750977]])
</code></pre>
<p>Here, B and C contain the same values, they are just different dimensions. We need to take the transpose of B in order to multiply with A, whereas we don’t need to for C.</p>
<p>I’m testing this out on Google Colab, and have tried both CPU and GPU.<br>
Thanks!</p>

ptrblck: <p>Maybe different kernels are used based on the layout and you could profile these with e.g. Nsight Systems or the native profiler.</p>

malfet: <p>Because floating point operations results depends on the operands order, i.e. floating point addition is strictly speaking non-associative:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; x = (0.1 + 0.2) + 0.3
&gt;&gt;&gt; y = 0.1 + (0.2 + 0.3)
&gt;&gt;&gt; x-y
1.1102230246251565e-16
</code></pre>
<p><code>torch.matmul</code> can perform the operations in different order depending on the inputs layout, which leads to a some discrepancies.</p>
How to calculate running_loss value

jcgarciaca: <p>I don’t understand how to calculate the <code>running_loss</code> value when training a model.</p>
<p>In <a href="https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html" rel="nofollow noopener">Training a classifier tutorial</a> when training, the <code>running_loss</code> is added with <code>loss.item()</code> and set to 0.0 every 2000 mini-batches as follow:</p>
<pre><code class="lang-auto">running_loss += loss.item()
if i % 2000 == 1999:    # print every 2000 mini-batches
	print('[%d, %5d] loss: %.3f' %
		(epoch + 1, i + 1, running_loss / 2000))
	running_loss = 0.0
</code></pre>
<p>But, in <a href="https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html" rel="nofollow noopener">Transfer learning tutorial</a> the <code>running_loss</code> is added with <code>loss.item() * inputs.size(0)</code> and it is not set to 0.0:</p>
<pre><code class="lang-auto">	...
	# statistics
	running_loss += loss.item() * inputs.size(0)
	running_corrects += torch.sum(preds == labels.data)
if phase == 'train':
	scheduler.step()

epoch_loss = running_loss / dataset_sizes[phase]
</code></pre>
<p>What is the difference?</p>

ptrblck: <p>It depends how you would like to print the running loss value.</p>
<p>In the first example the current loss average of the batch will be added to <code>running_loss</code>, printed after 2000 iterations and reset.</p>
<p>The second example prints the loss for the complete epoch and is a bit more accurate.<br>
Depending on the batch size and the length of your dataset, the last batch might be smaller than the specified batch size. If that’s the case, you could multiply the loss by the current batch size (<code>loss.item() * inputs.size(0)</code>) and divide by the total length of the dataset <code>running_loss / dataset_sizes[phase]</code>.</p>

jcgarciaca: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> thank you for your answer</p>

Murdock135: <p>Why 2000 iterations in particular?</p>
.pth model Usage

JIMUEL_SERVANDIL: <p>Hello. I was fine tuning <code>LayoutlmV3</code> and after the training, It has generated a <code>.pth</code> file. How can I implement this<code> .pth</code> model to predict the labels of the text in an image?</p>

ptrblck: <p>Could you describe how exactly this file was generated? I.e. does it contain a <code>state_dict</code> or did you save the model directly?</p>
nn.Parameter() not learning

promach: <p>Why is <a href="https://gist.github.com/buttercutter/b3331ca1fd9e2f5871b0eded6b758f39/d97190635db23f75ec66b17145a53914de0ee1aa#file-mamba-py-L74" rel="noopener nofollow ugc">self.A = nn.Parameter(F.normalize(torch.randn(d_model, state_size), p=2, dim=-1))</a> not learning ?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/1/51c5db09ddd71cce98dd7c9cc7d8420e694e514a.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/51c5db09ddd71cce98dd7c9cc7d8420e694e514a" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/1/51c5db09ddd71cce98dd7c9cc7d8420e694e514a_2_675x500.jpeg" alt="image" data-base62-sha1="bFozSXZkyHs8YhDDyDNzVnxitmy" width="675" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/1/51c5db09ddd71cce98dd7c9cc7d8420e694e514a_2_675x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/5/1/51c5db09ddd71cce98dd7c9cc7d8420e694e514a_2_1012x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/1/51c5db09ddd71cce98dd7c9cc7d8420e694e514a.jpeg 2x" data-dominant-color="040404"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1274×943 147 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Pytorchaudio problem in windows

zeynep.eee: <p>hello, I am a windows user and I want to use the pytrochaudio library but<br>
pycharm gives this error(FileNotFoundError: Could not find module ‘C:\Users\Lenovo\PycharmProjects\project_pytorch\venv\pythonProject36\venv\Lib\site-packages\torchaudio\lib\libtorchaudio.pyd’ . Try using the full path with constructor syntax.). What do you think could be the reason for this and how can ı fixed ? Thank you in advance.</p>
Torch.save: RuntimeError: Parent directory does not exist

Sam28: <p>Hi, I’m trying to save my trained model on Google Drive via Google Colab but I get this error although the directory exists in my Google drive:</p>
<blockquote>
<p>/content/drive/MyDrive/spawn-main</p>
<p>MMNet Training Script</p>
<p>2023-12-10 18:12:47.276362: E tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:9342] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered<br>
2023-12-10 18:12:47.276416: E tensorflow/compiler/xla/stream_executor/cuda/cuda_fft.cc:609] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered<br>
2023-12-10 18:12:47.276453: E tensorflow/compiler/xla/stream_executor/cuda/cuda_blas.cc:1518] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered<br>
2023-12-10 18:12:47.284903: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.<br>
To enable the following instructions: AVX2 AVX512F FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.<br>
2023-12-10 18:12:48.335569: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT<br>
Training model R5_MMNet_SUPERV-NYU-depth+rgb+normals_da<br>
train: Epoch 1/80 Loss: 1.9362 MIoU: 0.1180: 100% 199/199 [04:19&lt;00:00,  1.30s/it]<br>
valid: Epoch 1/80 Loss: 1.7581 MIoU: 0.1275: 100% 164/164 [04:16&lt;00:00,  1.57s/it]<br>
mIoU improved from 0.00000 to  0.12751<br>
Traceback (most recent call last):<br>
File “/content/drive/MyDrive/spawn-main/spawn_train.py”, line 262, in <br>
main()<br>
File “/content/drive/MyDrive/spawn-main/spawn_train.py”, line 258, in main<br>
train()<br>
File “/content/drive/MyDrive/spawn-main/spawn_train.py”, line 250, in train<br>
model = train_model(model, dev, dataloaders, unsup_dataloader, criterion, opt,<br>
File “/content/drive/MyDrive/spawn-main/utils/semi_sup_train.py”, line 304, in train_model<br>
torch.save(model.state_dict(), “/content/drive/MyDrive/spawn-main/weights/{}<em>EPOCH</em>{}”.format(model_name, epoch))<br>
File “/usr/local/lib/python3.10/dist-packages/torch/serialization.py”, line 618, in save<br>
with _open_zipfile_writer(f) as opened_zipfile:<br>
File “/usr/local/lib/python3.10/dist-packages/torch/serialization.py”, line 492, in _open_zipfile_writer<br>
return container(name_or_buffer)<br>
File “/usr/local/lib/python3.10/dist-packages/torch/serialization.py”, line 463, in <strong>init</strong><br>
super().<strong>init</strong>(torch._C.PyTorchFileWriter(self.name))<br>
RuntimeError: Parent directory /content/drive/MyDrive/spawn-main/weights does not exist.</p>
</blockquote>
<p>I’d appreciate it if you could help me.</p>
Label Change not working

Torcione: <p>Hi everyone,<br>
I have a binary dataset with labels 0, 1.<br>
I want to modify them in order to use the hinge loss (therefore I need to map them into -1, 1)<br>
To do so I followed a strategy similar to the one suggested <a href="https://discuss.pytorch.org/t/change-labels-in-data-loader/36823">here</a>; In particular I proceed as follows:</p>
<pre><code class="lang-auto">print('targets before', dict(Counter(train_ds.targets)))
if Loss_function=='Hinge':
    label_map ={-1:0, 1:1}
    for key in label_map:
        train_ds.targets[train_ds.targets==label_map[key]]= key
        valid_ds.targets[train_ds.targets==label_map[key]]= key
        
    print('targets after', dict(Counter(train_ds.targets)))

</code></pre>
<p>In the code above I define a mapping dict and use it to modify the targets; this, however does not work.<br>
Indeed from the print I always got (before and after the block) the same output, i.e.:</p>
<pre><code class="lang-auto">target before {1:10000, 0:10000}
target after {1:10000, 0:10000}
</code></pre>
<p>I had the suspect that the problem was due to incompatibility between the data type of targets and the one used in the dict; however the problem persist even using <code>label_map ={int(-1):int(0), int(1):int(1)}</code></p>
<p>Do you have any idea?</p>
<p><strong>Edit</strong></p>
<p>I realized that the problem was in the fact that <code>train_ds.targets</code> was a list of labels, not torch tensor.<br>
Converting the list list to a torch tensor solve the problem, i.e. the output of the print become:</p>
<pre><code class="lang-auto">target before {1:10000, 0:10000}
target after {1:10000, -1:10000}
</code></pre>
<p>The doubt that I have now is:<br>
the change in the labels do not reflect in the training loader (defined before the class mapping); why is this the case?</p>
<p>Even redefining the dataloader after the targets change do not solve the problem</p>
Is it possible to ignore masked part of sequences when creating an encoder?

Yuerno: <p>Hello! I am attempting to make an observation history encoder, where my goal is for a model that takes in as input a variable length sequence of dimension [Time, Batch, Features] (where sequences are padded to fit a fixed Time length) and outputs a variable of dimension [Batch, New_Features]. My concern is that when I’m doing dimensionality reduction with FC layers, they will take the padded data into consideration. Is there any way to avoid this?</p>
Essentially identical training time on CPU/MPS. Unconvinced I'm using MPS at all

kbwiley: <p>I realize there are ongoing explorations of whether MPS offers much speed up or even if it is slower than CPU in some circumstances. My question isn’t about that. My concern is that I get nearly identical training times (and nearly identical loss/accuracy curves) on CPU and MPS. And yet, if I print out the model device and the batch device I correctly get CPU for a CPU run and MPS for an MPS run.</p>
<p>I’m currently running on a 2020 M1 Mac Mini, which admittedly only has 8 GPU cores to compete with its 8 CPU cores. I will soon have access to an M1 Max Studio with 10 CPU and 24 GPU cores and perhaps I will get different results on that machine. But for the time being, I’m getting this strange result on an M1 Mac Mini: not just a GPU that underperforms the CPU, but which actually trains almost identically to the CPU such that I’m not confident I’m actually using the GPU at all, despite the fact that printing the device of the model and data confirms that I have moved to the MPS successfully.</p>
<p>Any thoughts?</p>
Using conda cuda package

dshin: <p>For my project, I downloaded cxx11-abi libtorch <a href="https://pytorch.org/get-started/locally/" rel="noopener nofollow ugc">here</a>, and then pointed my CMakeLists.txt to the installed location via,</p>
<pre><code class="lang-auto">find_package(Torch REQUIRED PATHS ${LIBTORCH_PATH})
</code></pre>
<p>This call, under its hood, appears to find my system cuda installation, and everything works.</p>
<p>To increase portability, I want to change the project to use the cuda package that is available on conda.</p>
<p>How do I get the above cmake command to find my conda cuda package rather than my system cuda installation?</p>
Activations during FSDP

vanshil_shah: <p>How are the activations stored when we use FSDP ?<br>
So in a multiple GPU setup every gpu has its own shard of the FSDP unit so during the forward pass,as we do all gather for the parameters of a FSDP unit and calculate the result? where are the results stored are the results stored in the GPU for the forward pass of every FSDP unit as these would be required again during backward passs? And the activations are not sharded right ?</p>
EmbeddingBag vs Padding

skumyol: <p>Hi,</p>
<p>I have a dilemma, I’m building a sentence classifier using two different models FFNN and LSTM. I used EmbeddingBag with offsets with FFNN and Embedding/pack padded sequence with RNN. I have questions:</p>
<ol>
<li>How could I add GloVe like embeddings using EmbeddingBag for randomly initialized embeddings?</li>
<li>Is it really necessary to use pack padded sequence to an RNN model, if I use EmbeddingBag, the offsets does the same job as the output of the EmbeddingBag is more uniform?</li>
</ol>

ChristLBUPT: <p>First, if you want to specify the embedding weights of embedding bags, you can use the following code</p>
<pre data-code-wrap="python"><code class="lang-python">eb = torch.nn.EmbeddingBag(glove_vocabulary_size, glove_embedding_dim)
eb.weight.data = glove_embeding_tensor # the embedding tensor loaded from glove embedding files
</code></pre>
<p>Second, passing a sequence to an RNN model may increase your classifier’s performance (especially when you have sufficient training data) compared to simply reducing the embedding tensors (like EmbeddingBag), since it involves more trainable parameters.</p>

vdw: <p>FFNN and LSTM are very different architectures.</p>
<p>The <code>nn.EmbeddingBag</code> gives you some aggregate (<code>sum</code>, <code>mean</code>, or <code>max</code>) over all relevant word embeddings – that is, all embeddings for the words contained in your sentence. This means that this aggregated embedding vector <strong>does no longer capture any sequence information</strong>. It’s therefore no longer a meaningful input for a LSTM/GRU.</p>
<p>An LSTM/GRU expects a <strong>sequence of vectors</strong> – each vector is an embedding for an individual word/token in your sentences – while <code>nn.EmbeddingBag</code> gives you a <strong>single vector</strong> for a whole sentence. Of course, this allows you to use this vector as input for an FFNN since those cannot handle sequences.</p>
<p>More specifically, using <code>nn.EmbeddingBag</code> will give you the same embedding vector for the following two sentences:</p>
<ul>
<li><em>“the movie was funny but not great”</em></li>
<li><em>“the movie was not funny but great”</em></li>
</ul>
<p>Both sentences contain the same words just in a different order.</p>
<p>If you want to consider word order, you need to use LSTMs/GRUs/RNNs. And in this case, you need to handle sentences/sequences of different lengths, at least as long as you want to use batches of size larger than 1. Padding is one way to do this.</p>
<p>In short, <code>nn.EmbeddingBag</code> and <code>nn.LSTM</code> do not meaningfully mix.</p>
What does it mean to move a loss function to device (GPU)?

John_Deterious: <p><code>loss = nn.CrossEntropyLoss().to(device)</code></p>
<p>It makes sense to me to move a tensor, a NN, but not a function<br>
Finally, what if I made my own loss, why doesn’t it have the method .to()  ?</p>

Mazhar_Shaikh: <p>In the context of nn Modules, the “to” method is explained in the documentation <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.Module.to" rel="nofollow noopener">here</a>.<br>
The move happens to only parameters and buffers. Hence, moving a loss function like CrossEntropy to GPU doesn’t change anything. In a custom Loss function made subclassing nn.Module, the “.to()” will be inherited and will move any parameters/buffers to the gpu.<br>
The custom loss function you made may not have subclassed nn.Module and hence, did not have the “.to()”  method.</p>

cramraj8: <p>Which one is faster implementation ?<br>
Having criterion = nn.CrossEntropyLoss()</p>
<p>but calling loss using predications (gpu tensor) and target (gpu tensor),</p>
<ol>
<li>criterion(predications, target)</li>
<li>criterion(predications.cpu(), target.cpu())</li>
</ol>
<p>Which of the above 2 is faster or both perform the same speed ?</p>

ptrblck: <p>If the model predictions and targets are already on the GPU, I would keep them there, since the <code>cpu()</code> operation would synchronize the code, and might lower the performance.</p>

Vedant_Gannu: <p>If I was implementing my own loss function, does it make sense to cast all of my intermediate tensors to the GPU? It would make sense to me, but I’m not totally sure what CrossEntropyLoss does. Does CrossEntropyLoss do all of it’s work on the GPU? Or does it do all calculations on the CPU?</p>

ptrblck: <p>It depends on the inputs you are passing to this loss function. I.e. if the model output and targets are on the GPU, the computation will also be performed on the GPU. If your custom loss function has internal states stored as tensors you should move it to the same device before calculating the loss. If it’s stateless you can just pass the inputs to it to calculate the loss.</p>
TypeError: expected Tensor as element 0 in argument 0, but got tuple

peony: <p>I have the following function</p>
<pre><code class="lang-auto">def stack_features(features, something):
       return torch.stack([torch.stack(ft, dim=0) for ft in features], dim=0):
</code></pre>
<p>but I get this error:</p>
<pre><code class="lang-auto">line 100, in stack_features
    return torch.stack([torch.stack(pad_features(ft), dim=0) for ft in features], dim=0)
RuntimeError: stack expects each tensor to be equal size, but got [1024, 7] at entry 0 and [1024, 3] at entry 1
</code></pre>
<p>So I tried to solve it by making a collate function:</p>
<pre><code class="lang-auto">    def collate_fn(ft):
        return tuple(zip(*ft))
</code></pre>
<pre><code class="lang-auto">    def stack_features(features, something):

        return torch.stack([torch.stack(collate_fn(ft), dim=0) for ft in features], dim=0)
</code></pre>
<p>but now I am getting this error:</p>
<p><code>TypeError: expected Tensor as element 0 in argument 0, but got tuple</code></p>
<p>Any ideas on how to solve it? Thank you in advance</p>

ptrblck: <p>I don’t understand how the <code>collate_fn</code> is supposed to solve the issue since you cannot stack tensors in <code>dim0</code> if their size differs in <code>dim1</code>, so you might want to resize, slice, or pad the tensors.</p>

peony: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="193387">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>resize</p>
</blockquote>
</aside>
<p>I now have this function for resizing:</p>
<pre><code class="lang-auto">    def pad_features(ft):
        for f in ft:
            f = f.view(7)
            
        return ft
</code></pre>
<p>but its now throwing this error:<br>
<code>RuntimeError: shape '[7]' is invalid for input of size 3</code></p>

peony: <p>I also tried</p>
<pre><code class="lang-auto"> def pad_features(ft):
        for f in ft:
            f = f.view(3)
            
        return ft
</code></pre>
<p>but I get :</p>
<p><code>RuntimeError: shape '[3]' is invalid for input of size 7</code></p>

peony: <p>Hello <a class="mention" href="/u/ptrblck">@ptrblck</a> , do we have <em>drop_last=True</em> in BucketIterator for torchtext?</p>
<p>I think my problem is my samples are not divisible by batch size, and I have to make the batch size more than 1 as I am implementing a baseline</p>

ptrblck: <p>No, <code>BucketIterator</code> does not provide the <code>drop_last</code> argument as it’s creating batches with similar lengths to minimize padding. I also don’t know why you would need to increase the batch size to avoid this error. Is the batch dimension missing? If so, you can <code>unsqueeze</code> it.</p>
"Expected all tensors to be on the same device " in torch.nn.quantizable.MultiheadAttention

cruvy: <p>I’m using torch.nn.quantizable.MultiheadAttention in my model that trains on GPU. This torch layer has the following code:</p>
<pre><code class="lang-auto">k_zeros = torch.zeros((k.size(0), 1) + k.size()[2:])
...
k = torch.cat([k, k_zeros], dim=1)
</code></pre>
<p>which throws “RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!”, as the tensor k comes from the <code>forward</code> input, and in on GPU.<br>
Is this a bug, and <code>device=k.device</code> should be added in the definition of <code>k_zeros</code>?</p>
GAN Produces the same output for any input at each epoch

Praveen_Benedict: <p>I am trying to develop a GAN on the MNIST dataset. But every time I fetch outputs from the generator, all the outputs look exactly the same like this.</p>
<p>Epoch <span class="hashtag-raw">#2:</span><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/5/9/595d87b0fc4725cde7b13fff2e51df2411c87367.jpeg" alt="1_1000" data-base62-sha1="cKySGEYgwhXE5AMmNVW9Dlf67Rl" width="516" height="270"></p>
<p>Here is the link to the code:</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://colab.research.google.com/drive/15jb7sh4bJeeWKejLyZgR7Gw6sEwIpIAn?usp=sharing">
  <header class="source">
      <img src="https://ssl.gstatic.com/colaboratory-static/common/ac0e6bae38aa2b5a52a4f6e10b43c43d/img/favicon.ico" class="site-icon" width="128" height="128">

      <a href="https://colab.research.google.com/drive/15jb7sh4bJeeWKejLyZgR7Gw6sEwIpIAn?usp=sharing" target="_blank" rel="noopener nofollow ugc">colab.research.google.com</a>
  </header>

  <article class="onebox-body">
    <img width="260" height="260" src="https://colab.research.google.com/img/colab_favicon_256px.png" class="thumbnail onebox-avatar">

<h3><a href="https://colab.research.google.com/drive/15jb7sh4bJeeWKejLyZgR7Gw6sEwIpIAn?usp=sharing" target="_blank" rel="noopener nofollow ugc">Google Colaboratory</a></h3>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Is there anything wrong with the code?</p>
Utilisation de huggingface pour les LLM

Indigo_AI: <p>avec “hugingface tools” il n y a pas de problème pour réaliser un fine tuning du modèle Mistral 7B</p>
<p>dans le code du modèle pytorch de mistral on peut voir que la classe Transformer possède un fonction forward_partial qui permet de récupérer un “embedding” de bonne qualité (test réalisé avec le modèle de base fourni par Mistral)</p>
<p>Malheureusement je ne sais pas réaliser un finetuning de ce modèle de base ni accéder au forward_partial dans le modèle huggingface</p>
<p>Merci si vous avez une idée brillante</p>

ptrblck: <p>Could you translate this post, please?</p>
Multiple Object Detection from scratch

0fByte: <p>Hello all,<br>
I am looking to create a model to do multiple object detection from scratch.</p>
<p>I understand the basics of how to create a (CNN) model for single object classification and localization but I wasn’t able to found a tutorial on how to model from scratch a PyTorch class to do a <strong>multiple</strong> object detection and classification.</p>
<p>Looking for implementation of some well known networks they seems to be using a sliding window algorithm or a grid, but I am not really sure on how to implement them.<br>
I am also having an hard time understanding how to return a variable number of labels, depending on the number of objects in the image, from the forward pass of the network.</p>
<p>Could someone please point me in the right direction?</p>
<p>Thanks! <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=12" title=":smile:" class="emoji" alt=":smile:" loading="lazy" width="20" height="20"></p>
What's the easiest code to train several models with the same structure but different data on several GPUs parallelly?

jack_S: <p>My desired implement is like this:</p>
<pre><code class="lang-auto">data = tensor([data_0, data_1, ... data_M])
models= [deep_copied_model_0, deep_copied_model_1, ..., deep_copied_model_M]
# these models have identical structures, 
# and the initial parameters at the corresponding positions are the same
gpus = ['gpu_0', 'gpu_1', ..., 'gpu_M']
model = create_parallel_model(models, gpus)
...
output = model(data)
# the output should be tensor([deep_copied_model_0(data_0), ..., deep_copied_model_M(data_M)])
# and the devices should be output[i].device = 'gpu_i'
loss = parallel_loss_function(output)
# the loss should be tensor([loss0, loss1, ..., loss_M])
# and the devices should be loss[i].device = 'gpu_i'
loss.backward()
# I want to backpropagate the loss parallelly
...
</code></pre>
<p>Can this be possible and how to implement it?</p>

ptrblck: <p>The easiest approach would be to launch each process separately and pass the desired GPU id to it.<br>
If you want to use a single script, you would need to make sure the CPU in the main thread is fast enough to schedule the workload for multiple devices as the actual GPU execution is asynchronous.</p>

jack_S: <p>Can PowerShell scripts solve this? For example, my desired PowerShell command is equivalent to run these commands on M terminals respectively:</p>
<pre><code class="lang-auto">python train.py --device gpu_0
python train.py --device gpu_1
...
python train.py --device gpu_M
</code></pre>
<p>Or some other Linux commands can help? Any help is appreciated.</p>
How to get the python bytecode that causes the graph break?

weilinquan: <p>Hi, everyone, In dynamo, it will break graph by some unsupport python operation, such as if else. I want to merge the breaked subgraphs to one. For this example, graph will be breaked to two subgraph.</p>
<pre><code class="lang-auto">def forward(self, b, c):
        if not torch.nn.functional.silu(b[0][0]):
            return torch.mm(b, c)
        else:
            return torch.add(b, c)
</code></pre>
<pre><code class="lang-auto">opcode         name       target                             args          kwargs
-------------  ---------  ---------------------------------  ------------  --------
placeholder    l_b_       L_b_                               ()            {}
call_function  getitem    &lt;built-in function getitem&gt;        (l_b_, 0)     {}
call_function  getitem_1  &lt;built-in function getitem&gt;        (getitem, 0)  {}
call_function  silu       &lt;function silu at 0x7f9c4beff920&gt;  (getitem_1,)  {}
output         output     output                             ((silu,),)    {}
</code></pre>
<pre><code class="lang-auto">opcode         name    target                                                  args          kwargs
-------------  ------  ------------------------------------------------------  ------------  --------
placeholder    l_b_    L_b_                                                    ()            {}
placeholder    l_c_    L_c_                                                    ()            {}
call_function  add     &lt;built-in method add of type object at 0x7f9cc57b0aa0&gt;  (l_b_, l_c_)  {}
output         output  output                                                  ((add,),)     {}
</code></pre>
<p>I want to merge two subgraphs to one graph. Is there a way to get the python bytecode that causes the graph break? Or something better to merge the graphs. Thank you for your help!</p>

marksaroufim: <p>You can diagnose graph breaks with <a href="https://pytorch.org/docs/main/logging.html" class="inline-onebox">torch._logging — PyTorch main documentation</a></p>
<p><code>TORCH_LOGS="graph_breaks" python your_code.py</code></p>

weilinquan: <p>I have added torch._logging.set_logs(graph_breaks=True). But it doesn’t seem to work and there is no log in my command line.</p>
<pre><code class="lang-auto">torch._dynamo.reset()
torch._logging.set_logs(graph_breaks=True)
class TestModule(torch.nn.Module):
    def __init__(self, *args, **kwargs) -&gt; None:
        super().__init__(*args, **kwargs)
        self.conv = torch.nn.Conv2d(3, 255, (5, 5), 3, bias=False)

    def forward(self, b, c):
        if not torch.nn.functional.silu(b[0][0]):
            return torch.mm(b, c)
        else:
            return torch.add(b, c)

model = TestModule()
model_opt = torch.compile(model)
print(model_opt(torch.randn((1024, 1024)), torch.randn((1024, 1024))))
</code></pre>