akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
After around 60 epochs, the encoder starts delivering nan values in transformer model

username: Daniel_Sabzi
cooked: <p>Hi everyone,<br>
I have a transformer model with 10 % dropout at the positional encoding and 20% dropout at both encoder and decoder layers.<br>
It works well with this setting in both train and test sections<br>
When I remove the dropout at positional encoding layer or increase it to 15% it still works well in the training section but after 60 epochs or so the encoder starts delivering nan values in test, while training is still working well.</p>
<p>I have an output with sequence length of 1, and thus a tgt mask [1, 1] and an src mask [1, sequence_length], with no padding mask.</p>
<p>Any idea what can cause nan values after 60 epochs? or its relationship between positional encoding dropout?</p>
<p>Thanks</p>

username: ptrblck
cooked: <p>Maybe you could try to use anomaly detection to raise an error if the first invalid gradient is created, which might then help isolating which layer failed first.<br>
You could use <a href="https://pytorch.org/docs/stable/autograd.html#torch.autograd.detect_anomaly"><code>torch.autograd.detect_anomaly</code></a> and run the forward/backward pass in this context manager.</p>

username: Daniel_Sabzi
cooked: <p>@ <a href="https://discuss.pytorch.org/u/ptrblck">ptrblck</a><br>
Thanks.<br>
I tried this, but as I said, the training section works just fine and it is only in the test section that I see “nan” results, even in the test the first 60 epochs work well. and after that it starts to give "nan results for loss.<br>
anyways, I ran it and it did not pick anything.<br>
do you have any idea what can be the source of such error?</p>
What is the sense of parameter qconfig_spec in quantize_dynamic

username: Miguel_Campos
cooked: <p>Hello all.</p>
<p>I have begun to learn about Quantization with “dynamic quantization” as a first try.<br>
According to several tutorials like this or this, it catches my attention that <code>torch.nn.Relu</code> is used in <a href="https://pytorch.org/tutorials/intermediate/dynamic_quantization_bert_tutorial.html#apply-the-dynamic-quantization" rel="noopener nofollow ugc">this example from the official tutorials</a> in the qconfig_spec argument of the function quantize_dynamic:</p>
<blockquote>
<p>quantized_model = torch.quantization.quantize_dynamic(<br>
model, {torch.nn.Linear}, dtype=torch.qint8<br>
)</p>
</blockquote>
<p>The tutorial says: “We specify that we want the torch.nn.Linear modules in our model to be quantized”. But to me, it is obvious that you want them to be converted to int8. You probably want everything you can convert to int8 converted.</p>
<p>With this already being said, I would like to know if it is necessary to specify the Linear module.</p>

username: jcaip
cooked: <p>HI <a class="mention" href="/u/miguel_campos">@Miguel_Campos</a></p>
<p>Are you talking about <code>nn.Linear</code>? I don’t see <code>nn.Relu</code> anywhere in the tutorial you linked. You need to specify the Linear.</p>
<p>Sometimes you have more than one type of module you want to quantize, for example in the case of LSTM + Linear, seen here: <a href="https://pytorch.org/tutorials/recipes/recipes/dynamic_quantization.html" class="inline-onebox" rel="noopener nofollow ugc">Dynamic Quantization — PyTorch Tutorials 2.0.1+cu117 documentation</a></p>
<p>Different layers of a network have a different effect on the accuracy so thats why you may want some quantized and some not.</p>

username: Miguel_Campos
cooked: <p>Sorry my bad, I mean nn.Linear.</p>
<p>Ok I see, thanks for your answer</p>
Does the PyTorch library support automatic selection of different types of network cards, such as Ethernet and high-speed IB network cards?

username: pengshuang
cooked: <p>Our scenario involves two heterogeneous GPU clusters, Cluster A and Cluster B, each consisting of 20 GPU machines (A100-80G). Cluster A is internally equipped with both high-speed IB cards and regular Ethernet cards, while Cluster B is internally equipped with high-speed RoCE cards and regular Ethernet cards. Due to the inability to establish a high-speed IB network between Cluster A and Cluster B, communication between them can only be done via Ethernet using TCP/IP (Socket).</p>
<p>Our objective is to have machines within Cluster A and Cluster B connected through high-speed cards, while machines between Cluster A and Cluster B are connected through regular Ethernet networking. I would like to inquire whether PyTorch or NCCL supports automatic card detection and configuration to enable distributed training between the heterogeneous clusters. If it doesn’t support it, is it possible to achieve this by modifying the source code?</p>
<p>I am unsure of the feasibility and would greatly appreciate your response. Thank you very much!</p>
Is there a way to set PyTorch's quantization parameters so that it can be converted to a TensorRT model?

username: Nanlalalalalal
cooked: <p>When converting quantized model to tensorrt, it says “TRT only supports symmetric quantization - zeroPt must be all zeros”. Is there a way to set PyTorch’s built-in quantization tool’s quantization parameters(symmetric quantization) so that it can be converted to a TensorRT model?<br>
<a href="https://github.com/NVIDIA/TensorRT/issues/2774" rel="noopener nofollow ugc">tensorrt-issue</a></p>

username: Nanlalalalalal
cooked: <p>It seems it is possible: <a href="https://github.com/pytorch/TensorRT/blob/main/py/torch_tensorrt/fx/test/quant/test_quant_trt.py#L37" rel="noopener nofollow ugc">TensorRT/test_quant_trt.py at main · pytorch/TensorRT · GitHub</a><br>
in fx mode right now</p>
How to specify colors for each class with draw_segmentation_masks for segmentation data

username: Abisoye_Durojaiye
cooked: <p>I’m currently working with a segmentation dataset. I decided to use the draw_segmentation_masks function is torchvision to visualize the data, but the  function returns different colors in the segmentation masks for a single object category in different images, and I can’t find a way to specify the classes/labels in the function arguments. Any workaround this?</p>

username: eqy
cooked: <p>If I understand correctly, it sounds like the issue is due to the fact that not every image has the same classes segmented/masked. Could you define your own map of classes to colors, and create a tuple of colors for the specific classes that show up for each image (they would need to be in the order they appear in the masks tensor), and pass that via the <code>colors</code> argument?</p>

username: Abisoye_Durojaiye
cooked: <p>Apologies for the last response. I tried to do what you suggested, but I’m not sure it’s possible. I’m reading the data with a custom Dataset class, as described here : <a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">TorchVision Object Detection Finetuning Tutorial — PyTorch Tutorials 2.0.1+cu117 documentation</a>. The issue is, the code, gets the number of unique objects in an image, not how many times they occur, and I can’t think of any way to get the number of occurences of each object in an image. If I could do that, I would be able to use the classes to colors mapping as you suggested.</p>

username: eqy
cooked: <p>I don’t think you need to count anything, you would simply do something like<br>
<code>colors = [label2color[labels[i]] for i in range(N)]</code> and pass that as the colors list, with <code>label2color</code> being your own predfined mapping from classes to colors.</p>
An error of torchrun in the docker container

username: f2d73cd01af2f9c3daab
cooked: <p>Hi, I can successfully run my code using distributed training mode on my Linux server, but an error happens when I run my code in a docker container with the same environment as my server. The error is as follows:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/2/a2a995ab039f3d3f21ae90e16c68e98e9ebe2ea4.png" data-download-href="https://discuss.pytorch.org/uploads/default/a2a995ab039f3d3f21ae90e16c68e98e9ebe2ea4" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/2/a2a995ab039f3d3f21ae90e16c68e98e9ebe2ea4.png" alt="image" data-base62-sha1="ncYGnS7MHdn8hL2RoHKSwLotWRK" width="690" height="323" data-dominant-color="080808"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1366×640 25.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I don’t know what (exit code: -7) means, and I don’t find a similar error on the web. Please help me.</p>

username: Minxiangliu
cooked: <p>Have you resolved it? I also encountered the same issue.</p>
Keypoint RCNN on C++

username: roboticswithamir
cooked: <p>How can I train a keypointrcnn_resnet50_fpn on Python and load it on C++? Or is there anyway to include it directly in C++ and train it?</p>

username: ptrblck
cooked: <p>You could try to <code>torch.jit.script</code> the model in Python and load it in <code>libtorch</code> afterwards as explained <a href="https://pytorch.org/tutorials/advanced/cpp_export.html">here</a>.</p>

username: roboticswithamir
cooked: <p>I already went through the documentation. It works fine for resnet18 model, but once I save the fasterrcnn_resnet50_fpn with the same instructions and try to load it on C++, I get this error:</p>
<pre><code class="lang-auto">
terminate called after throwing an instance of ‘torch::jit::ErrorReport’
what():
Unknown type name ‘NoneType’:
Serialized File “code/**torch**/torchvision/models/detection/transform.py”, line 11
image_std : List[float]
size_divisible : int
fixed_size : NoneType
~~~~~~~~ &lt;— HERE
def forward(self: **torch**.torchvision.models.detection.transform.GeneralizedRCNNTransform,
images: List[Tensor],

Aborted (core dumped)
</code></pre>

username: roboticswithamir
cooked: <p>I’m going to share the Python, C++ and CMake files here for more information:</p>
<p>Python:</p>
<pre><code class="lang-auto">import torch

import torchvision

model = torchvision.models.detection.fasterrcnn_resnet50_fpn()

# An example input you would normally provide to your model's forward() method.

example = torch.rand(1, 3, 224, 224)

# Use torch.jit.trace to generate a torch.jit.ScriptModule via tracing.

traced_script_module = torch.jit.script(model, example)

traced_script_module.save("rcnn_model.pt")
</code></pre>
<p>C++:</p>
<pre><code class="lang-auto">#include &lt;iostream&gt;
#include &lt;torch/torch.h&gt;
#include &lt;torch/script.h&gt;

int main()
{

	torch::jit::script::Module model;
	
    try 
    {
    model = torch::jit::load("/home/bluesky/Documents/Codes/AI/Keypoint_RCNN/rcnn_model.pt");
    } 
    
    catch (const c10::Error&amp; e) 
    {
        std::cerr &lt;&lt; "Error loading the model: " &lt;&lt; e.msg() &lt;&lt; std::endl;
    }
}
</code></pre>
<p>CMake:</p>
<pre><code class="lang-auto">cmake_minimum_required(VERSION 3.2)

project(torchtest)

set(CMAKE_PREFIX_PATH /home/bluesky/Downloads/libtorch-cxx11-abi-shared-with-deps-1.8.0+cu112/libtorch)
find_package(Torch REQUIRED)
 
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} ${TORCH_CXX_FLAGS}")
 
add_executable(${PROJECT_NAME} main.cpp)
 
target_link_libraries(${PROJECT_NAME} "${TORCH_LIBRARIES}")
 
set_property(TARGET ${PROJECT_NAME} PROPERTY CXX_STANDARD 14)
</code></pre>
How to index into the layers within the EfficentNet backbone block?

username: weight_theta
cooked: <p>Dear community,</p>
<p>I am trying to examine the outputs a specific layer within the EfficenNet backbone, however I am not able to index properly into the network. This is what I am doing:</p>
<pre><code class="lang-auto">mod = nn.Sequential(
              *list(models.efficientnet_v2_s(pretrained=True).children())[:-2][0][:5]
              )
mod
</code></pre>
<p>which returns the nn.sequential definition of the backbone. I am interested in the Backbone defined from the start to the <code>(4): MBConv( )</code> Block , however when trying to index into it with <code>[:4]</code> or by trying to remove the last block using <code>[:-1]</code> I get the <code>(3): MBConv( )</code> block. How can I examine the output of the <code>(4): MBConv( )</code> without it being attached to the <code>(5): MBConv( )</code> block?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="weight_theta" data-post="1" data-topic="181754">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/w/d26b3c/48.png" class="avatar"> weight_theta:</div>
<blockquote>
<p>I am trying to examine the outputs a specific layer within the EfficenNet backbone</p>
</blockquote>
</aside>
<p>Forward hooks would be a good way to inspect the output of a specific layer as given in e.g. <a href="https://discuss.pytorch.org/t/how-can-l-load-my-best-model-as-a-feature-extractor-evaluator/17254/6">this post</a>.</p>
<p>Based on your code snippet it seem you want to recreate the model somehow, which might be tricky since you are re-wrapping everything into an <code>nn.Sequential</code> block, which might miss important functional API calls from the original <code>forward</code> method.</p>

username: weight_theta
cooked: <p>Thank you Patrik. I will look in to the forward hook.</p>
<blockquote>
<p><code>nn.Sequential</code> block, which might miss important functional API calls from the original <code>forward</code> method.</p>
</blockquote>
<p>I was not aware of this. How would you use the backbone to ensure that the forward push is unaffected and not wrapped into an nn.sequential ? The liner layer and avg pooling layer would still have to be removed though.</p>

username: ptrblck
cooked: <p>A clean way would be to inspect the model including its source (in particular the <code>forward</code> definition) and to override it with your own <code>forward</code> method in a custom model.<br>
Wrapping submodules into <code>nn.Sequential</code> might work, but could also easily break as seen in other topic in this discussion board as e.g. often flattening operations are missing since these are often not defined as a module but used inside the <code>forward</code> via e.g. <code>x = x.view(x.size(0), -1)</code>.</p>
How to do inverse transformation in the test image data

username: new_eng
cooked: <p>I do some augmentation techniques in my image dataset  ```<br>
RandomRotation,RandomVerticalFlip, and others at the same time. after training I want to recover the original image with no augmentation in the test part. It there a flag or something that identifies which image was transformed and with which transformation do to the inverse transformation ?</p>
<pre data-code-wrap="How"><code class="lang-plaintext"></code></pre>

username: ptrblck
cooked: <p>No, the transformations do not track their randomly samples arguments, but you could track them manually and by sampling the random arguments first and then by applying them with the functional transformation API.<br>
<a href="https://discuss.pytorch.org/t/torchvision-transfors-how-to-perform-identical-transform-on-both-image-and-target/10606/7">This post</a> gives you an example how this API can be used.</p>
Creating a Dataloader with atleast 2 instances from each class

username: Abhranta
cooked: <p>I am performing a multiclass classification problem which requires me to find the intra class covariance in each epoch. For this reason, I need atleast 2 instancess from each class as the covariance formula has 1/(n-1) where n is the number of instance per class and if ther is only a single instance, then it gives Zero Divison Error. So, my question is <strong>how do I make a dataloader which has atleast 2 instanceper data class ?</strong></p>

username: ptrblck
cooked: <p>By default all samples will be used in each epoch, so assuming your dataset contains at least two samples per class you wouldn’t have to change anything.<br>
If that’s not the case you could try to oversample the classes with a single sample e.g. with a <code>WeightedRandomSampler</code> or you could create a custom sampler guaranteeing to sample specific classes.</p>

username: Abhranta
cooked: <p>Well, I need a dataloader where each batch contains atleast 2 samples from each class. Sorry for the confusion !!</p>

username: ptrblck
cooked: <p>In this case <a href="https://discuss.pytorch.org/t/how-to-sample-atleast-one-sample-from-each-class-for-a-batch-in-dataloaders/132765">this topic</a> might be interesting.</p>

username: Abhranta
cooked: <p>Hi, actually I haven’t been able to convey my doubt clearly. My doubt is that I want to create a dataloader where no batch has any class with a single instance.<br>
Let’s say, I have 100 classes → {1,2,3,4,5,…,100}<br>
My dataloader has a batch size of 10.<br>
The kind of batches that I want are → {1,1,2,2,3,3,9,9,9,9} , {79,79,79,32,32,64,64,64,64,64}<br>
The kind of batches that I don’t want are -&gt;{1,2,4,8,81,81,81,4,4,4} because there is only a single instance form class 1 and class 2. This is raising a error while calculating intra-class covariance during training.</p>

username: ptrblck
cooked: <p>I still think a custom <code>sampler</code> could still work as it’s responsible to create the sample indices and pass it to the <code>Dataset.__getitem__</code>. Creating a custom <code>BatchSampler</code> will also allow you to pass all batch indices to the <code>__getitem__</code> method and thus load the entire batch.</p>
Computational Graph continuity when replacing weights of a model

username: pirazh
cooked: <p>Hi all,</p>
<p>I have a question regarding the continuity of computational graph when replacing the weights of a model. I have a complete graph in which first I predict a set of weights corresponding to a neural net which is not explicitly optimized. I just need the model to compute a loss and therefore obtain gradients. However, after I predict weights of that model, upon replacing those using the <code>.copy_</code> method Pytorch throws an error for the modification of Leaf Nodes. Also only copying the weights does not transfer information for completing the graph. Does anyone have a suggestion regarding how to go about this issue?</p>
<p>Thanks</p>

username: eqy
cooked: <p>If the predicted weights aren’t really parameters of a model that are meant to be repeatedly updated, then you might want to consider using the <code>torch.nn.functional</code> API: <a href="https://pytorch.org/docs/stable/nn.functional.html" class="inline-onebox" rel="noopener nofollow ugc">torch.nn.functional — PyTorch 2.0 documentation</a> for its forward function/layers and pass the predicted weights as an input to the model that isn’t explicitly optimized.</p>

username: pirazh
cooked: <p><a class="mention" href="/u/eqy">@eqy</a> thanks for your response. The weights are in fact parameters. However rather than directly updating them using an optimizer, another model is supposed to predict those. Theoretically there is a complete computational graph, but I am not sure how I can handle in Pytorch.</p>

username: eqy
cooked: <p>Yes, they are parameters, but your use case suggests that they should not be treated as <code>nn.Parameter</code> type <a href="https://pytorch.org/docs/stable/generated/torch.nn.parameter.Parameter.html" class="inline-onebox" rel="noopener nofollow ugc">Parameter — PyTorch 2.0 documentation</a>. Instead you can pass the weights as inputs via the functional API: <a href="https://pytorch.org/docs/stable/nn.functional.html" class="inline-onebox" rel="noopener nofollow ugc">torch.nn.functional — PyTorch 2.0 documentation</a></p>

username: pirazh
cooked: <p>Thanks <a class="mention" href="/u/eqy">@eqy</a>. Your solution worked. You are right, since those would be weights involved in a computational graph and nothing more, the functional API is the way to go.</p>
Getting blobs of a mask

username: MUnique
cooked: <p>Hi,<br>
I have a heatmap image like this:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/5/f/5f48d494e4766874a9a8a26a4d381d06f6ff37b2.png" alt="a" data-base62-sha1="dAVoGhZX9DYav9TQPGtb2gdh21A" width="256" height="256"><br>
What is the best way to get the blobs and the peak of each blob position fully in pytorch?</p>
Deep Quadric Learning

username: diro5t
cooked: <p>Learn about my project Deep Quadric Learning - Episode 1 - ‘Motivation’ is out: <a href="https://github.com/diro5t/deep_quadric_learning" class="inline-onebox" rel="noopener nofollow ugc">GitHub - diro5t/deep_quadric_learning: Deep Quadric Learning</a></p>
Faster-RCNN label scores

username: Caio_Nogueira
cooked: <p>The Pytorch Faster-RCNN model returns a set of bounding boxes, labels, and scores. Each element of the <code>scores</code> list is the classification score of the label at the same index. Is there a way to obtain, for each detection, the classification score for each label?<br>
Instead of returning a <code>scores</code> with shape <code>(num_detections)</code>, I would like to obtain it with shape <code>(num_detections, num_classes)</code>.</p>
nn.CosineEmbeddingLoss(): Validation loss stays constant

username: vAboutPie
cooked: <p>The Validation loss stays constant when nn.CosineEmbeddingLoss() function is used, while nn.CrossEntropy() is working fine.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/8/d81d976d75f4e0183b2b5464084f2e4447c32a26.png" data-download-href="https://discuss.pytorch.org/uploads/default/d81d976d75f4e0183b2b5464084f2e4447c32a26" title="lossFunctionresults"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/8/d81d976d75f4e0183b2b5464084f2e4447c32a26_2_690x240.png" alt="lossFunctionresults" data-base62-sha1="uPQx3oYQpv9aHC74m1RxvTV9tBk" width="690" height="240" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/8/d81d976d75f4e0183b2b5464084f2e4447c32a26_2_690x240.png, https://discuss.pytorch.org/uploads/default/optimized/3X/d/8/d81d976d75f4e0183b2b5464084f2e4447c32a26_2_1035x360.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/d/8/d81d976d75f4e0183b2b5464084f2e4447c32a26.png 2x" data-dominant-color="FAFAFA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">lossFunctionresults</span><span class="informations">1260×439 32.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Creating a Multivariate Normal Distribution with multiple centers and sampling from them

username: legoh
cooked: <p>Hi all, I’m wondering how I could create a 2D Gaussian distribution, where we have multiple centers, and sample from it? I tried the code below, but whenever I tried to sample a batch, I would get samples from every center. What I actually want is to first create these multiple centers, then re-normalize the distributions such that overlapping regions would have higher probability. Then sample a batch of points out based on this.</p>
<pre><code class="lang-auto">centers = torch.tensor([[0.1, 0.1],
                        [0.9, 0.9],
                        [0.5, 0.5],
                        [0.9, 0.1],
                        [0.1, 0.9]])

mv = torch.distributions.multivariate_normal.MultivariateNormal(centers, torch.eye(2))
</code></pre>

username: KFrank
cooked: <p>Hi Legoh!</p>
<aside class="quote no-group" data-username="legoh" data-post="1" data-topic="181721" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/l/82dd89/48.png" class="avatar"> legoh:</div>
<blockquote>
<p>Hi all, I’m wondering how I could create a 2D Gaussian distribution, where we have multiple centers, and sample from it?</p>
</blockquote>
</aside>
<p>If I understand what you want correctly, you have a <em>multigaussian</em> distribution.<br>
That is, your probability distribution is the sum of multiple gaussian distributions<br>
(as distinct from a single random variate being a sum of multiple gaussian<br>
variates).</p>
<p>In your example, it looks like you have five “centers.”  Build a gaussian<br>
distribution for each of your five centers, generate a random integer from<br>
0 to 4 (e.g., <code>torch.multinomial()</code>) and use that to choose which of your<br>
five gaussian distributions to sample from.  If appropriate, you can weight<br>
your five distributions differently.</p>
<p>Note, it could end up being faster to generate your batch of five gaussian<br>
variates and then throw four away by multiplying the vector of your five<br>
gaussian variates with the one-hot version of your random integer.</p>
<p>Best.</p>
<p>K. Frank</p>
Hashing + embedding before passing the input to model

username: add023
cooked: <p>My features are a mix of univalent/multivalent dense &amp; sparse categorical string, and univalent/multivalent dense &amp; sparse categorical int. I need to convert the categorical string features to integer ids which can be then used for embedding lookup. What I am trying to do is Keras equivalent of:</p>
<pre><code class="lang-auto">hashing_layer = tf.keras.layers.Hashing(vocab_size)(feature)
embedding = tf.keras.layers.Embedding(vocab_size, 2, sparse=True)(hashing_layer)
embedding = reduce_mean(embedding, axis=1)
</code></pre>
<p>eg, for a batch size of 3:</p>
<pre><code class="lang-auto">raw_feature = [['z'], [], ['aa', 'bb']]
hashing_layer = [[939], [], [72, 82]]
embedding = 
       [[[0.019482743, 0.03436062]], 
        [], 
       [[-0.01437955, 0.042842973],
       [-0.007321369, -0.049441278]]]
input_to_model = 
       [[ 0.01948274,  0.03436062],
       [        0,         0],
       [-0.01085046, -0.00329915]]
</code></pre>
<p>What’s the PyTorch way of implementing this? Should I create the hash ids inside the Dataset class and implement the embedding (maybe embeddingbag?) inside the model? And what’s the Pytorch way to implementing hash id (hashing + mod)? I found <a href="https://pytorch.org/torcharrow/beta/generated/torcharrow.functional.sigrid_hash.html" rel="noopener nofollow ugc">sigrid_hash</a> but it needs the input to be an integer to begin with.</p>
Best and Worst reconstructed images/data from an Autoencoder

username: saahith
cooked: <p>I use an autoencoder model to encode and decode data matrices(instead of starting with images) of a variable (velocity) in a flow field. I would, now, like to extract the samples from the complete dataset that have been best and worst reconstructed in that last epoch. Is there a specific inbuilt function in torch to already do this or do I need to calculate the error for every sample during the last epoch?<br>
Thank you.</p>

username: ptrblck
cooked: <p>I don’t think PyTorch will automatically track any metrics for you so you would need to compute the error and use these stats to get the corresponding samples.</p>

username: saahith
cooked: <p>Thank you for the answer. Yes, I computed what I needed on my own.</p>
What if add an additional transition layer behind the last Denseblock layer in DenseNet?

username: jxl0131
cooked: <p>I know that the official DenseNet has no transition after the last Denseblock, but I found that an additional layer has been added in an open source code(<a href="https://github.com/fnzhan/Illumination-Estimation/blob/master/RegressionNetwork/DenseNet.py" class="inline-onebox" rel="noopener nofollow ugc">EMLight/DenseNet.py at master · fnzhan/EMLight · GitHub</a>). What will be the impact of this?</p>
Plagiarism Detection

username: ANKIT_KUMAR
cooked: <p>How to build Plagiarism detection AI model when you are not injecting data initially but you are injecting during the process to understand the pattern how the content person is following the pattern</p>
<ol>
<li>Can I use Passive Aggressive Classifier algorithm</li>
<li>Do tokenization, stopwords should be used</li>
<li>Is there necessary to include cosine similarity</li>
</ol>
Does shuffling the samples interfere in training with 1-CCC loss function?

username: davidfeijoo
cooked: <p>The Concordance Correlation Coefficient is defined as follows:</p>
<aside class="onebox wikipedia" data-onebox-src="https://en.wikipedia.org/wiki/Concordance_correlation_coefficient">
  <header class="source">

      <a href="https://en.wikipedia.org/wiki/Concordance_correlation_coefficient" target="_blank" rel="noopener nofollow ugc">en.wikipedia.org</a>
  </header>

  <article class="onebox-body">
    

<h3><a href="https://en.wikipedia.org/wiki/Concordance_correlation_coefficient" target="_blank" rel="noopener nofollow ugc">Concordance correlation coefficient</a></h3>

<p>In statistics, the concordance correlation coefficient measures the agreement between two variables, e.g., to evaluate reproducibility or for inter-rater reliability.
 The form of the concordance correlation coefficient 
  
    
      
        
          ρ
          
            c
          
        
      
    
    {\displaystyle \rho _{c}}
  
 as
 where 
  
    
      
        
          μ
          
            x
          
        
      
    
    {\displaystyle \mu _{x}}
  
 and 
  
    
   ...</p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/a/ea8d7dabea5bf605c1e542ae90772ddc2448cfeb.png" alt="Captura de pantalla 2023-06-08 a las 18.33.41" data-base62-sha1="xsWRZVJjIbviBXjjFonjJdDfzhF" width="394" height="124"></p>
<p>The loss function computed between the labels and the outputs of the model is L = 1 - CCC</p>
<p>My question is: if the samples have temporal dependency (in this case they are clips from the same audio which will probably have more similar labels), does it matter whether to shuffle the training set before training? And will the batch size be very critical?</p>
Is it necessary to load the SGD optimizer to resume training?

username: davidfeijoo
cooked: <p>I am training a DNN in PyTorch with 4 dense layers with ReLU as activation function, batch norm and dropout (0.7) in every hidden layer. I was wondering if it is necessary to save and load the SGD (learning_rate = 0.001, momentum = 0.9) optimizer state for resuming training? Also, what if I only want to train the output layer after loading?</p>
<p>Also, I am using 1-CCC (Concordance Correlation Coefficient: <a href="https://en.wikipedia.org/wiki/Concordance_correlation_coefficient" class="inline-onebox" rel="noopener nofollow ugc">Concordance correlation coefficient - Wikipedia</a>) as loss function. The data I am using is several clips from the same audios, the training, validation and test set all contain different audios from different speakers. My question is if it matters whether I should shuffle all the sets so that sentences are not ordered, shuffle only the train set or just don’t need to shuffle them. I’m not sure if it might affect the results.</p>

username: ptrblck
cooked: <p>I would generally recommend saving/loading the <code>state_dict</code> of the optimizer as a good practice even if it might not be needed for specific use cases. In your use case, the <code>momentum_buffers</code> would be filled since you are using <code>momentum=0.9</code> so restoring it would thus also be needed.</p>

username: davidfeijoo
cooked: <p>Thank you. About the loss function part. Do you know whether it is needed to shuffle all the data, just training or none of it? I am afraid that as the sentences that are continuous in time have similar labels, not getting them in order might affect to the purpose of using the 1-ccc loss.</p>

username: ptrblck
cooked: <p>Sorry, but I’m not deeply familiar with the loss function and your use case.</p>
<aside class="quote no-group" data-username="davidfeijoo" data-post="1" data-topic="181684">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/davidfeijoo/48/61089_2.png" class="avatar"> davidfeijoo:</div>
<blockquote>
<p>The data I am using is several clips from the same audios, the training, validation and test set all contain different audios from different speakers.</p>
</blockquote>
</aside>
<p>Based on this it seems you have split the datasets also based on the speakers and I would assume you want to keep it as it fits your use case best (I thus also assume that only new speakers will be used during the model deployment for inference).<br>
In this case you could try to shuffle the training dataset as usually shuffling helps in the model training.<br>
However, I don’t know if your data contains some temporal dependency and shuffling could also break it.</p>

username: davidfeijoo
cooked: <p>Ok. Thank you for your help! I’ll post a new question asking for this specifically.</p>
CIFAR10 add perturbation slow cudnn_convolution

username: pkun
cooked: <p>I generate a set of noises and add them on CIFAR-10. But, after add the noises, I found that the entire training time has increased by three times and simultaneously consuming more space.</p>
<p>I dont know what happen and use profile to see what module cost more time. Then, I found that <code>cudnn_convolution</code> consumed more time, but I don’t understand why this extra time was generated.</p>
<p>Here is my code for dataset which add noises.</p>
<pre><code class="lang-python">class SSLPerturbationDataset(CIFAR10):
    def __init__(self, root: str, train: bool = True, transform=None, target_transform=None, download: bool = False, noises=[]) -&gt; None:
        super().__init__(root, train, transform, target_transform, download)
        self.data = self.data.astype(np.float32)
        self.labels = self.targets
        self.noises = noises
        if len(self.noises.shape) == 4:
            self.noises = self.noises.mul(255).clamp_(
                0, 255).permute(0, 2, 3, 1).to('cpu').numpy()
        else:
            self.noises = self.noises.mul(255).clamp_(
                0, 255).permute(0, 1, 3, 4, 2).to('cpu').numpy()
        with torch.no_grad():
            for idx in tqdm(range(len(self)), desc='add noise'):
                self.data[idx] = self.data[idx] + self.noises[idx]
                self.data[idx] = np.clip(self.data[idx], a_min=0, a_max=255)
        self.data = self.data.astype(np.uint8)

    def __getitem__(self, index: int):
        img, target = self.data[index], self.targets[index]
        img = Image.fromarray(img)

        if self.transform is not None:
            img1 = self.transform(img)
            img2 = self.transform(img)

        if self.target_transform is not None:
            target = self.target_transform(target)

        return img1, img2</code></pre>

username: ptrblck
cooked: <p>I doubt any GPU operation will be affected by an offline CPU transformation so could you post a minimal and executable code snippet showing the slowdown, please?</p>
Transformer model is overfitting

username: thinkin-machine
cooked: <p>I am training a transformer model for english to hindi translation task. Train datasets contains 30k samples. And val dataset contains 5k samples. I tried regularising by adding dropout and weght_decay but model is still overfitting. What could be the possible solution to train model succesfully?</p>
<pre><code class="lang-auto">
SRC_VOCAB_SIZE = len(vocab_transform[SRC_LANGUAGE])
TGT_VOCAB_SIZE = len(vocab_transform[TGT_LANGUAGE])
EMB_SIZE = 512
NHEAD = 8
FFN_HID_DIM = 512
BATCH_SIZE = 128
NUM_ENCODER_LAYERS = 3
NUM_DECODER_LAYERS = 3

transformer = Seq2SeqTransformer(NUM_ENCODER_LAYERS, NUM_DECODER_LAYERS, EMB_SIZE,
                                 NHEAD, SRC_VOCAB_SIZE, TGT_VOCAB_SIZE, FFN_HID_DIM, dropout=0.6)

for p in transformer.parameters():
    if p.dim() &gt; 1:
        nn.init.xavier_uniform_(p)

transformer = transformer.to(DEVICE)

loss_fn = torch.nn.CrossEntropyLoss(ignore_index=PAD_IDX)

optimizer = torch.optim.RAdam(transformer.parameters(), lr=1e-5, betas=(0.9, 0.98), eps=1e-9, weight_decay=1e-1)
</code></pre>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/d/6db1ad137f461f394bcab2149be898bbf1803eea.png" alt="image" data-base62-sha1="fEoIeMhw9q5lp3voKYZaN7SCfFM" width="556" height="413"></p>
Role of DistributedSampler in FSDP tutorial code

username: AntreasAntoniou
cooked: <p>I’ve been trying to use FSDP in my research, and while reading the tutorial at <a href="https://pytorch.org/tutorials/intermediate/FSDP_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Getting Started with Fully Sharded Data Parallel(FSDP) — PyTorch Tutorials 2.0.1+cu117 documentation</a>. I came across the following lines</p>
<pre><code class="lang-python">sampler1 = DistributedSampler(dataset1, rank=rank, num_replicas=world_size, shuffle=True)
sampler2 = DistributedSampler(dataset2, rank=rank, num_replicas=world_size)
</code></pre>
<p>Followed by</p>
<pre><code class="lang-python">train_loader = torch.utils.data.DataLoader(dataset1,**train_kwargs)
test_loader = torch.utils.data.DataLoader(dataset2, **test_kwargs)
</code></pre>
<p>Then the tutorial code never uses the sampler, other than to</p>
<pre><code class="lang-python">def train(args, model, rank, world_size, train_loader, optimizer, epoch, sampler=None):
    model.train()
    ddp_loss = torch.zeros(2).to(rank)
    if sampler:
        sampler.set_epoch(epoch)
    for batch_idx, (data, target) in enumerate(train_loader):
        data, target = data.to(rank), target.to(rank)
        optimizer.zero_grad()
        output = model(data)
        loss = F.nll_loss(output, target, reduction='sum')
        loss.backward()
        optimizer.step()
        ddp_loss[0] += loss.item()
        ddp_loss[1] += len(data)

    dist.all_reduce(ddp_loss, op=dist.ReduceOp.SUM)
    if rank == 0:
        print('Train Epoch: {} \tLoss: {:.6f}'.format(epoch, ddp_loss[0] / ddp_loss[1]))
</code></pre>
<p>Is this intentional? Are the DistributedSampler lines legacy code that was not removed? If not, how is the distributed sampler used? My understanding is that with FSDP all nodes process the same batch, but each node only processes fprop and bprop for different model, parameter, and optimizer shards. Am I wrong? Can you shed some light?</p>
<p>Thanks</p>

username: ptrblck
cooked: <p>The samplers are added to the <code>train/test_kwargs</code>:</p>
<pre><code class="lang-python">    train_kwargs = {'batch_size': args.batch_size, 'sampler': sampler1}
    test_kwargs = {'batch_size': args.test_batch_size, 'sampler': sampler2}
    cuda_kwargs = {'num_workers': 2,
                    'pin_memory': True,
                    'shuffle': False}
    train_kwargs.update(cuda_kwargs)
    test_kwargs.update(cuda_kwargs)
</code></pre>
<p>directly after their initialization and thus used in the <code>DataLoader</code>s.</p>
C ++ code program needed to run

username: dellan
cooked: <p>Dear Team,</p>
<p>Im currently trying to find a site or program that will run this script, i have tried to run it but it fails to run, the website for this is <a href="https://www.kaggle.com/code/heeraldedhia/kmeans-clustering-for-customer-data" class="inline-onebox" rel="noopener nofollow ugc">KMeans Clustering for Customer Data | Kaggle</a></p>
<p>Can anyone please assist.</p>
<p>void create()<br>
{<br>
// file pointer<br>
fstream fout;</p>
<pre><code>// opens an existing csv file or creates a new file.
fout.open("benign_traffic.csv", ios::out | ios::app);

cout &lt;&lt; "Enter the details of 5 students:"
     &lt;&lt; " roll name maths phy chem bio";
&lt;&lt; endl;

int i, roll, phy, chem, math, bio;
string name;

// Read the input
for (i = 0; i &lt; 5; i++) {

    cin &gt;&gt; roll
        &gt;&gt; name
        &gt;&gt; math
        &gt;&gt; phy
        &gt;&gt; chem
        &gt;&gt; bio;

    // Insert the data to file
    fout &lt;&lt; roll &lt;&lt; ", "
         &lt;&lt; name &lt;&lt; ", "
         &lt;&lt; math &lt;&lt; ", "
         &lt;&lt; phy &lt;&lt; ", "
         &lt;&lt; chem &lt;&lt; ", "
         &lt;&lt; bio
         &lt;&lt; "\n";
}
</code></pre>
<p>}</p>

username: ptrblck
cooked: <p>I don’t completely understand how the Kaggle notebook is related to the posted code and how these are related to PyTorch so could you give more details, please?</p>
GeForce 3090 multi GPU hanging

username: Anne
cooked: <p>Hello, when I use the GeForce 3090 multi-GPU to run the code, there is a hang phenomenon, but I don’t know where the problem is, I try to continue training with num_workers=0, but it still hangs, and some GPUs are occupied 100% of cases<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/8/78bef9013a35f6272875352f8cc5a13eb4caaba6.png" data-download-href="https://discuss.pytorch.org/uploads/default/78bef9013a35f6272875352f8cc5a13eb4caaba6" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/8/78bef9013a35f6272875352f8cc5a13eb4caaba6.png" alt="image" data-base62-sha1="heas7leswXVIJUY1Sp8pFvGzRCS" width="690" height="89" data-dominant-color="2B322E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">737×96 14.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
How to pass include (-I) options for torch.compile

username: rwr
cooked: <p>I am getting an compilation error saying that python header files cannot be found when I use torch.compile. I know where they are (/usr/include/python3.8), but torch.compile and gcc cannot find them. How can I pass them to torch.compile?</p>
<p><code>torch._dynamo.exc.BackendCompilerFailed: debug_wrapper raised CalledProcessError: Command '['/usr/bin/gcc', '/tmp/tmpmt0tghjb/main.c', '-O3', '-I/usr/local/cuda/include', '-I/home/mkellman/__pypackages__/3.8/include', '-I/tmp/tmpmt0tghjb', '-shared', '-fPIC', '-lcuda', '-o', '/tmp/tmpmt0tghjb/triton_.cpython-38-x86_64-linux-gnu.so', '-L/usr/lib/x86_64-linux-gnu']' returned non-zero exit status 1.</code></p>

username: marksaroufim
cooked: <p>There isn’t a way to pass in python header files via torch.compile, to me it looks like you have a problem with your installation. Would suggest using this script <a href="https://github.com/pytorch/pytorch/blob/main/tools/dynamo/verify_dynamo.py" class="inline-onebox" rel="noopener nofollow ugc">pytorch/verify_dynamo.py at main · pytorch/pytorch · GitHub</a> to debug</p>
Random grad error when backward a convolution layer

username: Mactarvish
cooked: <p>I’ve just met a weird situation when using <code>backward()</code>, the grad will be slightly different every time I call <code>backward()</code> for identical conv layer and input, here’s my code:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
torch.set_printoptions(precision=8)    


def compare_for():
    grads = []
    for i in range(10):
        r2_input, r2_conv = torch.load("r2_input_conv.pkl")
        r2_output = r2_conv(r2_input)
        loss2 = nn.SmoothL1Loss(reduction='none')(r2_output, torch.ones_like(r2_output)).mean()
        loss2.backward()

        # assert (r2_conv.weight.grad == r3_conv.weight.grad).all()
        g2 = r2_conv.weight.grad 
        grads.append(g2)

    for i in range(len(grads)):
        for j in range(i + 1, len(grads)):
            is_same = (grads[i] == grads[j]).all()
            print(is_same)
            if not is_same:
                print(torch.abs(grads[i] - grads[j]).sum())


if __name__ == "__main__":
    compare_for()
            
        
</code></pre>
<p>And the <code>r2_input_conv.pkl</code> can be downloaded from:</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://pan.baidu.com/share/init?surl=JoQLukOQYgT3I5hiztky9g&amp;pwd=6wwa">
  <header class="source">
      <img src="https://pan.baidu.com/m-static/base/static/images/favicon.ico" class="site-icon" width="" height="">

      <a href="https://pan.baidu.com/share/init?surl=JoQLukOQYgT3I5hiztky9g&amp;pwd=6wwa" target="_blank" rel="noopener nofollow ugc">pan.baidu.com</a>
  </header>

  <article class="onebox-body">
    

<h3><a href="https://pan.baidu.com/share/init?surl=JoQLukOQYgT3I5hiztky9g&amp;pwd=6wwa" target="_blank" rel="noopener nofollow ugc">百度网盘 请输入提取码</a></h3>

  <p>百度网盘为您提供文件的网络备份、同步和分享服务。空间大、速度快、安全稳固，支持教育网加速，支持手机端。注册使用百度网盘即可享受免费存储空间</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p>
链接: <a href="https://pan.baidu.com/s/1JoQLukOQYgT3I5hiztky9g?pwd=6wwa" rel="noopener nofollow ugc">https://pan.baidu.com/s/1JoQLukOQYgT3I5hiztky9g?pwd=6wwa</a> 提取码: 6wwa 复制这段内容后打开百度网盘手机App，操作更方便哦</p>
<p>What factor causes the difference? I think the grad shall be itentical no matter how many times I run the code.</p>
Crop the tensor using mask

username: MetallicMonkeyu
cooked: <p>I have a tensor in form (B, C, H, W) and range (x_min, y_min, x_max, y_max). I would like to apply a filter with the range on tensor’s dimension H and W. E.g for a tensor (1,3,500,500) and a range (100, 100, 200, 200), I would like to get the tensor result of (1,3,100:200,100:200) of shape (1, 3, 100, 100). Any ideas of how to achieve this? What I have tried is to use reduce function and make a mask to filter out the out-of-range pixels.</p>
<pre><code class="lang-auto">mask = reduce(torch.logical_and, (img[:,:, ?,:] &gt;= range[1],
                                         img[:,:, ?,:] &lt; range[3],
                                         img[:,:, :,?]  &gt;= range[0],
                                         img[:,:, :,?] &lt; range[2]))
</code></pre>
<p>But the issue is I don’t know what to put at the position of “?” so that it compares the value at that dimension to the range value.</p>

username: J_Johnson
cooked: <p>Is this what you mean?</p>
<pre><code class="lang-auto">
def crop_img(img, x_tuple, y_tuple):
    return img[:,:,x_tuple[0]:x_tuple[1], y_tuple[0], y_tuple[1]]

x = torch.rand(1, 3, 500, 500)
x_tuple=(100, 200)
y_tuple=(100, 200)
print(crop_img(x, x_tuple, y_tuple).size())
</code></pre>

username: MetallicMonkeyu
cooked: <p>I think this works. But I forgot to mention that my range tuple is also in tensor format (which has shape of (B, 4), where B is number of image in batch and 4 is (x_min, y_min, x_max, y_max) since each image might have different crop range. Is there a convenient way to apply this range tensor to the image tensor (B, C, H, W) and crop accordingly or I have to loop each range tensor and apply your slicing method to corresponding image?</p>

username: J_Johnson
cooked: <p>If we assume the output sizes will be the same on dim=(2,3) then you could do:</p>
<pre><code class="lang-auto">cropped_imgs = torch.stack([img_batch[ i , : , map[i,0] : map[i,1] , map[i,2] : map[i,3]] for i in range(map.size(0))])

</code></pre>
<p>Note: wrote this on my phone, so it might be missing a bracket or have a typo.</p>

username: MetallicMonkeyu
cooked: <p>Alright thank you! I will take the loop solution for now. I still have a question on cropping. Let’s say if we already know a range, for example, (500,500) and I have an image of (496,504), if I want to use a mask to padding missing pixels and crop additional pixels so that reshape the image to (500,500). Is there anyway to do this by mask?</p>

username: J_Johnson
cooked: <p>You could use a combination of crop and pad from torchvision:</p>
<p><a href="https://pytorch.org/vision/stable/transforms.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/vision/stable/transforms.html</a></p>