akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
mIOU decreases with increasing epoch

Rakibul_Hasan_Rajib: <p>I am training a model for semi-supervised semantic segmentation using resnet as backbone and feature pyramid network as decoder .Cityscapes dataset is used. As epoch increases, mean IOU and class average accuracy decreases but overall accuracy increases. Cross entropy loss also decreases. I have kept learning rate very small 1e-5. What might be the reason behind this?<br>
Epoch1: CE Loss : 3.07788 Overall ACC: 0.8059 | Mean ACC: 0.3714 | mIoU: 0.2459<br>
Epoch2: CE Loss : 2.74041 Overall ACC: 0.8323 | Mean ACC: 0.3008 | mIoU: 0.2322<br>
Epoch3: CE Loss : 2.54747 Overall ACC: 0.8435 | Mean ACC: 0.2959 | mIoU: 0.2302<br>
Epoch4: CE Loss : 2.50084 Overall ACC: 0.8490 | Mean ACC: 0.2842 | mIoU: 0.2173</p>
<p>Here’s the code for calculating accuracy and mean iou</p>
<pre><code class="lang-auto">def _fast_hist(label_true, label_pred, n_class):
    mask = (label_true &gt;= 0) &amp; (label_true &lt; n_class) # Exclude unlabelled data.
    hist = np.bincount(n_class * label_true[mask] + label_pred[mask],\
                       minlength=n_class ** 2).reshape(n_class, n_class)
    
    return hist


def scores(label_trues, label_preds, n_class):
    hist = np.zeros((n_class, n_class))
    for lt, lp in zip(label_trues, label_preds):
        hist += _fast_hist(lt.flatten(), lp.flatten(), n_class)
    return hist


def get_result_metrics(histogram):
    tp = np.diag(histogram)
    fp = np.sum(histogram, 0) - tp
    fn = np.sum(histogram, 1) - tp 

    iou = tp / (tp + fp + fn)
    prc = tp / (tp + fn) 
    opc = np.sum(tp) / np.sum(histogram)

    result = {"iou": iou,
              "mean_iou": np.nanmean(iou),
              "precision_per_class (per class accuracy)": prc,
              "mean_precision (class-avg accuracy)": np.nanmean(prc),
              "overall_precision (pixel accuracy)": opc}


    return result
type or paste code here
</code></pre>

KFrank: <p>Hi Rakibul!</p>
<aside class="quote no-group" data-username="Rakibul_Hasan_Rajib" data-post="1" data-topic="173656" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/rakibul_hasan_rajib/48/57734_2.png" class="avatar"> Rakibul_Hasan_Rajib:</div>
<blockquote>
<p>mean IOU and class average accuracy decreases but overall accuracy increases.</p>
</blockquote>
</aside>
<p>The mismatch between your class-average and “overall” accuracies suggests<br>
that you have an unbalanced dataset.  How many classes do you have, and<br>
what are the pixel counts (or percentages) for each class?  Be alert to the<br>
possibility that your “background” class could be highly overrepresented.</p>
<p>If your dataset is unbalanced, consider using <code>CrossEntropyLoss</code>’s <code>weight</code><br>
constructor argument to account for the imbalance.</p>
<p>Best.</p>
<p>K. Frank</p>
How to do tensor binary erosion operation in Pytorch

qiminchen: <p>Suppose I have a tensor <code>a</code>, are there any pytorch (binary) <code>erosion</code> operations that can give tensor like <code>b</code>, so basically similar to <a href="https://docs.scipy.org/doc/scipy/reference/generated/scipy.ndimage.binary_erosion.html" rel="noopener nofollow ugc">scipy.ndimage.binary_erosion</a></p>
<pre><code class="lang-auto">print(a)
tensor([[0, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0],
        [0, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0],
        [0, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
        [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
        [1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1],
        [1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1],
        [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
        [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
        [0, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0],
        [0, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0],
        [0, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0]])

print(b)
tensor([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
        [0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0],
        [0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0],
        [0, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0],
        [0, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 0],
        [0, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 0],
        [0, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 0],
        [0, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 0],
        [0, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0],
        [0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0],
        [0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0],
        [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]])
</code></pre>

KFrank: <p>Hi Qimin!</p>
<aside class="quote no-group" data-username="qiminchen" data-post="1" data-topic="173630" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/qiminchen/48/20829_2.png" class="avatar"> qiminchen:</div>
<blockquote>
<p>are there any pytorch (binary) <code>erosion</code> operations</p>
</blockquote>
</aside>
<p>If I understand your example tensors correctly, you want the following:</p>
<p>If a pixel and its 4-connected (NSEW) pixels are all <code>1</code>, the resulting pixel<br>
value should be <code>1</code> and <code>0</code> otherwise.</p>
<p>Build a 3x3 convolution kernel with zeros in the corners and ones for the<br>
center and 4-connected entries.  Initialize the <code>weight</code> of a <code>Conv2d</code> (no <code>bias</code>)<br>
with this kernel, pad the border of your input tensor, convolve it, and test<br>
the resulting pixels against <code>5</code>:  <code>== 5</code> → <code>1</code> and <code>&lt; 5</code> → 0.</p>
<p>Best.</p>
<p>K. Frank</p>
Activation layer insert to class constructor as input(what data-type(=class) should I choose)

DongkyuLee: <p>Hi</p>
<p>I have questions while porting Swin-V2 model to c++ code with Libtorch.</p>
<p>I want to insert activation function to class constructor input, but I have no idea to know what data-type(class) should I write down.</p>
<p>Here is Python class<br>
<em><strong>class Mlp(nn.Module):</strong></em><br>
<em><strong>def <strong>init</strong>(self, in_features, hidden_features=None, out_features=None, act_layer=nn.GELU, drop=0., norm_layer=None, mlpfp32=False):</strong></em></p>
<p>Points are act_layer and norm_layer, they has a lot of variations. for example act_layer sometime it is RELU, sometime it is GELU.</p>
<p>Now, I don’t know what kind of data-type should I choose when I make the new class on C++.</p>
<p>Here is wrong case constructor on C++<br>
<em><strong>MlpImpl(int64_t in_features, int64_t hidden_features, int64_t out_features, torch::nn::ModuleHolder act_layer, float drop = 0.0, torch::nn::Module norm_layer, bool mlpfp32 = false);</strong></em></p>
<p>torch::nn::ModuleHolder is based on “auto”, that’s why I can’t use these.</p>
<p>Please let me know what data-type should I use.</p>
<p>Thanks.</p>
No implementation found for 'torch.index_select' on types that implement __torch_function__: [<class '__main__.Vertex'>]

abhi_khoyani: <p>I’m creating a new subclass using torch.Tensor and as mentioned in docs, overriding all the torch API using <strong>torch_function</strong> as below but still I’m getting above error. I checked torch.index_select is working if I run it directly on instance as <code>torch.index_select(x, 0, torch.tensor([0,1]))</code> considering <code>x</code> as a <code>Vertex</code> class.</p>
<pre><code class="lang-auto">class Vertex(torch.Tensor):
    def __init__(self, data):
        '''
        Theta shall be in radians. List is not supported yet.
        '''
        assert data.ndim in [1,2]
        assert len(data) == 4 if data.ndim==1 else len(data[0])==4, "Please follow g2o format"
        self.id = self[0]
        self.x = self[1]
        self.y = self[2]
        self.theta = self[3]
        super().__init__()
    
    @classmethod
    def __torch_function__(cls, func, types, args=..., kwargs=None):
        return super().__torch_function__(func, types, args, kwargs)

</code></pre>
<p>Any solution of this error?</p>
DDP crash after fix number of iterations

yytang220: <p>I use ddp for distributed training. After around 1022 iterations (changing batch size will result in different but fixed crash iteration), the program will crash with error log: “an illegal memory access was encountered”. After I replace all Linear layer with torch.matmul and manully generated weight tensor, the error still there, and it shows that:<br>
" CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling <code>cublasSgemm( handle, opa, opb, m, n, k, &amp;alpha, a, lda, b, ldb, &amp;beta, c, ldc)</code>"</p>
<p>Any idea why it happen?</p>

ptrblck: <p>I would recommend trying to narrow down the illegal memory access via <code>cuda-gdb</code>, <code>compute-sanitizer</code>, or by creating a CUDA coredump for further debugging. cuBLAS might be the victim of the triggered sticky CUDA failure.<br>
In case you are using an older release I would recommend updating to the latest stable or nightly release.</p>
Dataloader option for pinned memory

JasonFantl: <p>Hello, how can I have a dataloader in c++ load the batches into pinned memory? I have multiple workers loading the batches, is there a way to have these background threads doing the pinning rather than having to pin it myself after popping a batch off the dataloader?</p>
<p>For example, my current dataloader looks like</p>
<pre><code class="lang-cpp">loader =
      torch::data::make_data_loader&lt;torch::data::samplers::RandomSampler&gt;(
          std::move(c), torch::data::DataLoaderOptions().batch_size(batch_size).workers(num_workers).drop_last(true));
</code></pre>
<p>Could I change it to something similar to</p>
<pre><code class="lang-cpp">loader =
      torch::data::make_data_loader&lt;torch::data::samplers::RandomSampler&gt;(
          std::move(c), torch::data::DataLoaderOptions().batch_size(batch_size).workers(num_workers).drop_last(true).pin_memory(true));
</code></pre>
<p>Thanks</p>

ptrblck: <p>I don’t see the <code>pin_memory</code> option listed in <a href="https://github.com/pytorch/pytorch/blob/b818b3fe1c1fa90529b8500cd8ef800bec8415e8/torch/csrc/api/include/torch/data/dataloader_options.h#L12-L40"><code>DataLoaderOptions</code></a> so I would assume your code would fail.<br>
If so, then you might need to manually pin the memory on the host for your data.</p>

JasonFantl: <p>I see in pytorch an option for pinning memory in the dataloader, do you know if it simply hasn’t been added to libtorch yet, or if there’s another approach needed for the workers to pin the memory? I’m trying to avoid rebuilding from source if possible. My goal is to minimize training time, which is why I’m trying to avoid doing the pinning in the main process.</p>

ptrblck: <p>I would assume this functionality wasn’t ported to <code>libtorch</code>. The Python implementation can be found <a href="https://github.com/pytorch/pytorch/blob/b7c2a65139f8083e1cf7eeb454afd15116496ccb/torch/utils/data/dataloader.py#L679-L680">here</a> for the <code>_SingleProcessDataLoaderIter</code> and <a href="https://github.com/pytorch/pytorch/blob/b7c2a65139f8083e1cf7eeb454afd15116496ccb/torch/utils/data/dataloader.py#L1047-L1065">here</a> for <code>_MultiProcessingDataLoaderIter</code>. The docs in this source file can also he helpful to understand how this functionality was implemented.</p>
2x3090 RTX or 1x4090 RTX for training a model?

f10r3: <p>Hi! I have a doubt regarding the combination of GPUs for training the models. I would like to use two 4090 RTX, but I saw that they do not have the NVLink capability. I do not know if it is possible to train a model by using two detached 4090 RTX since they do not have this capability. If I am not wrong, 3090 RTX has this capability (thanks to NVLink), so I suppose it is possible to combine them and use the two-combined GPUs for the training of the model.<br>
If it is not possible, is it better to use 2x3090 RTX with NVLink or only one 4090 RTX? Otherwise, do you have any other proposals?</p>
<p>Thanks <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

rafaelsilva: <p>There’s no need for the NVLink to train models using 2 RTX 4090. Even for the 3090s, the NVlink will speed up p2p communication a bit only, not resulting in a big performance gain.</p>
<p>I’m using 2x 3090 with the NVlink, but considering a single 4090 since it simplifies things.</p>
<p>If you have the budget for 2x 4090, this is the correct choice today.</p>
Question about FCNResnet50 pretrained weights on torchhub

Harsh_Mankodiya: <p>I wanted to use a pretrained FCNResnet50 model to run segmentation task on Pascal VOC dataset. However, when I import the <a href="http://pytorch.org/vision/main/models/generated/torchvision.models.segmentation.fcn_resnet50.html" rel="noopener nofollow ugc">fcn_resnet50 — Torchvision main documentation (pytorch.org)</a> weights and run inference on Pascal VOC validation dataset, I get nowhere near to the performance that is described <a href="http://pytorch.org/vision/main/models/generated/torchvision.models.segmentation.fcn_resnet50.html" rel="noopener nofollow ugc">here</a>.<br>
So, my question is whether the above mentioned FCNResnet50 actually trained on Pascal VOC dataset or am I missing something ?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/6/36ead541e5d56f659fc59454ae40a0939a5ce710.png" data-download-href="https://discuss.pytorch.org/uploads/default/36ead541e5d56f659fc59454ae40a0939a5ce710" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/6/36ead541e5d56f659fc59454ae40a0939a5ce710_2_690x472.png" alt="image" data-base62-sha1="7POUtmFdEJpEk0pactZBdgYu0z6" width="690" height="472" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/6/36ead541e5d56f659fc59454ae40a0939a5ce710_2_690x472.png, https://discuss.pytorch.org/uploads/default/optimized/3X/3/6/36ead541e5d56f659fc59454ae40a0939a5ce710_2_1035x708.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/3/6/36ead541e5d56f659fc59454ae40a0939a5ce710.png 2x" data-dominant-color="F4F3F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1134×776 72.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Training with heavily sparse vectors

bryanpiguave1: <p>I would like train an autoencoder that receives sparse vectors. For example,<br>
a sample of my dataset could be [1 1 1 0 0 0 0 0 …]  with more than 8000 zeros. Is there an efficient way to build a mlp model that can make us of high dimensional coordinate lists so I can train it in that way?</p>
Attn_mask in nn.MultiheadAttention

bogdan_ka: <p>Hi All,</p>
<p>Trying to understand why the mask does not seem to work. There is a similar question <a href="https://discuss.pytorch.org/t/use-attn-mask-to-enforce-causality-in-multiheadattention/96410">here</a> that seems to resolve the issue. But same code below doesn’t return the 0s in upper triangle of the attention matrix. Torch version is 1.13.1.</p>
<pre><code class="lang-auto">inputs = torch.ones((4,3))
maten = nn.MultiheadAttention(embed_dim=3, num_heads=1)
attn_mask = torch.tril(torch.ones((4,4)))
attn_mask = attn_mask&gt;0
query2,_ = maten(inputs, inputs, inputs, attn_mask=~attn_mask)
query2

</code></pre>
<pre><code class="lang-auto">tensor([[-0.0128,  0.0919,  0.1716],
        [-0.0128,  0.0919,  0.1716],
        [-0.0128,  0.0919,  0.1716],
        [-0.0128,  0.0919,  0.1716]], grad_fn=&lt;SqueezeBackward1&gt;)```</code></pre>

bogdan_ka: <p>I think I might be wrong, because I expected 0s in upper triangle of the attention matrix. But I think that’s wrong. Since attn_mask is added to query*key product and then it’s a dot product between that and values, it should not be 0s in upper triangle?</p>
<p>I think it’s basically this and the term before ‘@’ has 0 above diagonal. Would appreciate if anyone could confirm.</p>
<pre><code class="lang-auto">attn_mask[attn_mask == 0]  =  -float("Inf")
softmax((Q@K.transpose(1,0)) +  attn_mask) @ V
</code></pre>

Theo: <p>Attention mask is a lower triangular matrix, and the entries that are non-zero should be -inf.</p>
<pre><code class="lang-auto">arr = [[-np.inf for _ in range(size)] for _ in range(size)]
arr = torch.tensor(arr)
mask = torch.triu(arr, diagonal=1) # Here diagonal=1 is used to make diagonal entries 0
</code></pre>
<p>When you print the mask, you get:</p>
<pre><code class="lang-auto">tensor([[0., -inf, -inf,  ..., -inf, -inf, -inf],
        [0., 0., -inf,  ..., -inf, -inf, -inf],
        [0., 0., 0.,  ..., -inf, -inf, -inf],
        ...,
        [0., 0., 0.,  ..., 0., -inf, -inf],
        [0., 0., 0.,  ..., 0., 0., -inf],
        [0., 0., 0.,  ..., 0., 0., 0.]])
</code></pre>
<p>Size variable here depends on the dimensions of your <em>Q</em> and <em>K</em> matrices.<br>
Your application of the mask is correct, it is added to the matrix multiplication of <em>Q</em> and <em>K</em>. However, you must <strong>scale</strong> <code>Q@K.T</code> before the masking.</p>

bogdan_ka: <p>Thank you <a class="mention" href="/u/theo">@Theo</a>, that makes total sense. Since we are adding the two matrices we want 0s on the lower triangle.</p>
How to return pd.DataFrame from DataLoader

MikeTensor: <p>My CustomDataset returns:<br>
image, target (which is a Pandas DataFrame).</p>
<p>Using the DataLoader throws an error:</p>
<pre><code class="lang-auto">TypeError: default_collate: batch must contain tensors, numpy arrays, numbers, dicts or lists; found … Pandas bla
</code></pre>
<p>Any clue how to fix this?</p>
<p>If I return target as a normal list, and convert the DataLoader target-Output to pd.Dataframe - the format is pretty strange. Instead of ‘Car’ I get '[‘Car]’</p>
<p>This didn’t happened before I have used the DL.<br>
Any idea how to use the DL with pd.DF as return OR how to fix the format?</p>
<p>– Any help would be appreciated.</p>

MikeTensor: <p>batch_size is –and always will be– 1.<br>
collate_fn = None – doesn’t have any effect.</p>

ptrblck: <p>Follow-up is also tracked <a href="https://discuss.pytorch.org/t/torch-dataloader-gives-torch-tensor-as-ouput/31084/12">here</a>. Let me know if transforming the data to tensors would work for your use case.</p>
Understanding GPU capacity and model size with large images

NearsightedCV: <p>Hi all</p>
<p>I am trying to understand a bit better why you run out of memory on the GPU. Specifically I would like to calculate whether something would fit or not before I try and code it up.</p>
<p>This wonderful answer from ptrblk, see link the below.</p>
<p>Gives me a value that is quite low. I don’t question the accuracy but I am missing the impact of the input. I deal with images that are 2000x2000. How do I combine facts about my input with a model size of say 400MB?</p>
<p>Based on model size I should be fine running on a 16GB GPU but I know that doesn’t work.</p>
<p>How can I get some back on envelope calculations based the data I have?</p>
<p>(<a href="https://discuss.pytorch.org/t/finding-model-size/130275/2" class="inline-onebox">Finding model size - #2 by ptrblck</a>)</p>

ptrblck: <p>The parameters and buffers could use a small fraction of the overall memory usage depending on the model architecture. Especially for e.g. conv layers this could be true as their parameters (<code>weight</code> and <code>bias</code>) are often tiny compared to the intermediate forward activations.<br>
<a href="https://discuss.pytorch.org/t/resnet-50-takes-10-13gb-to-run-with-batch-size-of-96/117402/2">This post</a> gives you a better estimate, but also note that additionally to these activations also internal temp. tensors could be created, the CUDA context would use memory on the device, libs could use workspaces (e.g. cuDNN and cuBLAS) etc. which would depend on the actual execution of the model.</p>
What might PyTorch->TensorRT be able to achieve

aifartist: <p>Now that I got a PyTorch build with the latest TensorRT working I’m pondering.<br>
VoltaML uses TensorRT to achieve a huge speed up by ‘accelerating’ the model(ckpt file) into 3 plan files which can then be used.  When I got VoltaML to work I went from 39.5 it/s to 88 it/s.</p>
<p>model = torch.compile(model, <strong>backend=“tensorrt”</strong>)<br>
seems to work but I see no signs of any speed up.  Also, PyTorch 2.0 markets itself as having little to no perf improvements for inference.  Why would a compiled TensorRT model produced by torch not have the same kind of speed up as I see with someone else’s tensorRT compiled version of the same stable diffusion model?</p>

ptrblck: <p>I don’t know how exactly VoltaML creates the computation graph and what is provided to TensorRT, but in general you would see the largest speedup if TensorRT is allowed to optimize the entire graph (or as much as it can).<br>
Different backends could break the computation graph into sub graphs, which could decrease the potential speedup you could achieve.<br>
CC <a class="mention" href="/u/narendasan">@narendasan</a> who might know more details.</p>

aifartist: <p>I should have mentioned that VoltaML takes 20 minutes for their one time generation of the saved plan.  Also, later after I posted this I found something called “Torch-TensorRT” which is its own package that claims up to 6x faster inference.  This was announced in Dec 02 2021 on NVidia’s own dev blog site:</p>
<h1>
<a name="accelerating-inference-up-to-6x-faster-in-pytorch-with-torch-tensorrt-1" class="anchor" href="#accelerating-inference-up-to-6x-faster-in-pytorch-with-torch-tensorrt-1"></a>Accelerating Inference Up to 6x Faster in PyTorch with Torch-TensorRT</h1>

narendasan: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Is right, it really comes down to how the graph is broken up. I don’t know how VoltaML splits up the graph or if it uses some third party IR like ONNX but for Torch-TensorRT it comes down to support for converting operations into TRT ops. If ops don’t have a way to be converted, they are left in PyTorch. There is also may be cost associated with switching between PyTorch and TensorRT execution many times.</p>
Dataset Class for sequential data

0x97: <p>Dear all,<br>
I’m dealing with sequential data, and I would like to find a way to use the torch Dataset class, but I cannot find any example for sequential data.</p>
<p><strong>Context</strong><br>
My dataset is a stack of M matrices, where each matrix is a high-dims time-series of shape (N,T) (T=time series length), resulting in a dataset tensor of shape (M,N,T).<br>
The model operates in a one-to-many style, meaning that i feed a tensor of shape (N), and it outputs a tensor of shape (N,t), with t&lt;T the sequence length.<br>
The idea during training is to extract a batch of timeseries of size m&lt;M, forming tensors of shapes (m,N,t), with initial values (m,N).</p>
<p><strong>Question</strong><br>
Given the above context, is the Dataset class a good way to operate? I’ve already written a class myself with some methods for dealing with my specific case, but I’ve read the Dataset class could be used to parallelize training, since the main problem is loading the entire tensor on RAM, which in certain cases could be too expensive.</p>

Jamie_Donnelly: <p>A <code>Dataset</code> class is definitely useful for parallel training since the <code>DistributedSampler</code> used in the <code>DataLoader</code> constructor will handle backend co-ordination using <code>rank</code> etc.</p>
<p>All you really need to implement for a simple <code>Dataset</code> class are <code>__len__</code> and <code>__getitem__</code> methods. Depending on how your data is stored on disk you don’t need to load all the data into memory at once.</p>
<p>E.g., if you save your data for each timestep on disk as a <code>tuple</code> like (input, target) at <code>.../data_time_i.pt</code> you can implement something as simple as,</p>
<pre><code class="lang-auto">class TestDataset(Dataset):
    def __init__(self,root):
         self.files = [os.path.join(root,i) for i in os.listdir(root)]

    def __len__(self):
         return len(self.files)

    def __getitem__(self, index: List):
          x, y = torch.load(self.files[index])
          return x, y
</code></pre>
<p>Obviously how your <code>Dataset</code> class looks will depend a lot on how your data is currently stored on disk, but above code is easily changed to reflect minor changes, such as if it’s saved in numpy arrays or csv files, etc.</p>
Create an abstract dataset class

mttbrv: <p>Hello, I’m currently trying to create three dataset classes to manage my training, validation and test data. The rationale behind this choice is that the data in the three splits need to be manipulated in different ways. I’m not quite sure how to go about this and would appreciate your feedback on my current implementation.</p>
<p>My idea was to create an <code>AbstractDataset</code> class that inherits both from <code>ABC</code>, so it can have abstract methods, and from <code>Dataset</code>. As you can see from the sketch below the three dataset classes I need to implement would all inherit <code>AbstractDataset</code> and define their own <code>__getitem__</code> method. Is this a sensible approach or can I achieve my goal in a cleaner way?</p>
<pre><code class="lang-auto">class AbstractDataset(ABC, Dataset):
    def __init__(self, csv_file: Path, checkpoint: str) -&gt; None:
        ... &lt;common to all 3 classes&gt; ...

    def _tokenize_sample(self, sample: List[str]) -&gt; BatchEncoding:
        ... &lt;common to all 3 classes&gt; ...

    def __len__(self) -&gt; int:
        ... &lt;common to all 3 classes&gt; ...

    def _generate_embedding(self, tokenized: BatchEncoding) -&gt; torch.Tensor:
        ... &lt;common to all 3 classes&gt; ...

    @abstractmethod
    def __getitem__(self, index: int):
        return NotImplementedError
</code></pre>

Jamie_Donnelly: <p>This looks like a very sensible implementation if the aim is to develop vastly different <code>__getitem__</code> methods.</p>
<p>If by ‘manipulated in different ways’ means different transformations on the partitions, then you could just <code>Subset</code> a single <code>Dataset</code> object, copy the underlying datasets and then manually set different transformations using <code>partition1.transforms = ..., partition2.transforms...,</code></p>
Torch.onnx.export and pack_sequence/unpack_sequence

Pirvu_Mihai: <p>So, I spent a few (good) hours this weekend getting a model exported via <code>torch.onnx.export</code> which contains <code>pack_sequence</code> and <code>unpack_sequence</code>. I managed to get it working after a while and only after rewriting these two little functions myself directly using tensors/for loops.</p>
<p>Made a small repo, perhaps someone might be intersted in the future until this is supported ootb: <a href="https://gitlab.com/mihaicristianpirvu/torch-pack-sequence-onnx-export/-/tree/master/" class="inline-onebox" rel="noopener nofollow ugc">Files · master · Pîrvu Mihai Cristian / torch-pack-sequence-onnx-export · GitLab</a></p>
<p>TL;DR</p>
<ul>
<li>pack_sequence → manually pad with zeros and make a for loop</li>
<li>unpack_sequence → take the last step from the output of the rnn model</li>
<li>only works after you jit the model</li>
<li>list of tensor inputs are required and proper typing is necessary throughout the code</li>
</ul>
<p>As a sidenote: <code>onnxruntime</code> works well to pass <code>List[Tensor]</code> as inputs, but <code>onnxruntime-web</code> doesn’t, but that is a different runtime/project, outside of torch: <a href="https://github.com/microsoft/onnxruntime/discussions/14828" class="inline-onebox" rel="noopener nofollow ugc">How do I pass a list of tensors in onnxruntime-web? · Discussion #14828 · microsoft/onnxruntime · GitHub</a></p>
Randomised freezing of layers while training

Jamesswiz: <p>Suppose my model has 3 convolutional layers where I want to randomly train all or only one of the layers during forward pass.</p>
<pre><code class="lang-auto">class myConv(nn.Module):
    self.conv1 = nn.conv2d()
    self.conv2 = nn.conv2d()
    self.conv3 = nn.conv2d()
def forward(x):
    z = torch.randint(0,4,(1,))
    if z == 0:
       out = self.conv1(x)*self.conv2(x)*self.conv3(x) 
    elif z == 1:      
       out = self.conv1(x)
    elif z == 2:      
       out = self.conv2(x)
    else:
       out = self.conv3(x)                
    return out
</code></pre>
<p>How, do I correctly freeze the weights of the conv layers so that I can train this kind of a layer.<br>
I would prefer ways that can be coded inside the forward pass rather than in training loop since ‘myConv’ is a part of much larger model.<br>
Note that at test time value of ‘z’ will be fixed so I am concerned about training phase only.</p>
<p>Any other suggestions are also welcomed.<br>
Thanks in advanced.</p>

ptrblck: <p>In the branches where layers are not used you might not need to freeze their parameters since Autograd won’t compute gradients for them.<br>
Since these parameters were not used in the forward pass they were never added to the computation graph and are thus irrelevant in the backward pass.<br>
If you explicitly want to freeze these parameters nevertheless, setting their <code>.requires_grad</code> attribute to <code>False</code> is the right approach.</p>
<p>Also, note that optimizers with running internal states might still update frozen parameters even with a zero gradient. To avoid it you should delete their <code>.grad</code> attribute or use <code>.zero_grad(set_to_none=True)</code>.</p>

Jamesswiz: <p>Thanks for the confirmation. The model was indeed training fine but I just wanted a validation if I am doing it right or is their a better efficient way. I found <code>.requires_grad</code> True/False as desired needs to be set explicitly within each if-else condition otherwise the state from previous batch will apply to next batch and the model will not learn.</p>
<p>Your pointer about <code>.zero_grad(set_to_none=True)</code> is something new to me and a nice way to save compute.</p>
Creating a batch of patches out of a single image

JungHwang: <p>I am wondering if there are any vectorized methods for creating a batch patches out of a single image.</p>
<p>Suppose I have an image. (a Tenosr of size C, H, W)<br>
And I am trying to create a specific number of randomly cropped patches, say 16.<br>
The final shape would be [number of patches (16), number of channels (3), height after cropped, width after cropped]</p>
<p>Then the most straightforward way might be looping each patch, 16 times in this case,<br>
then create each patch with something similiar to the following snippet.</p>
<pre><code class="lang-python">patches[loop_idx] = img[:, min_h : max_h, min_w : max_w]
</code></pre>
<p>I guess this will do what I need.</p>
<p>I, however, am wondering if there are any vectorized ways that do the same thing so that I don’t have to loop the number of patches.</p>
Extracting sliding window patches using indices rather than unfold

George_Killick: <p>Hi,</p>
<p>I’m working on a problem where I am trying to perform convolution on non-uniform pixel arrangements. Because the pixels have no canonical ordering, the input is of shape (Batch, Channels, N_pixels). I construct a KNN graph to determine the indices of the input that each output neuron is connected to which I used to extract ‘sliding window’ patches. I’ve attached an example module that implements the operation.</p>
<pre><code>class GraphConv(nn.Module):
    def __init__(self, in_channels, out_channels, k, in_size):
        super().__init__()
        
        #idx generated from knn_graph
        idx = torch.randint(0, in_size, (in_size, k))
        weights = torch.rand((in_channels, out_channels, in_size, k))
        self.weights = nn.Parameter(weights, True)
        self.register_buffer('idx', idx)
        self.bias = nn.Parameter(torch.zeros(1, out_channels, 1))

    def forward(self, x):
        x = x[:,:,self.idx]
        #b, i, o, n, k - batch, in_channels, out_channels, n_pixels, k
        x = torch.einsum('bink,ionk-&gt;bon', x, self.weights)
        x = x + self.bias
        return x
</code></pre>
<p>If I am understanding correctly, I can’t use the unfold operation to extract sliding windows of the input due to the random ordering of the input pixels. While this works it is ~10x slower than Conv1D. Is there any way to optimise this to run faster. The pixel arrangement of the input is known beforehand and is the same for all input images.</p>
<p>Thanks in Advance</p>
Replace double for loop

ShutCoffee: <p>I have a tensor which contains indices that correspond to nodes in two different graphs that were chosen based of some criteria. So let’s say <code>indices = tensor([[0, 1], [3, 2]]</code>. This means that in the first graph, the node with index 0 and in the second graph the node with index 1 were chosen. Also, in the first graph the node with index 3 and in the second graph the node with index 2 were chosen. This kind of format is necessary in my case.</p>
<p>Now i need to compare each element in <code>indices</code> with each other with a function called <code>is_adjacent</code> to see which pair of nodes i still need.</p>
<p>My current solution:</p>
<pre><code class="lang-auto">    edge_index = []
    for k, i in enumerate(indices):
        for l, j in enumerate(indices):
            if _is_adjacent(i[0], j[0], graph_1) and _is_adjacent(i[1], j[1], graph_2):
                edge_index.append([k,l])
    
    edge_index = torch.tensor(edge_index, dtype=torch.float)
</code></pre>
<p>Where the following function is used:</p>
<pre><code class="lang-auto">def _is_adjacent(id_a, id_b, graph):
    tens = [id_a, id_b]
    adjacency_matrix = torch.reshape(graph.edge_index, (-1,2)).tolist()

    return tens in adjacency_matrix
</code></pre>
<p>This works as it should, but it is not really fast. i would like to replace my for loops with some kind of vectorization.</p>
<p>I hope this makes sense.</p>
Prevent reducing gradients in .backward() across processes DDP

ajrheng: <p>I have a use case where I need to calculate the gradients of parameters of a NN on a per sample basis, i.e., batch size of 1, and I want <code>params.grad</code> after <code>loss.backward()</code>. I want to speed this up with <code>DistributedDataParallel</code>, so I have each process take in a batch of 1 and perform the calculation.</p>
<p>The problem is <code>loss.backward()</code> with DDP automatically reduces the gradients among processes, so that the gradients I get from <code>params.grad</code> for each process will be the same. What I need are the individual gradients without reducing (batch size 1, rather than averaged over the N x 1 data points over N GPUs).</p>
<p>Is there a way to disable DDP from automatically reducing the backward pass? Note that I am not actually optimizing the loss or anything, I am just calculating the first order derivative for a trained model (to calculate the Fisher Information Matrix if anyone is wondering).</p>
<p>Thanks!</p>

ptrblck: <p>I think <a href="https://pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html#torch.nn.parallel.DistributedDataParallel.no_sync"><code>DistributedDataParallel.no_sync()</code></a> should work for your use case:</p>
<blockquote>
<p>A context manager to disable gradient synchronizations across DDP processes. Within this context, gradients will be accumulated on module variables, which will later be synchronized in the first forward-backward pass exiting the context.</p>
</blockquote>

ajrheng: <p>Thank you, this looks like exactly what I need.</p>
How to sample from a defined probability region of a Multi-Variate Normal Distribution using PyTorch?

alexgabriel28: <p>Hi everyone.<br>
Is there a straightforward way to sample from a multivariate normal distribution in PyTorch and only keep the samples that are in a defined low-probability region (e.g. P(x) &lt; 5 %)?</p>
<p>Concrete example: I have a batch of representation tensors as an output of a ResNet with a size of <code>(batch_size, feature_size)</code>.</p>
<p>I can then generate the mean vector <code>prototype</code> with <code>size = (1, feature_size)</code> of those representations, and the corresponding empirical <code>cov</code> matrix with <code>size = (feature_size, feature_size)</code>. From my understanding I can use PyTorchs <code>distributions</code> package to sample from the multivariate normal distribution, defined by <code>prototype</code> and <code>cov</code> like so:</p>
<pre><code class="lang-auto">from torch.distributions.multivariate_normal import MultivariateNormal
import torch.nn as nn

dist = MultivariateNormal(prototype, covariance_matrix = cov)
samples = dist.sample(torch.Size([10000]))
</code></pre>
<p>I’d like to know how to determine the probability region that a sample belongs to, i.e., I only want to keep samples with low probability. I am aware that there is <code>dist.log_prob(value)</code>. However, I’m having a hard time gaining an intuition from that. The outputs don’t seem to make a lot of sense. Any idea what I’m missing here?</p>
<p>Thanks for your help.</p>

Tanmoy_Mukherjee: <p>I do think the that log_prob will work.<br>
So something of this stuff<br>
negative_samples = dis.rsample((1000,))<br>
neg_prob_density = dis.log_prob(negative_samples)</p>
<p>should be fine. The other option is to use k-NN method for the same.</p>
Assignment with advance indexing fails to modify tensor

Rane90: <p>Hi,</p>
<p>I have a tensor of shape <code>(7, 4, 1, 80, 2)</code> and I would like to modify some of its indices. I’ve tried to do this in two different indexing steps.</p>
<ol>
<li>index relevant indices at dimension 1 (of size 4)<br>
<code>x[torch.arange(x.size(0)).unsqueeze(1), idx]</code> where <code>idx</code> is of shape <code>(7,3)</code>, meaning I index <code>3</code> out of <code>4</code> indices.</li>
<li>index relevant indices at dimension 4 (of size 80) which turns my code into:</li>
</ol>
<pre><code class="lang-auto">x[torch.arange(x.size(0)).unsqueeze(1), idx][[
            torch.arange(x.size(0))[:, None, None, None],
            torch.arange(x.size(1) - 1)[:, None, None],
            torch.arange(x.size(2))[:, None],
            inner_idx
        ]]
</code></pre>
<p>where <code>inner_idx</code> is of shape <code>(7,3, 1, 60)</code>, meaning I index <code>60</code> out of <code>80</code> indices</p>
<p>finally I modify the tensor values:</p>
<pre><code class="lang-auto">x[torch.arange(x.size(0)).unsqueeze(1), idx][[
            torch.arange(x.size(0))[:, None, None, None],
            torch.arange(x.size(1) - 1)[:, None, None], # x.size(1) - 1 since 4 turned into 3
            torch.arange(x.size(2))[:, None],
            inner_idx
        ]]  = new_values
</code></pre>
<p>where <code>new_values</code> is of shape <code>(7, 3, 1, 60, 2)</code></p>
<p>However x is not modified.</p>
<p>Is there anyway to solve this issue?</p>
<p>Thanks</p>
<p><strong>Edit:</strong></p>
<p>Reading more on this issue, I think it might have to do with gradients.</p>
<p>I will elaborate that this issue still persists regardless of <code>x.requires_grad = True/False</code>. For <code>new_values</code>, <code>requires_grad</code> is set to <code>True</code>, since I need this for training the model.</p>

ptrblck: <p>You might need to remove the chained indexing and use a single indexing operation to manipulate <code>x</code>.</p>

Rane90: <p>Yes that seems to have worked. Thanks!</p>
Training is very slow after one epoch

Rakibul_Hasan_Rajib: <p>I am trying to train a deep learning model for segmentation using resnet as backbone and feature pyramid network as decoder. The model is being trained on cityscapes dataset. The first epoch gets completed within 30 minutes which is normal I guess given that I am iterating over the train dataset twice in each epoch. However, after one epoch training is very slow and never completes even if 3-4 hours passed.  I am guessing the problem lies on data loading. Here’s my data loader.<br>
sup_loader = data.DataLoader(sup_data, batch_size=args.compute_batch_size,<br>
shuffle=False, num_workers=args.num_workers, pin_memory=True,<br>
collate_fn=collate_eval, worker_init_fn=worker_init_fn(args.seed))<br>
I am training on google colab pro. What might be the reason behind this anamoly?</p>

nkdatascientist: <p><a class="mention" href="/u/rakibul_hasan_rajib">@Rakibul_Hasan_Rajib</a> can you share the train function as well as data loader parameter, because the data loader call looks fine, but if the number of worker is higher then you will face this problem.</p>

Rakibul_Hasan_Rajib: <p>def run_mini_batch_kmeans(args, sup_loader, model,view):</p>
<pre><code>kmeans_loss  = AverageMeter()
faiss_module = get_faiss_module(args)
data_count   = np.zeros(args.num_classes)

featslist   = []
num_batches = 0

sup_loader.dataset.view = view
unsup_loader.dataset.view = view

centroids = np.zeros((args.num_classes,args.in_dim)).astype('float32')

model.train()
with torch.no_grad():
    for i_batch, (indice, image,label) in tqdm(enumerate(sup_loader)):
        image = image.cuda(non_blocking=True)
        feats = model(image)
        label = label.cuda(non_blocking=True)

        if args.metric_test == 'cosine':
            feats = F.normalize(feats, dim=1, p=2)
           
        feats = F.interpolate(feats,image.shape[-2:],mode='bilinear',align_corners=False)
        feats = feature_flatten(feats).detach().cpu().numpy().astype('float32')
        label = label.flatten().type(torch.LongTensor)

        for k in range(args.num_classes):
            idx_k = np.where(label == k)[0]
            data_count[k] += len(idx_k)
            centroids[k] += np.sum(feats[idx_k],axis=0)
            
for k in range(args.num_classes):
    centroids[k] /= data_count[k] + 1e-6
return centroids
</code></pre>
Mask meaning in TransformerDecoderLayer

sachinruk: <p>I am trying to map my understand of the masks used in <code>TransformerDecoderLayer</code> to that of huggingface where <code>attention_mask</code> is used. Suppose I have the following model (and data).</p>
<pre><code class="lang-python">decoder_layer = nn.TransformerDecoderLayer(d_model=64, nhead=8, batch_first=True)
memory = torch.rand(4, 5, 64)
tgt = torch.rand(4, 3, 64)
out = decoder_layer(tgt, memory)
</code></pre>
<p>In the context of a translation task <code>tgt</code> are the embeddings of the translated (output) sentence, and <code>memory</code> is the output of the encoder model. Given that input’s outputs can have different lengths in the sentences, suppose I have the following <code>attention_mask</code>s.</p>
<pre><code class="lang-python">tgt_attention_mask = torch.LongTensor(
    [
        [1, 1, 1, 1, 1],
        [1, 1, 1, 0, 0],
        [1, 0, 0, 0, 0],
        [1, 1, 0, 0, 0],
    ]
)
memory_attention_mask = torch.LongTensor(
    [
        [1, 1, 1],
        [1, 1, 1],
        [1, 0, 0],
        [1, 0, 0],
    ]
)
</code></pre>
<p>Importantly, notice how the sequence length of the <code>tgt</code> is 5, while the <code>memory</code> is of length 3.</p>
<p>From my understanding of decoder transformers, there are two types of attention matrices. 1. Self attention matrix of size (batch_size, 5, 5), 2. Cross attention matrix of size (batch_size, 5, 3). The question is how do I pass these attention masks into the <code>mask</code> arguments in here. There are 4 mask arguments in <code>TransformerDecoderLayer</code> and I’m not sure how these came to be.</p>
<p>Thanks in advance.</p>
<h2>
<a name="aside-1" class="anchor" href="#aside-1"></a>Aside</h2>
<p>Incase you need it the cross attention matrix would look like this:</p>
<pre><code class="lang-auto">cross_attention_mask = tgt_attention_mask.unsqueeze(-1) @ memory_attention_mask.unsqueeze(1)
</code></pre>
How do I see the shape and dtype of a datastructure holding pytorch tensors and normal scalars?

NightMachinery: <p>I am looking for something like the following in PyTorch</p>
<pre><code class="lang-auto">def jax_shape_get(input):
    return jax.eval_shape(lambda x: x, input)
</code></pre>
<p><a href="https://jax.readthedocs.io/en/latest/_autosummary/jax.eval_shape.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://jax.readthedocs.io/en/latest/_autosummary/jax.eval_shape.html</a></p>

NightMachinery: <pre><code class="lang-auto">import jax

def torch_shape_get(input):
    def h_shape_get(x):
        return x.dtype, x.shape

    jax.tree_map(h_shape_get, input)
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/6/568cadc5b08a542ba4fb790d8f26fd02761c5599.png" data-download-href="https://discuss.pytorch.org/uploads/default/568cadc5b08a542ba4fb790d8f26fd02761c5599" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/6/568cadc5b08a542ba4fb790d8f26fd02761c5599_2_632x500.png" alt="image" data-base62-sha1="clEsQY7FaKjwVsZAybAQYmDAkfn" width="632" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/6/568cadc5b08a542ba4fb790d8f26fd02761c5599_2_632x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/5/6/568cadc5b08a542ba4fb790d8f26fd02761c5599_2_948x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/6/568cadc5b08a542ba4fb790d8f26fd02761c5599.png 2x" data-dominant-color="F0F0F1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1200×948 88.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
I have some problems with my video usage

junegao1106: <p><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/8/58c8bc93f5e9f4f6c1e5ab6bf039df0c8668c142.png" alt="截屏2023-02-26 13.11.36" data-base62-sha1="cFq5NZNhW0wiFq7YFmu7kMZDaFk" width="614" height="47"><br>
today I change a server to train my network, but the display of video usage confuse me a lot.<br>
what’s wrong with my code(which part of code should I provide?), and how can I fix it? thanks for your questions and replies!</p>

ptrblck: <p>Could you explain a bit more what issue you are seeing, please?</p>

junegao1106: <p>thanks for your reply, I trained my networks with DDP by 4 cards. As the figure and the command nvidia-smi shows. I have another 3 processes on each card, but their video memory usage is 0. BTW, the same code runs on the other server with 8 A5000 GPUS is normal works without this performance<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/9/0/90de33961ccb4d8ca1232c160c93db248e522724.png" alt="截屏2023-02-26 13.11.09" data-base62-sha1="kFyORal5a028kSgr3gUaOJIMfbu" width="558" height="61"><br>
.</p>

ptrblck: <p>Do you see any training progress on this machine? If so, could you print the <code>.device</code> attribute of some tensors to check if your script is using all GPUs?</p>

junegao1106: <p>I guess it is not the problem caused by some tensors, because of the 0 video memory usage. I think it is caused by some initial settings, such as torch.cuda.set_decice().</p>

junegao1106: <p>if I use N cards I have N-1 processes which has 0 video memory usage.</p>