akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Sampling a xarray dataset into positive and negative samples for self supervised feature learning

username: Balboa
cooked: <p>I have a xarray dataset with 2 variables depicting cloud cover value and precipitation values (coordinates: Latitude, Longitude and Time )which i need to sample into positive and negatives pairs for training the model.</p>
<p>This basically is a part for self supervised feature learning. The sampling has to be done based on region i.e all values lying inside a region is considered as positive pairs and one value inside and one value outside is considered as negative pair for which i would annotate the labels 0 for positive pairs and 1 for negative pairs respectively and thereby feed it to the model.</p>
<p>I’m not sure how to achieve the sampling data with latitude, longitude and time as they are 3D. I’m a beginner in this field, any sort of help would be much appreciated <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>Format of data:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/8/f8cf4bd3b9644f6fb3b9423323e2ea34b24971ee.png" data-download-href="https://discuss.pytorch.org/uploads/default/f8cf4bd3b9644f6fb3b9423323e2ea34b24971ee" title="Screenshot 2022-07-06 at 12.47.10 PM"><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/8/f8cf4bd3b9644f6fb3b9423323e2ea34b24971ee.png" alt="Screenshot 2022-07-06 at 12.47.10 PM" data-base62-sha1="zv4xEfP1WTKpm1VLnRoGhj5yri6" width="690" height="245" data-dominant-color="383838"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2022-07-06 at 12.47.10 PM</span><span class="informations">717×255 16.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
EMA and nn.utils.spectral_norm()

username: schwenn
cooked: <p>I’m trying to keep track of an exponential moving average (EMA) of my model weights, to be used in inference.<br>
I noticed whenever the model contains spectral normalization (using <code>nn.utils.spectral_norm()</code>), the resulting EMA model does not produce expected results.<br>
As <code>spectral_norm()</code> seems to ‘patch’ the wrapped module, adding some parameters, buffers, attributes, etc., I’m not really sure how these should be handled. Averaging trainable parameters, the entire state dict, removing/applying spectral normalization before/after the averaging update, doesn’t seem to work at least.</p>
<p>What would be the correct way to apply EMA to a module with spectral normalization applied?</p>

username: TheshowN
cooked: <p>Hey, I meet the same problem and question. Does you fix it?</p>
Multiscale Prediction Loss

username: Brattley
cooked: <p>Hello, I am currently using a self-made model that extracts feature maps of different scales (currently 256x80x80 and 128x160x160). Next i use two classification heads that each act on one of the feature maps. After this, I just calculate the loss of each head and add them together. Lastly, i use total_loss.backward() where total_loss is the combined loss.</p>
<p>I usually get worse results doing this than just predicting with 1 head. Am I doing something wrong? What  are good practices when dealing with multiscale models?</p>
How to add noise (dithering) at background only?

username: Aarya
cooked: <p>I am trying to train a model with some noisy images having dithering.</p>
<p><strong>What I have :</strong></p>
<ol>
<li>clean pdfs with white background</li>
<li>coloured pdfs(RGB) and grayscale pdfs (with 3 channels, RGB)</li>
</ol>
<p><strong>What I want:</strong></p>
<ol start="3">
<li>
<p>convert only white background (not text) into gray background, if possible only half page should be converted</p>
</li>
<li>
<p>Add dithering to the gray background without loosing the text</p>
</li>
</ol>
<p><strong>what I tried:</strong></p>
<blockquote>
<p>import os</p>
<pre><code>from PIL import Image
from numpy import asarray


ORIGIN_PATH = "/home/dithering/temp/"
DESTIN_PATH = "/home/dithering/temp_try/"
"""for filename in os.listdir(ORIGIN_PATH):
    img = Image.open(ORIGIN_PATH + filename).convert("L")
    rbg_grayscale_img = img.convert("RGB")
    rbg_grayscale_img.save(DESTIN_PATH + filename)"""


for filename in os.listdir(ORIGIN_PATH):

    img = Image.open(ORIGIN_PATH + filename).convert("L", dither=Image.Dither.FLOYDSTEINBERG)

    # convert image to nparray
    numpydata = asarray(img)
    numpydata[numpydata &gt; 250] = 128

    # data
    print(numpydata)

    # convert array to image
    final_image = Image.fromarray(numpydata)

    # img show
    final_image.show()

    # img save
    final_image.save(DESTIN_PATH + filename)
</code></pre>
</blockquote>
<p>I expect something like this, <a href="https://i.stack.imgur.com/JIvtj.png" rel="noopener nofollow ugc"><img src="https://i.stack.imgur.com/JIvtj.png" alt="enter image description here" width="690" height="265"></a></p>
<p>Any help would be appreciated, thanks in advance! <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
How to make symbolic function at this torch.augtograd.Function

username: ims
cooked: <p>I tried to export <a href="https://github.com/wuhuikai/FastFCN" rel="noopener nofollow ugc">FastFCN</a> model to onnx however it failed since…</p>
<pre><code class="lang-auto">RuntimeError: ONNX export failed: Couldn't export Python operator ScaledL2

Defined at:
/home/im/FastFCN/encoding/nn/encoding.py(37): forward
/home/im/anaconda3/envs/regseg/lib/python3.7/site-packages/torch/nn/modules/module.py(1039): _slow_forward
/home/im/anaconda3/envs/regseg/lib/python3.7/site-packages/torch/nn/modules/module.py(1051): _call_impl
/home/im/anaconda3/envs/regseg/lib/python3.7/site-packages/torch/nn/modules/container.py(139): forward
/home/im/anaconda3/envs/regseg/lib/python3.7/site-packages/torch/nn/modules/module.py(1039): _slow_forward
/home/im/anaconda3/envs/regseg/lib/python3.7/site-packages/torch/nn/modules/module.py(1051): _call_impl
</code></pre>
<p>The ScaledL2 function is like this.</p>
<pre><code class="lang-auto">class ScaledL2(autograd.Function):
    @staticmethod
    def forward(ctx, X, C, S):
        SL = (X.unsqueeze(2).expand(X.size(0), X.size(1), C.size(0), C.size(1)) -
              C.unsqueeze(0).unsqueeze(0)).pow_(2).sum(3).mul_(S.view(1, 1, C.size(0)))
        ctx.save_for_backward(X, C, S, SL)
        return SL

    @staticmethod
    def backward(ctx, GSL):
        X, C, S, SL = ctx.saved_variables

        tmp = (X.unsqueeze(2).expand(X.size(0), X.size(1), C.size(0), C.size(1)) - C.unsqueeze(0).unsqueeze(0)).mul_(
            (2 * GSL).mul_(S.view(1, 1, C.size(0))).unsqueeze(3)
        )

        GX = tmp.sum(2)
        GC = tmp.sum((0, 1)).mul_(-1)
        GS = SL.div(S.view(1, 1, C.size(0))).mul_(GSL).sum((0, 1))

        return GX, GC, GS

scaled_l2 = ScaledL2.apply
__all__ = [ 'scaled_l2']
</code></pre>
<p>I searched for solution and most said <a href="https://pytorch.org/docs/1.10.0/onnx.html#torch-autograd-functions" rel="noopener nofollow ugc">this</a> will solve my problem. Therefore, I tried to make symbolic static method to ScaledL2, but I have no idea with torch._C<br>
Please let me know how to make symbolic method to this ‘ScaledL2’</p>
<p>Sincerely.</p>
Are score threshold and nms threshold hyperparameters?

username: Ian_Scott
cooked: <p>Hi</p>
<p>I am training an object detection model (retinanet) which has score threshold and nms threshold, as I understand other models do also.  The documentation states these are part of postprocessing.  Does postprocessing happen before the loss gets calculated, or is it something that happens when the model is actually performing inference?</p>
<p>What I am trying to understand is if these two values should be used as hyper parameters and tuned during training, OR are they something to play around with on your trained model.</p>
<p>Sorry if this is a really beginner question!</p>
<p>Many thanks<br>
Ian</p>
ValueError: Only sparse matrices with 32-bit integer indices are accepted. Got int64 indices

username: cola888666
cooked: <p>when  training svm by sklearn with fashion_mnist dataset .<br>
ValueError: Only sparse matrices with 32-bit integer indices are accepted. Got int64 indices.<br>
Can you help me? Thanks!</p>

username: ptrblck
cooked: <p>It seems you are running into a <code>scikit-learn</code> error, which doesn’t seem to be PyTorch-related, so you would get a better and faster answer in a scikit-specific board.<br>
In any case, check the <code>dtype</code> of the input values to this classifier, compare it to the docs, and make sure they are valid.</p>

username: cola888666
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="155944">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>scikit-specific</p>
</blockquote>
</aside>
<p>import hub</p>
<p>ds = hub.load(‘hub://activeloop/caltech101’)</p>
<p>dataloader = ds.pytorch(num_workers=0, batch_size=4, shuffle=False)</p>
<p>hi, i want to train model with this dataset.but how to i split this data set into training set and test set?</p>
[Solved, Self Implementing] How to return sparse tensor from nn.Embedding()

username: ReLuTropy
cooked: <p>Hi.</p>
<p>I want to get the sparse tensor when LongTensor is passed to Embedding Layer.</p>
<p>I want to get adjacency matrix as shape (N,N), and it has value at (i,j) from Embedding Layer.<br>
There are a lot of zeros in the adjacency matrix, so I want to get sparse tensor.</p>
<p>how to do that?</p>

username: ReLuTropy
cooked: <p>I made it.</p>
<p>Assumption</p>
<p>The input is LongTensor which contains indices for embedding as shape has (batchsize, N * N).<br>
In default, embedded value is 0 if index is 0. If you want to change this index, change “padding_idx” any number.<br>
The output is adjacency sparse matrix as shape has (batchsize, N, N).</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/e/eb188162034636bd147b76a90f4506f8b8a51e1b.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/eb188162034636bd147b76a90f4506f8b8a51e1b" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/e/eb188162034636bd147b76a90f4506f8b8a51e1b_2_572x500.jpeg" alt="image" data-base62-sha1="xxKHU4OG1FRykV9Zp9Vj0YRwZV9" width="572" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/e/eb188162034636bd147b76a90f4506f8b8a51e1b_2_572x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/2X/e/eb188162034636bd147b76a90f4506f8b8a51e1b_2_858x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/e/eb188162034636bd147b76a90f4506f8b8a51e1b_2_1144x1000.jpeg 2x" data-dominant-color="F8F8F9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1668×1456 368 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><a href="https://github.com/shllln/SparseEmbedding" class="onebox" target="_blank" rel="noopener nofollow ugc">https://github.com/shllln/SparseEmbedding</a></p>
<p>thank you.</p>

username: pintonos
cooked: <p>I want to do the exact same thing. However, the repository is not available anymore…any ideas how to do this efficiently?</p>
How to visualize GraphModule

username: MartinZhang
cooked: <p>Hi all,</p>
<p>I  want to visualize the torch.fx.GraphModule , how to do it?</p>

username: AlphaBetaGamma96
cooked: <p>You could give <a href="https://github.com/szagoruyko/pytorchviz" rel="noopener nofollow ugc">torchviz</a> a go and see if that works?</p>

username: ParGG
cooked: <p>You can use this: <a href="https://github.com/pytorch/pytorch/blob/master/torch/fx/passes/graph_drawer.py" class="inline-onebox" rel="noopener nofollow ugc">pytorch/graph_drawer.py at master · pytorch/pytorch · GitHub</a></p>

username: MartinZhang
cooked: <p>Thank you so much, it worked for me.</p>
Can libtorch run on the bare-metal device?

username: m53949218.cs10
cooked: <p>Hi, I found some information on the release page that pytorch builds can support the Newlib c library<br>
“Added Pytorch build support with Newlib c library (<span class="hashtag">#60345</span>, <span class="hashtag">#60052</span>)”<br>
URL: <a href="https://github.com/pytorch/pytorch/releases" class="inline-onebox" rel="noopener nofollow ugc">Releases · pytorch/pytorch · GitHub</a></p>
<ol>
<li>
<p>Does this mean we can infer torchscript model with libtorch on the bare-metal device (without OS)?</p>
</li>
<li>
<p>I found another message saying “Glow builds and runs on macOS and Linux”, does this conflict with the release notes I mentioned above (“Pytorch build support with Newlib c library”)?<br>
REF URL: <a href="https://discuss.pytorch.org/t/code-generation-for-pytorch-c-bare-metal-without-os/27716/2" class="inline-onebox">Code Generation for PyTorch C++ Bare Metal (without OS)? - #2 by pkmital</a></p>
</li>
</ol>
<p>It would be grateful if you could help with these doubts</p>
<p>Thanks.</p>
Dataloader shared list across workers with ddp

username: JonasFrey96
cooked: <p>Hello,<br>
I have a question regarding dataloaders and implementing costom datasets in PyTorch:<br>
What is the best way to share data between mutiple dataloader workers.<br>
I would like to sometimes instead of  returning the index that is provided in def <strong>getitem</strong>(self, index) return a different index with a certain probability. The goal is to implement an efficient replay buffer fully in the dataloader.<br>
I would like to have a way to set a list of possible indexes to rehearsel in form of a list in each dataloader. Also each dataloader should be able to append indexe to the list.</p>
<pre><code class="lang-auto">def __getitem__(self, index):
   flag = True
   if random.random() &gt; 0.5:
       # rehearsel element
       index = random.choice( self._shared_list_between_all_workers-ddp )
   elif random.random() &gt; 0.05:
       if not index in self._shared_list_between_all_workers-ddp:
           # add to the list with a 5% chance if not rehearseld
           self._shared_list_between_all_workers-ddp.append(index)

   ### do stuff

</code></pre>
<p>Best Jonas</p>

username: JonasFrey96
cooked: <p>Okay I looked a little bit into it and using simple multiprocessing arrays does the job.<br>
<a href="https://docs.python.org/3/library/multiprocessing.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://docs.python.org/3/library/multiprocessing.html</a></p>

username: Pietro_Cicalese
cooked: <p>This was done in DP. See here:</p><aside class="quote quote-modified" data-post="9" data-topic="11473">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pietro_cicalese/48/27365_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/communicating-with-dataloader-workers/11473/9">Communicating with Dataloader workers</a> 
  </div>
  <blockquote>
    I looked everywhere and worked on this problem for a few days before solving it. Basically, data structures instantiated using the multiprocessing package do not behave reliably. For example, things break predictably when you nest shared memory structures within other nested shared memory structures. The ideal way to have asynchronous communication between PyTorch dataloader workers is to use process Queues, which shuttle active child process state information to the next active worker which the…
  </blockquote>
</aside>
Pairwise distance between multidimentional tensors

username: Hadar933
cooked: <p>I have two multidimensional tensors <code>t1</code> and <code>t2</code>, shaped <code>(b,c,h,w)</code>. I would like to calculate the pairwise distance between every pair in the batch, that is, the distance <code>||t1[i]-t2[j]||</code> for all <code>i,j=1,2,...,b</code>. Do notice that</p>
<ul>
<li>
<p>every element <code>t1[i]</code> or <code>t2[j]</code> is in itself a tensor of shape <code>(c,h,w)</code></p>
</li>
<li>
<p>I expect a result with shape <code>(b,b)</code></p>
</li>
<li>
<p>Ive tried using cdist after flattening, that is, <code>d=torch.cdist(t1.flatten(1),t2.flatten(2))</code>, but the metric (it seems) doesn’t seem to really result in small values for similar tensors and vice versa.</p>
</li>
</ul>
<p>appreciate the help!</p>

username: ptrblck
cooked: <p>I guess broadcasting would work, but I’m unsure if you want to sum the difference or use any other norm:</p>
<pre><code class="lang-python">b, c, h, w = 2, 3, 4, 4

a = torch.randn(b, c, h, w)
res = a.unsqueeze(1) - a.unsqueeze(0)
res = res.sum(dim=[2, 3, 4])
print(res)
# tensor([[ 0.0000, -3.6586],
#         [ 3.6586,  0.0000]])
</code></pre>
CUDA stream not working

username: beann
cooked: <p>I try to use in CUDA stream in pytorch with a multi-branch network. However, it seems these branches are still executed sequentially. The code snippt is like,</p>
<pre><code class="lang-auto">def forward(x):
    with cuda.stream(s1):
        a = f1(x)
    with cuda.stream(s2):
        b = f2(x)
</code></pre>
<p>Do anyone have some experiences about this?</p>

username: ptrblck
cooked: <p>If not enough compute resources are available, the kernels would be launches sequentially once enough SMs are free.</p>

username: beann
cooked: <p>Thank you for your anser. I’ve tried some really small networks in 2080Ti, but it still not working.<br>
E.g.,<br>
f1 = nn.Sequential(nn.Linear(1024, 10), nn.Linear(10, 10))<br>
f2 = nn.Sequential(nn.Linear(1024, 10), nn.Linear(10, 10))</p>
<p>Or is there any way to figure out what size the gpu can support?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="beann" data-post="3" data-topic="155749">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/b/48db29/48.png" class="avatar"> beann:</div>
<blockquote>
<p>Or is there any way to figure out what size the gpu can support?</p>
</blockquote>
</aside>
<p>It depends on the kernel implementation and the overall occupancy. I.e. each SM has limited resources in shared memory, threads, and registers. If one of these resources is exhausted, there won’t be any way to fit another block onto the SM.<br>
The <a href="https://www.youtube.com/watch?v=QQceTDjA4f4">GTC 2022 - How CUDA Programming Works</a> talk by Stephen Jones, CUDA Architect, NVIDIA, might be interesting to you. Especially the section starting at 23:00.</p>
Difference in backprop depending on whether weighted attention is used or not when using Cosine similarity loss

username: Jo-won
cooked: <p>I have a model for continuous learning between triplets. I leave you with questions about how this model is learned.</p>
<ol>
<li>In the learning process, features are extracted from the backbone network for the triplet (anchor, positive, negative) and the weights are multiplied for each feature.</li>
<li>These weights are calculated individually by a model named <strong>A</strong> taking the triplet feature as an input.</li>
<li>From these weighted triplet features, calculate the cosine similarity between anchor-positive and anchor-negative pairs.</li>
<li>The calculated cosine similarity is converted into refined similarity scores through a model called <strong>B</strong>, and the model is learned by the triplet margin loss through these two.</li>
</ol>
<p>Only the model <strong>A</strong> and model <strong>B</strong> are backpropagated throughout the entire process, and all others are frozen.</p>
<p>My question here is number 3. The cosine similarity calculation process includes L2 normalization of each feature anyway, so it was confirmed that even if each feature is multiplied by weight, it will return to before the multiplication. (Because cosine similarity is related to the angle between the two vectors, regardless of scale.)</p>
<p>So I thought that this process of multiplying each feature by weight would have no effect on learning because of the cosine similarity. However, contrary to my opinion, calculating weight through model <strong>A</strong> and multiplying each feature in the learning process showed better performance when evaluating than when excluding this process. As a result of checking specifically, the degree to which model <strong>B</strong> was learned has changed in the presence or absence of weight multiplication process through model <strong>A</strong>. The current assumption is that the weight multiplication process from model <strong>A</strong> was meaningless during forwarding during learning, but in the backward process, there were more gradients involved with the chain rule, indicating the direction of model <strong>B</strong> learning more specifically</p>
<p>Can anyone tell me what the actual performance improvement is?</p>
Torch.fx subgraph

username: Andre_Chang
cooked: <p>Is there a way to extract all leaf nodes (recursive call_modules) and extract some nodes as subgraphs using torch.fx ?</p>
<p>Something like <a href="https://github.com/pytorch/tvm/blob/master/torch_tvm/fusion_pass.cpp#L119" rel="noopener nofollow ugc">this</a> in torchscript</p>
Apple Silicon support buggy?

username: sebastian
cooked: <p>Since M1 GPU support is now available (<a href="https://pytorch.org/blog/introducing-accelerated-pytorch-training-on-mac/" class="inline-onebox" rel="noopener nofollow ugc">Introducing Accelerated PyTorch Training on Mac | PyTorch</a>) I did some experiments running different models. While everything seems to work on simple examples (mnist FF, CNN, …), I am running into problems with a more complex model known as SwinIR (<a href="https://github.com/JingyunLiang/SwinIR" class="inline-onebox" rel="noopener nofollow ugc">GitHub - JingyunLiang/SwinIR: SwinIR: Image Restoration Using Swin Transformer (official repository)</a>). Running the code pulled from github with device=‘mps’ and the environment variable ‘PYTORCH_ENABLE_MPS_FALLBACK=1’ to fall back to the cpu in cases where mps is not supported yet, produces the wrong results:<br>
When upscaling an image of Lincoln (works correctly on cpu) it produces this instead:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/5/7548df6c3c84899c3e93834bc0fda750489f21f2.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/7548df6c3c84899c3e93834bc0fda750489f21f2" title="Lincoln_SwinIR"><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/5/7548df6c3c84899c3e93834bc0fda750489f21f2.jpeg" alt="Lincoln_SwinIR" data-base62-sha1="gJxZzMr190m8uOfxyd8FXJIrzIS" width="500" height="500" data-dominant-color="479AB5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Lincoln_SwinIR</span><span class="informations">640×640 126 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Does anyone know what is going on, or what I could try to fix this?</p>

username: eqy
cooked: <p>Thanks for trying out the new backend! I think we have seen issues arising from the new backend (e.g., <a href="https://github.com/pytorch/pytorch/issues/80732" class="inline-onebox" rel="noopener nofollow ugc">[MPS] AdaptiveAvgPool2D doesn't accept input spatial size smaller than output shape · Issue #80732 · pytorch/pytorch · GitHub</a>), so it would be helpful if you could isolate the discrepancy (e.g., vs. CPU) to a particular layer or shape in order to file a bug.</p>
Sharing file access in DataLoader workers

username: ptoews
cooked: <p>I’m struggling with a bug where the tensors returned by my DataLoader are garbage if num_workers &gt; 1, and I assume this is caused by the underlying dataset using a file pointer (the whole dataset is one file) which is forked to the workers.</p>
<p>What’s the best way to distribute such a dataset to multiple workers? Is there some easy way of instantiating the dataset separately/cloning the file pointers for each worker? Or are there other better approaches?</p>
<p>Thanks in advance.</p>

username: nivek
cooked: <p>Are you currently passing in a <code>worker_init_fn</code> to DataLoader? It is a function that gets passed to each worker to initialize.</p>
<pre><code class="lang-python">def worker_init_fn(worker_id):
    worker_info = torch.utils.data.get_worker_info()
    dataset = worker_info.dataset # the copy of the dataset object in this process. Note that this will be a different object in a different process than the one in the main process.
    # Either instantiate the dataset or create a new file pointer
</code></pre>
<p>These sections in the documentation may be helpful:</p>
<ol>
<li><a href="https://pytorch.org/docs/stable/data.html#multi-process-data-loading" rel="noopener nofollow ugc">Multi-process data loading</a></li>
<li><a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.IterableDataset" rel="noopener nofollow ugc">Example 2 of this section</a></li>
</ol>

username: ptoews
cooked: <p>Thank you, that was exactly what I needed!</p>

username: Pietro_Cicalese
cooked: <p>Somewhat related, see here:</p><aside class="quote quote-modified" data-post="9" data-topic="11473">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pietro_cicalese/48/27365_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/communicating-with-dataloader-workers/11473/9">Communicating with Dataloader workers</a> 
  </div>
  <blockquote>
    I looked everywhere and worked on this problem for a few days before solving it. Basically, data structures instantiated using the multiprocessing package do not behave reliably. For example, things break predictably when you nest shared memory structures within other nested shared memory structures. The ideal way to have asynchronous communication between PyTorch dataloader workers is to use process Queues, which shuttle active child process state information to the next active worker which the…
  </blockquote>
</aside>
Dataloader communication between workers

username: Etienne_Perot
cooked: <p>I am developing a small utility package to load data to train recurrent nets. The idea is to stream temporally coherent batches. I reuse the IterableDataset with some tweaks to allow to join over multiple streams in the same batch, and to do so with multiple workers:</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://github.com/etienne87/pytorch-stream-dataloader">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/etienne87/pytorch-stream-dataloader" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/345;"><img src="https://opengraph.githubassets.com/d11fd0acea1083b41a2b2714835302711de7137b65ba2851225eb31ead7deab5/etienne87/pytorch-stream-dataloader" class="thumbnail" width="690" height="345"></div>

<h3><a href="https://github.com/etienne87/pytorch-stream-dataloader" target="_blank" rel="noopener nofollow ugc">etienne87/pytorch-stream-dataloader</a></h3>

  <p>Contribute to etienne87/pytorch-stream-dataloader development by creating an account on GitHub.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>In order to split the load of the numerous streams accross the workers, i try to distribute the load evenly, but this is not very ideal as some streams might be longer than others.</p>
<p>Ideally, i would like to communicate between different workers to access a single list of streams and be notified of which streams have already been read.</p>
<p>Is there a workaround with some Pytorch Data framework, like a custom sampler, or do i need to hack this myself using multiprocessing library?</p>
<p>EDIT: I did a first version with a multiprocessing.Mutex. It works as expected.</p>

username: Pietro_Cicalese
cooked: <p>See here:</p><aside class="quote quote-modified" data-post="9" data-topic="11473">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pietro_cicalese/48/27365_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/communicating-with-dataloader-workers/11473/9">Communicating with Dataloader workers</a> 
  </div>
  <blockquote>
    I looked everywhere and worked on this problem for a few days before solving it. Basically, data structures instantiated using the multiprocessing package do not behave reliably. For example, things break predictably when you nest shared memory structures within other nested shared memory structures. The ideal way to have asynchronous communication between PyTorch dataloader workers is to use process Queues, which shuttle active child process state information to the next active worker which the…
  </blockquote>
</aside>
Performance difference in torch.einsum

username: kgbounce
cooked: <p>I would like to learn the internals of some PyTorch high-level ops, such as <code>einsum</code> and <code>bmm</code>. I created a code snippet as follows:</p>
<pre><code class="lang-auto">B, C_qk, H, N_s, N_t = 10, 128, 1, 32, 64
q = torch.randint(0, 100, (B, C_qk, H, N_s))
k = torch.randint(0, 100, (B, N_t, H, C_qk))

def einsum_1(q, k):
    return torch.einsum('bchq,bkhc-&gt;bkhq', [q, k])
def einsum_2(q, k):
    return torch.einsum('bkhc,bchq-&gt;bkhq', [k, q])

t1 = benchmark.Timer(stmt='einsum_1(q, k)',
                     setup='from __main__ import einsum_1',
                     globals={'q': q, 'k': k})

t2 = benchmark.Timer(stmt='einsum_2(q, k)',
                     setup='from __main__ import einsum_2',
                     globals={'q': q, 'k': k})

print(t1.timeit(10000))
print(t2.timeit(10000))

num_threads = torch.get_num_threads()
print(f'Benchmarking on {num_threads} threads')

t1_threaded = benchmark.Timer(stmt='einsum_1(q, k)',
                              setup='from __main__ import einsum_1',
                              num_threads=num_threads,
                              globals={'q': q, 'k': k})

t2_threaded = benchmark.Timer(stmt='einsum_2(q, k)',
                              setup='from __main__ import einsum_2',
                              num_threads=num_threads,
                              globals={'q': q, 'k': k})

print(t1_threaded.timeit(50000))
print(t2_threaded.timeit(50000))
</code></pre>
<p>Here, <code>t2</code> (1.28ms) and <code>t2_threaded</code> (1.27ms) are consistently slower than <code>t1</code> (1.22ms) and <code>t1_threaded</code> (1.23ms).</p>
<p>Note that the two einsums yield the same output as they’re equivalent.</p>
<pre><code class="lang-auto">res_1 = torch.einsum('bchq,bkhc-&gt;bkhq', [q, k])
res_2 = torch.einsum('bkhc,bchq-&gt;bkhq', [k, q])
assert torch.equal(res_1, res_2), "torch.einsum failed to auto align the summation dim"
</code></pre>
<p>What are the reasons behind the performance difference? Is there a tool I can use to trace these high-level ops to see what primitive ops are called/call counts/latency breakdown?</p>

username: kgbounce
cooked: <p>The primitive ops can be reported by <code>torch.profiler</code>. But the breakdown doesn’t answer the original question.</p>
Deepcopy TypeError: can't pickle torch._C.Generator objects

username: Se_di
cooked: <p>Hey,</p>
<p>I have a Policy class consisting of an lstm network and a cnn. During the training process I need to do a deepcopy. But when doing <em>deepcopy(policy)</em> I get the following error:</p>
<p>TypeError: can’t pickle torch._C.Generator objects</p>
<p>Couldnt find any helpful answer to this problem yet. Can anyone help?</p>

username: Qi_Liu
cooked: <p>Have you solved this problem?<br>
I face with the same problem.</p>

username: acxz
cooked: <p>Upstream issue: <a href="https://github.com/pytorch/pytorch/issues/43672" class="inline-onebox" rel="noopener nofollow ugc">Make `torch.Generator` picklable · Issue #43672 · pytorch/pytorch · GitHub</a></p>

username: acxz
cooked: <p>If you post the entire definition of the class, maybe we can help out.</p>
<p>My guess is that you might have a self.torch_seed function call. Try to separate that call from the object you are pickling i.e. your Policy class.</p>
Patch Selection from Visual transformer Patches based on attention map

username: Azadbakht
cooked: <p>Hi,<br>
I am currently working with visual transformers and I have the input of [B, 64, 160] where 64 is the number of patches and 160 is the embedding dimension, I have an attention map of size [B, 64] computed with a nn.Linear layer from input patches. I want to select the top 32 patches from input with respect to the attention map( I want the output to be in the shape of [B, 32, 160] and contain the patches with the strongest scores). I tried several available solutions but none of them account for backpropagation on nn.Linear attention layer in other words selection process( slicing, torch.select, …) is non-differentiable with respect to indices, so PyTorch won’t compute any gradient for the attention layer.</p>

username: ptrblck
cooked: <aside class="quote no-group quote-modified" data-username="Azadbakht" data-post="1" data-topic="155863">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/5e9695/48.png" class="avatar"> Azadbakht:</div>
<blockquote>
<p>in other words selection process( slicing, torch.select, …) is non-differentiable so PyTorch won’t compute any gradient</p>
</blockquote>
</aside>
<p><code>torch.select</code> and slicing are differentiable and will provide the gradient to the selected items:</p>
<pre data-code-wrap="python"><code class="lang-python"># select
x = torch.randn(10, requires_grad=True)
out = torch.select(x, 0, 1)
print(out.grad_fn)
# &lt;SelectBackward0 object at 0x7fdca5774ca0&gt;

out.backward()
print(x.grad)
# tensor([0., 1., 0., 0., 0., 0., 0., 0., 0., 0.])

# slicing
x = torch.randn(10, requires_grad=True)
out = x[1:3]
print(out.grad_fn)
# &lt;SliceBackward0 object at 0x7fdca57183d0&gt;
out.mean().backward()
print(x.grad)
# tensor([0.0000, 0.5000, 0.5000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,
#         0.0000])
</code></pre>
Using isend / ircv works synchronously

username: Mirian_Hipolito
cooked: <p>Hello, I’m trying to create a Server-Worker communicationi with NCCL, where the Server can receive the acknowledge from the Workers as soon as they finish, in an asynchronous way. However when I try to run the following code, I noticed the workers are always sending the tensors in the same order, even when the worker 1 has a time.sleep (on purpose) and this should make it be the last one.</p>
<pre><code class="lang-auto">class Server:
    def __init__(self,rank,size):
        print("Server has started")
        tuple_list = []
        for worker in range(1,size):
            ack = torch.zeros(1)
            req = dist.irecv(tensor=to_device(ack), src=worker)
            tuple_list.append((worker, req))
            print("Server: Finished reading the tuples from server")
        

        while len(tuple_list)!=0:
            for currtuple in tuple_list:
                worker, req = currtuple
                if(req.is_completed()):
                    print(f" Server: Worker {worker} has finished")
                    tuple_list.remove(currtuple)
                else:
                    print(f" Server: Worker {worker} not ready yet ")
</code></pre>
<pre><code class="lang-auto">class Worker:
    def __init__(self,rank,size):

        if rank == 1:
            time.sleep(7)

        print(f"Worker: {rank} has started")
        mytensor = torch.tensor(1)
        dist.isend(tensor=to_device(mytensor), dst=0)
        print(f"Worker: {rank} send the tensor")
</code></pre>
<pre><code class="lang-auto">def main(rank, world):

    if rank == 0:
        Server(rank,world)
    else:
        Worker(rank,world)
</code></pre>
<p>This is an example of the output from the code above using 3 nodes:</p>
<blockquote>
<p>Worker: 2 has started<br>
Server has started<br>
Worker: 1 has started<br>
Server: Finished reading the tuples from server<br>
<strong>Worker: 1 send the tensor</strong><br>
Server: Finished reading the tuples from server<br>
Server: Worker 1 has finished<br>
Server: Worker 2 not ready yet<br>
<strong>Server: Worker 2 not ready yet</strong><br>
…<br>
…<br>
Server: Worker 2 not ready yet<br>
Server: Worker 2 has finished<br>
<strong>Worker: 2 send the tensor</strong></p>
</blockquote>
<p>I’ve already tried with different combinations of irec/isend but still cannot make them work asynchronously.</p>
<p>ps. the <em>to_device</em> function is only to sending to cuda if available:</p>
<pre><code class="lang-auto">def to_device(x):
    return x.cuda() if torch.cuda.is_available() else x
</code></pre>

username: pbelevich
cooked: <p>CC <a class="mention" href="/u/kwen2501">@kwen2501</a> <a class="mention" href="/u/yaroslav_bulatov">@Yaroslav_Bulatov</a></p>
CUDA out of memory in simple subtraction

username: PabloVD
cooked: <p>I have two tensors, <code>a</code> and <code>b</code>, and I want to subtract them in CUDA, inside a neural network evaluation.<br>
Each of these tensors contain ~1e5 float32 elements, specifically with shape <code>torch.Size([161858])</code> and 0.647432 Mb each. When I try to subtract them in a naive way:</p>
<pre><code class="lang-python">c = a - b 
</code></pre>
<p>I get the following out of memory error:</p>
<p><code>RuntimeError: CUDA out of memory. Tried to allocate 97.60 GiB (GPU 0; 15.75 GiB total capacity; 52.60 MiB already allocated; 11.73 GiB free; 82.00 MiB reserved in total by PyTorch) If reserved memory is &gt;&gt; allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF</code></p>
<p>Using <code>torch.subtract</code> I get the same.</p>
<p>However, I can avoid the memory issue through the following (less efficient) method:</p>
<pre><code class="lang-python">c = torch.zeros(a.shape[0],device=self.device)
for i in range(a.shape[0]):
	c[i] = a[i]-b[i]
</code></pre>
<p>What’s going on here? Why CUDA tries to allocate 97GB, when each of the tensors are less than 1 Mb? And why the latter method works, while the former doesn’t?</p>
<p>Thank you in advance,<br>
Pablo.</p>

username: ptrblck
cooked: <p>Could you post the exact shapes of <code>a</code>, <code>b</code>, and <code>c</code>? I would guess that some unwanted broadcasting might happen in the subtraction calls.</p>

username: PabloVD
cooked: <p>You were right, <code>a</code> had shape <code>torch.Size([161858])</code> but <code>b</code> had <code>torch.Size([161858,1])</code>. Using <code>b.view(-1)</code>, the memory issue was fixed. Thanks!</p>
Implementing a sub class in parent class in pytorch

username: saran
cooked: <p>i want to implement a pytroch nn.class with a subclass in it  for a neural network but i am getting<br>
this error</p>
<p>raise NotImplementedError<br>
NotImplementedError</p>
<p>pleae some one can figure out</p>
<p>This (ConvBNR) is the sub class which i want to implemet in the main class(DPclass) of neural netwrok</p>
<pre><code class="lang-auto">class ConvBNR(nn.Module)
      def __init__(self,in_channels, out_channels):
             #def __init__():
            super(ConvBNR,self).__init__()
            self.net = nn.Sequential(
            nn.Conv2d(32,32,3, 1, 1),
            nn.BatchNorm2d(32),
            nn.ReLU()
        )

       def call(self, x, training=None):
        x = self.net(x, training=training)

        return x



 This is my main class of NN

class (DPclass) (nn.Module):
    def __init__(self):
        #super(DPclass)()
    super().__init__()
         self.start = nn.Sequential(
            nn.Conv2d(in_channels=3, out_channels=32, kernel_size=3, stride=1, padding=1),
            nn.BatchNorm2d(32),
            nn.ELU()
            #nn.Dropout(0.25)
        )
       
        #self.start2 = ConvBNRelu(32,64,3,1,1)
        self.start2 = ConvBNRelu(32,64)
        some other layers......... 

   def forward(self, x):
         A = self.start(x)#here1
        print("A.shape1",A.shape)
        A = self.start2(x)#here1
       .........
</code></pre>

username: saran
cooked: <p>sorry i cant properly ident the code here but i hope you will figure out</p>

username: ptrblck
cooked: <p>The <code>ConvBNR</code> modules implements the <code>call</code> method while you should provide a <code>forward</code> method.</p>

username: saran
cooked: <p>oh right thank you sir</p>
Does F.lstm exist?

username: mariaalfaroc
cooked: <p>Is there a functional version of torch.nn.lstm?</p>

username: ptrblck
cooked: <p>I don’t think there is a functional API in <code>torch.nn.functional</code>, but <code>torch._VF.lstm</code> is used inside the <code>nn.LSTM</code> module as seen <a href="https://github.com/pytorch/pytorch/blob/b9d516138b01adf1be6e7eb24e089df4065de5d2/torch/nn/modules/rnn.py#L773-L778">here</a> so you might want to use it.</p>

username: mariaalfaroc
cooked: <p>Thanks! <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>So If I wanted to use a Bidirectional LSTM, that receives as input a <code>torch.tensor</code> of shape (batch, timesteps, features):</p>
<p><code>hx</code> should be a tensor of zeros as they do in the docs:</p>
<pre><code class="lang-auto">            num_directions = 2
            real_hidden_size = self.proj_size if self.proj_size &gt; 0 else self.hidden_size
            h_zeros = torch.zeros(self.num_layers * num_directions,
                                  max_batch_size, real_hidden_size,
                                  dtype=input.dtype, device=input.device)
            c_zeros = torch.zeros(self.num_layers * num_directions,
                                  max_batch_size, self.hidden_size,
                                  dtype=input.dtype, device=input.device)
            hx = (h_zeros, c_zeros)
</code></pre>
<p>And but I’m having trouble understanding how they flatten the weights before passing it to the layer. Is it a list of the different LSTM variables flatten in a certain order?</p>

username: ptrblck
cooked: <p>The order of the parameters is given in the <a href="https://pytorch.org/docs/stable/generated/torch.nn.LSTM.html#torch.nn.LSTM">docs for <code>nn.LSTM</code></a> in the <code>Variables</code> section.</p>
Softmax don't give sum to 1 when batch size is big

username: xingsen19
cooked: <pre><code class="lang-python">import torch
import torch.nn as nn
batch_size = 16
a = torch.rand(batch_size,12,4864,4864).cuda()
m = nn.Softmax(dim=3)
print(torch.sum(m(a),dim=3))
</code></pre>
<p>when batch_size is big, the output is not 1, when I change batch_size to 8, The output is all 1.<br>
Is there any way to make the calculation of the softmax function more accurate?</p>

username: KFrank
cooked: <p>Hi Xingsen!</p>
<aside class="quote no-group" data-username="xingsen19" data-post="1" data-topic="155792" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/xingsen19/48/51152_2.png" class="avatar"> xingsen19:</div>
<blockquote>
<p>Is there any way to make the calculation of the softmax function more accurate?</p>
</blockquote>
</aside>
<p>How inaccurate is <code>Softmax</code> in your test?  You are summing over a<br>
large number of values, and it is to be expected that some floating-point<br>
round-off error occurs.</p>
<p>Look at the largest deviations from the expected value of <code>1.0</code>.  Do your<br>
results agree with <code>1.0</code> up to about 6 or 7 decimal places – that is, within<br>
expected floating-point round-off error?</p>
<p>You can make the calculation of the softmax function more accurate by<br>
performing the calculations in double-precision.</p>
<p>Best.</p>
<p>K. Frank</p>
Is there a way to export graph.print_tabular()

username: Neel_Bhattacharyya
cooked: <p>gm = torch.fx.symbolic_trace(m)</p>
<p>gm.graph.print_tabular()</p>
<p>I am trying to export the graph for model, is there anyway to convert graph.print_tabular() into dictionary format?</p>

username: ParGG
cooked: <p>Something of this kind?</p>
<pre><code class="lang-python">graph = {
            f"op_{idx}": {
                "opcode": n.op,
                "name": n.name,
                "target": n.target,
                "args": n.args,
                "kwargs": n.kwargs,
                ...
            } for idx, n in enumerate(gm.graph.nodes)
        }
</code></pre>

username: Neel_Bhattacharyya
cooked: <p>Yes, this will work, Thank you so much</p>

username: ParGG
cooked: <p>You might try something like this as well which is simplier, but I don’t remember by heart what Node.<strong>dict</strong> contains exactly.</p>
<pre><code class="lang-python">graph = {f"op_{idx}": n.__dict__ for idx, n in enumerate(gm.graph.nodes)}
</code></pre>
Is it OK to disable `amp` of BN by decorating its forward function?

username: songyuc
cooked: <p>Hi guys,<br>
I want to disable <code>amp</code> of BN by decorating its forward function, as,</p>
<pre><code class="lang-python">keep_forward_float_(self.bn)
</code></pre>
<p>where</p>
<pre><code class="lang-python">def keep_forward_float_(m):
    def float_forward(self, x, forward):
        assert isinstance(self, nn.Module)
        with autocast(enabled=False):
            return forward(x.float())

    m.forward = MethodType(functools.partial(float_forward, forward=m.forward), m)
</code></pre>
<p>Is this implementation good?</p>
<p>Your answer and guide will be appreciated!</p>
[Torch Geometric] Load multiple files using multiprocessing in Dataset get() method

username: imayachita
cooked: <p>Hi,<br>
I read <a href="https://pytorch-geometric.readthedocs.io/en/latest/notes/create_dataset.html#creating-larger-datasets" rel="noopener nofollow ugc">here</a> that we can use <code>get()</code> method in <code>Dataset</code> to load a single file before passing the <code>Data</code> type object as a list to <code>DataLoader</code> to be fed into model training. In my case, I want to load multiple files with multiprocessing instead of 1 file at a time to speed up the process. How can I do that? Thanks</p>

username: ArchieGertsman
cooked: <p>To be clear, are you asking how to call <code>get()</code> in multiple processes simultaneously and gather each of their resulting <code>Data</code> objects in the main process? And do you want each subprocess to load multiple files? Here’s a simple example of using <code>Pool</code> to load two files, <code>data0.pt</code> and <code>data1.pt</code>, with two subprocesses:</p>
<pre><code class="lang-auto">import torch
from torch.multiprocessing import Pool

def load(idx):
    return torch.load(f'data{idx}.pt')

if __name__ == '__main__':
    with Pool(2) as p:
        data_list = p.map(load, [0,1])
</code></pre>
<p>but beware of the overhead of spawning subprocesses.</p>

username: imayachita
cooked: <p>Hey <a class="mention" href="/u/archiegertsman">@ArchieGertsman</a>, thanks for the answer!<br>
Basically, I want to do something like <code>num_parallel_calls</code> (for multiprocessing) and <code>.prefetch()</code> (to process the next batch of data in CPU while training is done in GPU) like in tf.data.Dataset <a href="https://cs230.stanford.edu/blog/datapipeline/#best-practices" rel="noopener nofollow ugc">here</a> using torch geometric <code>Dataset</code>. How can we do that? What I understand so far <code>__getitem__</code> or <code>get()</code> only loads 1 file at a time.</p>