akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
When using pytorch elastic training, how to broadcast the latest checkpoint to the newest nodes？

username: zzibc
cooked: <p>Hello, All!</p>
<p>I’d like to ask how to broadcast the latest checkpoint from the existing nodes to the newest node when I use elastic distributed training?</p>
<p>Assuming that I submit an elastic training job to a GPU cluster with <code>nnodes=1:2</code>. And I launch training script by following command:</p>
<pre><code class="lang-bash">MIN_SIZE=1 MAX_SIZE=2 \
python -m torch.distributed.run \
    --nnodes=$MIN_SIZE:$MAX_SIZE \
    --nproc_per_node=4 \
    --rzdv_id=$JOB_ID \
    --rzdv_endpoint=$ADDR:$PORT \
    --rdzv_backend=c10d \
    my_train_script.py
</code></pre>
<p>Initially, I got the first node with 4 GPUs. And the training will start because I set <code>MIN_SIZE=1</code>. In <code>my_train_script.py</code>, I saved checkpoint at the end of every epochs.</p>
<p>After some training steps(let’s say 3 epochs), I got the second node with 4 GPUs and I run the aforementioned command on the second node. I want that the training workers on the second node can load the checkpoint on the first node so that all the workers will have same model state, optimizer state, etc. Could anyone please to tell me how to implement it?</p>

username: mrshenli
cooked: <p>It will make it a lot easier if the checkpoint was written to a shared storage.</p>
<p>If the storage is local, one way to implement this can be binding ranks 0-3 to the first node, and always broadcast model states from rank0 to all other ranks before training loop starts. <a href="https://pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html" class="inline-onebox" rel="noopener nofollow ugc">DistributedDataParallel — PyTorch 1.12 documentation</a> already does this for you in the constructor.</p>

username: zzibc
cooked: <p>Shared storage is a good solution. Thanks <a class="mention" href="/u/mrshenli">@mrshenli</a> !</p>
[Caffe2] codes related to normalization

username: yeungkong
cooked: <p>Hi developers of pytorch. I am trying to figure out some caffe codes related to normalization. When I read normalize_op.h, I see some codes like</p>
<pre><code class="lang-auto">ConstStridedVec xVec(xData + base, 1, m, InnerStride(sf));
StridedVec yVec(yData + base, 1, m, InnerStride(sf));
</code></pre>
<p>I am not sure what ConstStridedVec or StridedVec do and I can hardly find some information about them. Could you please enlight me what they do? Please let me know if here is not a right place to ask questions about Caffe2.</p>
Compute accuracy in the regression

username: jahanifar
cooked: <p>Hello,<br>
I have an excel file that contains two columns, “input” and “label” (examples of the file are in the blow). I want to implement a regression task, and I need to implement the below lines of code in the k-fold cross validation:</p>
<p>Some examples of the dataset:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/1/916fc24b090365b4b640f34299ed5bc549d5c9ed.jpeg" alt="regression" data-base62-sha1="kKAFTiMwctIESwCVxUPkeQOgb0x" width="266" height="370"></p>
<pre><code class="lang-auto">with torch.no_grad():
    for data in testloader:
        images, labels = data
        # calculate outputs by running images through the network
        outputs = net(images)
        # the class with the highest energy is what we choose as prediction
        _, predicted = torch.max(outputs.detach(), 1)
        total += labels.size(0)
        correct += (predicted == labels).sum().item()

print(f'Accuracy of the network on the 10000 test images: {100 * correct // total} %')
</code></pre>
<p>I think since it is not a classification problem, it is not correct to use this line in the code:</p>
<p><code>correct += (predicted == labels).sum().item()</code></p>
<p>Could you please let me know how I can change the codes to get accuracy in this scenario?</p>

username: srishti-git1110
cooked: <p>Hi <a class="mention" href="/u/jahanifar">@jahanifar</a></p>
<p>For regression tasks, accuracy isn’t a metric. You could use MSE-</p>
<pre><code class="lang-auto">∑(y - yhat)2/ N

</code></pre>
<p>where yhat is the predicted outcome and N is the total no. of examples.</p>
<p>So you could do-</p>
<pre><code class="lang-auto">MSE = ((torch.pow((predicted - labels), 2)).sum()) / total
</code></pre>

username: jahanifar
cooked: <p>Thanks a lot, it looks like computing the cost function. hence I think it is related to the loss, not accuracy. Am I right?</p>

username: srishti-git1110
cooked: <p>Right.<br>
MSE is a popular loss function for regression tasks.<br>
And used as a evaluation metric, too.</p>
<p>While evaluating regression models, you really cannot use accuracy like how it’s defined for classification tasks; as if y = 45.5 and yhat = 45.6, accuracy will term it as a wrong prediction but it is a fairly good prediction for a regression tasks.</p>
<p>Other popular metrics are R2, RMSE, MAE etc. that are similarly defined.</p>
Parallelizing model heads?

username: Albert_T
cooked: <p>I have a model with a large number of simple heads (&gt; 100). Currently, at the final stage of the <code>forward</code> call, I have to iterate through all the heads and append the results into a list, which is quite slow and underutilizes my hardware. Is there a way to parallelize running the heads? I think tensorflow does this automatically since everything is built into the graph but I can’t seem to find any equivalent functionality in Pytorch.</p>
Segmentation with more detail

username: A_Ebrahimi
cooked: <p>hi , i’m trying to do a segmentation on satellite images but the details are not promising . is there anyway to train a model that can detect smaller details . right now i’m using deeplabV3 with resnet50 as backbone</p>

username: KFrank
cooked: <p>Hi Ali!</p>
<aside class="quote no-group" data-username="A_Ebrahimi" data-post="1" data-topic="160696" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/919ad9/48.png" class="avatar"> A_Ebrahimi:</div>
<blockquote>
<p>i’m trying to do a segmentation on satellite images</p>
</blockquote>
</aside>
<p>You haven’t said much about your use case.  In particular, are you<br>
performing <em>semantic segmentation</em> (where you wish to label each<br>
pixel with the class to which it belongs)?  Or are you performing<br>
<em>instance segmentation</em> (where you label each pixel with the specific<br>
instance* of one or more classes of objects to which it belongs)?</p>
<p>Also, what is the structure of your images?  Do, for example, the pixels<br>
in the lower left of your image provide any significant information about<br>
to which classes the pixels in the upper right of your image belong?</p>
<p>Without knowing anything more, I would suggest that you try U-Net, but<br>
how deep you want your U-Net to be (which is to say how large a field<br>
of view you want it to have) will depend on your specific use case.</p>
<p>*)  An example of instance segmentation:  Let’s say you have images<br>
that contain bicycles and cars (and people and trees and fire hydrants,<br>
but you don’t care about those).  Any pixel that is not part of a bicycle<br>
or car is a background pixel.  Let’s say that some specific image contains<br>
two bicycles and three cars.  Instance segmentation is used to label each<br>
pixel as being “background,” “bicycle-1,” bicycle-2," “car-1,” “car-2,” or<br>
“car-3.”  (By way of contrast, semantic segmentation would label each<br>
pixel as being “background,” “bicycle,” or “car,” but wouldn’t tell you<br>
which specific bicycle or car the pixel is in.)</p>
<p>Best.</p>
<p>K. Frank</p>
Why do I have the error: Given groups=1, weight of size [8, 1024, 1, 1], expected input[8, 304, 9, 40] to have 1024 channels, but got 304 channels

username: zakaseb
cooked: <p>I am working with <a href="https://github.com/Owen-Liuyuxuan/visualDet3D" rel="noopener nofollow ugc">yolostereo3d</a> for stereo3d object detection (solely stereo camera, no velydone)on kitti dataset with <a href="https://github.com/mmaaz60/EdgeNeXt" rel="noopener nofollow ugc">edgeNext</a> as the backbone instead of resNet.</p>
<p>Before changing the backbone from resNet to edgeNext with the same kitti dataset, everything was working fine. However, I started having the below error afterwards:</p>
<pre><code class="lang-auto">RuntimeError: Given groups=1, weight of size [8, 1024, 1, 1], expected input[8, 304, 9, 40] to have 1024 channels, but got 304 channels instead
</code></pre>
<p>Here is how I changed the backbone:</p>
<pre><code class="lang-auto">class YoloStereo3DCore(nn.Module):
    """
        Inference Structure of YoloStereo3D
        Similar to YoloMono3D,
        Left and Right image are fed into the backbone in batch. So they will affect each other with BatchNorm2d.
    """
    def __init__(self, backbone_arguments):
        f = open("/home/zakaseb/Thesis/YoloStereo3D/Stereo3D/Sequence.txt", "a")
        f.write("yolosterero3dCore_init \n")
        f.close()
        super(YoloStereo3DCore, self).__init__()
        self.backbone =edgenext_small(**backbone_arguments) # Resnet, change backbone from here

        base_features = 256 #if backbone_arguments['depth'] &gt; 34 else 64 # meaning which depth of resnet
        self.neck = StereoMerging(base_features) #stereomerging outputs features and depth output.
</code></pre>
<p>Here is the edgenext_small()</p>
<pre><code class="lang-auto">@BACKBONE_DICT.register_module
def edgenext_small(pretrained=False, **kwargs):
    FPS @ BS=1: 93.84 &amp; @ BS=256: 1785.92 for MobileViT_S
    model = EdgeNeXt(depths=[3, 3, 9, 3], dims=[48, 96, 160, 304], expan_ratio=4,
                     global_block=[0, 1, 1, 1],
                     global_block_type=['None', 'SDTA', 'SDTA', 'SDTA'],
                     use_pos_embd_xca=[False, True, False, False],
                     kernel_sizes=[3, 5, 7, 9],
                     d2_scales=[2, 2, 3, 4],
                     classifier_dropout=0.0)

    return model
</code></pre>

username: eqy
cooked: <p>It looks like you are specifying some kind of feature dimension for the downsampling layers of <code>EdgeNeXt</code> in the <code>dims</code> parameter, and this creates a mismatch if the layers downstream of the backbone expect a different feature dimension (e.g., 1024). Perhaps you could simply change the value of <code>304</code> to <code>1024</code> here, though I’m not sure if that has other implications for the efficiency of the model architecture.</p>

username: zakaseb
cooked: <p>Thanks. I’ve done that and I’ve found myself in another masking-related error. So I backtracked thinking those errors were related. But I doubt they are. Thanks anyway!</p>
Convert a float number to float tensor with 100% precision

username: jahanifar
cooked: <p>Hi<br>
I have an excel file that contains float numbers like the below:<br>
-4.787870188<br>
-3.50905703<br>
(the length of them is 15 or 17)</p>
<p>But when I want to convert them to tensor, it will be rounded like this:<br>
(tensor([-4.7879])<br>
(tensor([-3.5091])</p>
<p>Is there a way to convert the original numbers into the tensor with their actual precisions?</p>

username: AlphaBetaGamma96
cooked: <p>Hi <a class="mention" href="/u/jahanifar">@jahanifar</a>,</p>
<p>This is most likely due to the print options settings in pytorch (docs <a href="https://pytorch.org/docs/stable/generated/torch.set_printoptions.html" rel="noopener nofollow ugc">here</a>)</p>
<p>Can you set <code>torch.set_printoptions(precision=10)</code> (just after you <code>import torch</code>) and re-run this? You’ll get the correct values. If not, use <code>torch.float64</code> instead!</p>
Apply math operation dynamically between two nn.Modules or loss functions

username: dan.torch
cooked: <p>I would like to apply math operations dynamically between two loss functions or nn.Modules. I am not sure how exactly to implement it. But any help is really appreciated.</p>
<p>For example: In below example I would like to add two loss functions.</p>
<pre><code class="lang-auto">nn.L1Loss() + nn.CosineEmbeddingLoss()
</code></pre>
<p>If I do this it gives me error:</p>
<pre><code class="lang-auto">----&gt; 1 nn.L1Loss() + nn.CosineEmbeddingLoss()
TypeError: unsupported operand type(s) for +: 'L1Loss' and 'CosineEmbeddingLoss'`
</code></pre>
<p>I also tried creating a wrapper with forward function and torch operations like below, but it doesn’t work either.</p>
<pre><code class="lang-auto">class Execute_Op(nn.Module):
    def __init__(self):
        super().__init__()
        
    def forward(self, x, y, op):
        if op == 'add':
            return torch.add(x, y)
        elif op == 'subtract':
            return torch.subtract(x - y)

exec_op = Execute_Op()
exec_op(nn.L1Loss(), nn.CosineEmbeddingLoss(), 'add')
</code></pre>
<p>It gives error like below:</p>
<pre><code class="lang-auto">Execute_Op.forward(self, x, y, op)
      5 def forward(self, x, y, op):
      6     if op == 'add':
----&gt; 7         return torch.add(x, y)
      8     elif op == 'subtract':
      9         return x - y

TypeError: add(): argument 'input' (position 1) must be Tensor, not L1Loss
</code></pre>

username: ptrblck
cooked: <p>You could either use the functional APIs or create objects for both loss functions:</p>
<pre><code class="lang-python">criterion1 = nn.L1Loss()
criterion2 = nn.CosneEmbeddingLoss()
...
output = model(input)
loss1 = criterion1(output, target1)
loss2 = criterion2(output, target2)
loss = loss1 + loss2
loss.backward() # gradient calculation using both loss functions
</code></pre>

username: dan.torch
cooked: <p>Thanks for the response <a class="mention" href="/u/ptrblck">@ptrblck</a>.<br>
Your response makes sense for a general case.<br>
But I am looking for a dynamic way to create loss functions where I can combine the loss equation without passing the inputs such as <code>output</code> or <code>target</code>, and then at the run time, I can pass the input values. Is there a way or workaround to do that?</p>

username: ptrblck
cooked: <p>No, you won’t be able to “combine” objects directly, but could initialize them and add both losses “in a dynamic way” (i.e. if your conditions are met) during the forward pass.<br>
E.g. you could create a <code>list</code> of loss functions based on your conditions and apply them (during the loss calculation).</p>
Efficient loading of Tensor too large for RAM (for training)?

username: st-vincent1
cooked: <p>I have very large Tensors which I use as input for training my model (talking ~900GB size), and of course these are too large for loading to RAM. I was using <code>TensorDataset(torch.FloatStorage)</code> which worked well but is apparently extremely slow on the Lustre filesystem that I have to use.</p>
<p>Are there any alternatives that I have not considered? For my applciation, I need the data to be accessible by index (as is the case with <code>TensorDataset</code>).</p>
How to implement Laplace Posteriori Approximation on BERT in PyTorch?

username: Nikola12344
cooked: <p>Here is the link to SO where I provided code and explained the question:</p>
<aside class="onebox stackexchange" data-onebox-src="https://stackoverflow.com/questions/73599356/how-to-implement-laplace-posteriori-approximation-on-bert-in-pytorch">
  <header class="source">

      <a href="https://stackoverflow.com/questions/73599356/how-to-implement-laplace-posteriori-approximation-on-bert-in-pytorch" target="_blank" rel="noopener nofollow ugc">stackoverflow.com</a>
  </header>

  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/12134579/nikola-greb" target="_blank" rel="noopener nofollow ugc">
    <img alt="Nikola Greb" src="https://lh3.googleusercontent.com/a-/AAuE7mDoiWonhm3NGYvT9vuyywqbAnO2OHBWfHHmiwTZ=k-s256" class="thumbnail onebox-avatar" width="256" height="255">
  </a>

<h4>
  <a href="https://stackoverflow.com/questions/73599356/how-to-implement-laplace-posteriori-approximation-on-bert-in-pytorch" target="_blank" rel="noopener nofollow ugc">How to implement Laplace Posteriori Approximation on BERT in PyTorch?</a>
</h4>

<div class="tags">
  <strong>python, nlp, pytorch, bert-language-model</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stackoverflow.com/users/12134579/nikola-greb" target="_blank" rel="noopener nofollow ugc">
    Nikola Greb
  </a>
  on <a href="https://stackoverflow.com/questions/73599356/how-to-implement-laplace-posteriori-approximation-on-bert-in-pytorch" target="_blank" rel="noopener nofollow ugc">12:22PM - 04 Sep 22 UTC</a>
</div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Please help me if you know how.</p>
When I use Layer Norm I get an errro while without using Layer Norm I do not get this error

username: rebeen
cooked: <p>How to solve this problem</p>
<pre><code class="lang-auto">out_layer1 = nn.Sequential(
    nn.Conv2d(80, 32,kernel_size=3,groups=2),
    nn.LayerNorm(32),
    nn.Dropout(0.1)
)
data= torch.Tensor(64, 80, 3,3)
out = out_layer1(data)
print(out.shape)
</code></pre>
<p>RuntimeError: Given normalized_shape=[32], expected input with shape [*, 32], but got input of size[64, 32, 1, 1]</p>

username: KaiHoo
cooked: <pre><code class="lang-auto">data= torch.Tensor(64, 80, 3,3)
out_w, out_h = 1, 1
out_layer1 = nn.Sequential(
    nn.Conv2d(80, 32,kernel_size=3,groups=2),
    nn.LayerNorm([32,out_w,out_h]),
    nn.Dropout(0.1)
)
out = out_layer1(data)
print(out.shape)

data= torch.Tensor(64, 80, 5,5)
out_w, out_h = 3, 3
out_layer1 = nn.Sequential(
    nn.Conv2d(80, 32,kernel_size=3,groups=2),
    nn.LayerNorm([32,out_w,out_h]),
    nn.Dropout(0.1)
)
out = out_layer1(data)
print(out.shape)
</code></pre>

username: rebeen
cooked: <p>Thank you very much <a class="mention" href="/u/kaihoo">@KaiHoo</a></p>

username: Mota_Lee
cooked: <p>Could you tell me how to solve problems like this with other data? Or how do you fix this bug?</p>
STN training crash in Apple M1

username: Jack75
cooked: <p>I am trying to speed up STN sample which is in<br>
<a href="https://pytorch.org/tutorials/intermediate/spatial_transformer_tutorial.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/intermediate/spatial_transformer_tutorial.html</a></p>
<p>I have install nighty version with conda<br>
$ python</p>
<p>Python 3.9.12 (main, Apr 5 2022, 01:52:34)</p>
<p>[Clang 12.0.0 ] :: Anaconda, Inc. on darwin</p>
<p>Type “help”, “copyright”, “credits” or “license” for more information.</p>
<blockquote>
<blockquote>
<blockquote>
<p>import torch</p>
</blockquote>
</blockquote>
</blockquote>
<blockquote>
<blockquote>
<blockquote>
<p>import platform</p>
</blockquote>
</blockquote>
</blockquote>
<blockquote>
<blockquote>
<blockquote>
<p>print(platform.platform())</p>
</blockquote>
</blockquote>
</blockquote>
<p>macOS-12.3.1-arm64-arm-64bit</p>
<blockquote>
<blockquote>
<blockquote>
<p>torch.<strong>version</strong></p>
</blockquote>
</blockquote>
</blockquote>
<p>‘1.13.0.dev20220903’</p>
<blockquote>
<blockquote>
<blockquote>
<p>torch.device(‘mps’)</p>
</blockquote>
</blockquote>
</blockquote>
<p>device(type=‘mps’)</p>
<blockquote>
<blockquote>
<blockquote></blockquote>
</blockquote>
</blockquote>
<p>I modified the code<br>
<span class="hashtag-raw">#device</span> = torch.device(“cuda” if torch.cuda.is_available() else “cpu”)</p>
<h1><a name="speed-up-for-apple-m1-1" class="anchor" href="#speed-up-for-apple-m1-1"></a>speed up for Apple M1</h1>
<p>device = torch.device(‘mps’)</p>
<p>however, the kernel is crash<br>
-:27:11: error: invalid input tensor shapes, indices shape and updates shape must be equal</p>
<p>-:27:11: note: see current operation: %25 = “mps.scatter_along_axis”(%23, %arg0, %24, %1) {mode = 6 : i32} : (tensor&lt;150528xf32&gt;, tensor&lt;28xf32&gt;, tensor&lt;50176xi32&gt;, tensor) → tensor&lt;150528xf32&gt;</p>
<p>/AppleInternal/Library/BuildRoots/560148d7-a559-11ec-8c96-4add460b61a6/Library/Caches/com.apple.xbs/Sources/MetalPerformanceShadersGraph/mpsgraph/MetalPerformanceShadersGraph/Core/Files/MPSGraphExecutable.mm:1267: failed assertion `Error: MLIR pass manager failed’</p>
Class weights for CrossEntropyLoss

username: top.g
cooked: <p>I have an imbalanced dataset, and I’m trying to give custom weights to <a href="https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html" rel="noopener nofollow ugc">CrossEntropyLoss</a>. However, I’m not sure whether I should give higher weights to minority classes and lower weights to majority classes, or vice versa. What I understand is that the higher the weights the larger the penalization, but I found in this <a href="https://stackoverflow.com/a/67836741/900394" rel="noopener nofollow ugc">stackoverflow thread</a> that they’re passing the actual weights of the classes. I.e. if we have classes: A: 100, B: 200, C:300, then the weights would be: 1/6, 2/6, 3/6 respectively (or just 1, 2, 3). My question is: shouldn’t it be the inverse of that? I.e. 6, 3, 2?</p>
<p>I know that there are several formulas to compute the weight, but I’m asking about the general idea. Should the minority classes be given higher weights or lower weights in <code>CrossEntropyLoss</code>?</p>

username: ptrblck
cooked: <p>You would increase the weight for the minority class for add a penalty to the training for a wrong classification of it.<br>
<a href="https://discuss.pytorch.org/t/predictions-stuck-at-zero-when-positive-label-1-is-only-16-of-data/145203/4">This post</a> explains it with an example.</p>
Sharp edges when merging small images

username: A_Ebrahimi
cooked: <p>hi i have a question . i’m segmenting satellite images and for that i have to split images to smaller ones then apply the model . the problem is that when i’m merging the images together to make the whole image , the edge of my smaller images are visible . is there anyway to solve this problem</p>

username: KFrank
cooked: <p>Hi Alireza!</p>
<aside class="quote no-group" data-username="A_Ebrahimi" data-post="1" data-topic="160367" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/919ad9/48.png" class="avatar"> A_Ebrahimi:</div>
<blockquote>
<p>when i’m merging the images together to make the whole image , the edge of my smaller images are visible . is there anyway to solve this problem</p>
</blockquote>
</aside>
<p>Yes.  Look at the “overlap-tile strategy” described in the <a href="https://arxiv.org/abs/1505.04597" rel="noopener nofollow ugc">original U-Net paper.</a></p>
<p>The basic idea is that you split your large image to be segmented<br>
into <em>overlapping</em> smaller images, each of which is larger than its<br>
corresponding predicted (segmented) sub-image and contains enough<br>
context around its borders that there are no artifacts at the edges of<br>
the predicted sub-images.</p>
<p>When the predicted sub-images are then combined together to produce<br>
the final large predicted image, there are no edges where the sub-images<br>
were glued back together.</p>
<p>How well this scheme will work for you will depend on the details of the<br>
segmentation model you are using.  If you use the original U-Net model<br>
(properly), there will be no edge artifacts whatsoever interior to your<br>
final large predicted image.</p>
<p>Best.</p>
<p>K. Frank</p>

username: A_Ebrahimi
cooked: <p>thanks for your reply . actually i think u are right . i only have this problem for two classes which my model cant separate them very well .</p>
Results vary by batch size for MobileNet_V2 model on GPU

username: ringohoffman
cooked: <p>I’ve taken steps from the docs on <a href="https://pytorch.org/docs/stable/notes/randomness.html#reproducibility" rel="noopener nofollow ugc">reproducibility</a>, and I turned on eval mode so that batchnorm should not be affecting the results. I still get different results for different batch sizes for the same model. This does not happen when I use the CPU. Anything I’m missing?</p>
<pre><code class="lang-python">import os
import random

import numpy as np
import torch
from torch import hub
from torchvision.models import MobileNet_V2_Weights

os.environ["CUBLAS_WORKSPACE_CONFIG"] = ":4096:8"

np.random.seed(0)
random.seed(0)
torch.manual_seed(0)

torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False
torch.use_deterministic_algorithms(True)

model = hub.load(
    "pytorch/vision:v0.6.0",
    "mobilenet_v2",
    weights=MobileNet_V2_Weights.IMAGENET1K_V1,
    verbose=False,
).to("cuda:0").eval()

_input = torch.ones(size=(4,3,224,224), device="cuda:0")

output = model(_input)
half_output = model(_input[:2])

output, half_output
</code></pre>
<pre><code class="lang-auto">(tensor([[-0.2532,  0.0923, -0.4648,  ..., -1.6945, -0.1472,  0.9123],
...
        device='cuda:0', grad_fn=&lt;AddmmBackward0&gt;),
 tensor([[-0.2558,  0.0903, -0.4674,  ..., -1.6931, -0.1455,  0.9128],
...
        device='cuda:0', grad_fn=&lt;AddmmBackward0&gt;))
</code></pre>

username: ringohoffman
cooked: <p>I also saw this: <a href="https://discuss.pytorch.org/t/different-outputs-when-using-different-batch-size-only-on-cuda/155886/2" class="inline-onebox">Different outputs when using different batch size (only on cuda) - #2 by ptrblck</a></p>
<p>Does this constitute a different workload ?</p>

username: eqy
cooked: <p>Yes, different batch sizes can constitute different workloads. If you wish to verify that this is the case, you could check the kernels that are actually being executed for each case by e.g., running <code>nsys nvprof python mymodel.py</code> and comparing the profiler results with different batch sizes.</p>
What is the difference between torch.rand(size=(3,4)) and torch.rand(3,4))

username: AiDi_Arman
cooked: <p>Hello, I’m very new in Pytorch, what is the difference between torch.rand(size=(3,4)) and torch.rand(3,4))<br>
I don’t really understand the “size” means, I checked Docs but still not… please help, thanks. <img src="https://discuss.pytorch.org/images/emoji/apple/stuck_out_tongue_winking_eye.png?v=12" title=":stuck_out_tongue_winking_eye:" class="emoji" alt=":stuck_out_tongue_winking_eye:" loading="lazy" width="20" height="20"></p>

username: ptrblck
cooked: <p>There is no difference, since the first argument is the <code>size</code> arg and the name can thus be skipped.<br>
Both will return a tensor with the size <code>[3, 4]</code> containing random values drawn from a uniform distribution in <code>[0, 1)</code>:</p>
<pre><code class="lang-python">torch.rand(size=(3,4))
# tensor([[0.7919, 0.7024, 0.1932, 0.8998],
#         [0.5805, 0.9587, 0.5769, 0.1992],
#         [0.5168, 0.5733, 0.6506, 0.5737]])

torch.rand(3,4)
# tensor([[0.9716, 0.0944, 0.5419, 0.0745],
#         [0.8470, 0.7681, 0.2453, 0.4496],
#         [0.2281, 0.8196, 0.1646, 0.8195]])
</code></pre>

username: AiDi_Arman
cooked: <p>Thank you so much!!! <img src="https://discuss.pytorch.org/images/emoji/apple/smiling_face_with_three_hearts.png?v=12" title=":smiling_face_with_three_hearts:" class="emoji" alt=":smiling_face_with_three_hearts:" loading="lazy" width="20" height="20"></p>
Pytorch model training stops at the end of the first Epoch

username: rabrab
cooked: <p>Hello Everyone,<br>
I have seen related post about my issue, but after applying the advice and changing the num_workers, my training still gets intrrupted without any error as shown in screenshot bellow, I have a Pytorch 1.12.1+cpu  and I have set max_steps =1  and still it says trainer.fit stopped “max_steps=1 reached” and mine is 10, I don’t know where it takes this value.<br>
I would appreciate any help to solve this issue, thanks!</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/7/4719e1799d40a314bd7c96f8c0bd9079adc29d05.png" data-download-href="https://discuss.pytorch.org/uploads/default/4719e1799d40a314bd7c96f8c0bd9079adc29d05" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/7/4719e1799d40a314bd7c96f8c0bd9079adc29d05.png" alt="image" data-base62-sha1="a8ZlGacwcGft3d55kWVigRSA0CN" width="690" height="280" data-dominant-color="1B1B1B"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1899×772 62.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: rabrab
cooked: <p>I found the error, Actually it is due to parameter " FAST_DEV_RUN" that was set to true and that sets max_steps to 1  for debugging purposes, so in normal use:  FAST_DEV_RUN should be false</p>
(Urgent) Repacking Dataset after Modifications

username: hasnainnaeem
cooked: <p>I am performing some transformations on the Cifar10 dataset. Is there a way to repack the dataset in its original format?</p>
<p>I have searched for this extensively but could not come up with a solution. I am aware that I could save images and read them using ImageFolder class. But, this has certain limitations, only repacking will be compatible with the codebase.</p>
RuntimeError: shape '[128, 1, 1]' is invalid for input of size 1

username: PA_Nik
cooked: <p>I trained GAN and now i wanna test how it is gonna transfer style.So I have image <strong><code>torch.Size([1, 3, 256, 256])</code></strong>. I pass it to discriminator. Last conv layer is</p>
<pre><code class="lang-auto">nn.Conv2d(1024, 1, kernel_size=4, stride=1, padding=0, bias=False),
    # out: 1 x 1 x 1

    nn.Flatten(),
    nn.Sigmoid())
</code></pre>
<pre><code class="lang-auto">lat = model['discriminator'](images)
</code></pre>
<p>lat shape is <strong><code>torch.Size([1, 1])</code></strong>. Generator should get</p>
<pre><code class="lang-auto">#latent_size = 128
nn.ConvTranspose2d(latent_size, 1024, kernel_size=4, stride=1, padding=0, bias=False),
    nn.BatchNorm2d(1024),
    nn.ReLU(True),
</code></pre>
<p>I tried to use <strong><code>torch.view()</code></strong> to increase number of feature maps to 128,but then I got error that is in title. So how I can pass lat to generator</p>
<pre><code class="lang-auto">new_img = model['generator'](lat)
</code></pre>
<p>I guess I should disable flatten and sigm for test and somehow increase number of feature maps to 128</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="PA_Nik" data-post="1" data-topic="160654">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pa_nik/48/52759_2.png" class="avatar"> PA_Nik:</div>
<blockquote>
<p>I tried to use <strong><code>torch.view()</code></strong> to increase number of feature maps to 128</p>
</blockquote>
</aside>
<p><code>view</code> cannot increase the number of elements but is used in a similar way as <code>reshape</code>, but avoids a copy. If you want to increase the number of elemens, use <code>repeat</code> or <code>expand</code>.</p>
Seriously ... torch::stack not works! please

username: BongHoe_Koo
cooked: <p>Hi team.</p>
<p>I was wrapping pytorch library for my project.<br>
but it seems torch::stack not works in debug mode.<br>
it only works in release mode.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/c/0cbe156c39087e182ca19adbdde8d7e5b28136dd.png" data-download-href="https://discuss.pytorch.org/uploads/default/0cbe156c39087e182ca19adbdde8d7e5b28136dd" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/c/0cbe156c39087e182ca19adbdde8d7e5b28136dd.png" alt="image" data-base62-sha1="1OIYT1I68kaUDswVVjTWVtcWIrj" width="690" height="254" data-dominant-color="4B4B4B"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1236×455 31.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>currently i am using newest libtorch.<br>
could you guys help me?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/d/5dff6a29fad0a65ac14e796d466cad75f142e378.png" data-download-href="https://discuss.pytorch.org/uploads/default/5dff6a29fad0a65ac14e796d466cad75f142e378" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/d/5dff6a29fad0a65ac14e796d466cad75f142e378_2_690x246.png" alt="image" data-base62-sha1="dpxCWUDehcyPHCvl573fXymBCUo" width="690" height="246" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/d/5dff6a29fad0a65ac14e796d466cad75f142e378_2_690x246.png, https://discuss.pytorch.org/uploads/default/original/3X/5/d/5dff6a29fad0a65ac14e796d466cad75f142e378.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/d/5dff6a29fad0a65ac14e796d466cad75f142e378.png 2x" data-dominant-color="F5DFDC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1001×358 26.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: BongHoe_Koo
cooked: <p>Also please test it on window platform please</p>
<p>this is my environment info (VS2022)<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/4/4/44b4ffd0964f524e8360c045e46ae4a211482705.png" alt="image" data-base62-sha1="9NOfWux7WDGP4CGyJSTafEPEuEJ" width="189" height="84"></p>

username: ptrblck
cooked: <p>This might be a Windows issue so I would suggest to create a GitHub issue with a minimal code snippet instead of cross-posting the same post into different threads here.</p>
How can Torch.FX work with autograd.Function?

username: Chaofei
cooked: <p>My code need to customize a backward function using autograd.Function, but if I use fx.symbolic_trace to transform the model, I find my customized backward function is not traced.<br>
How can I get around this?<br>
For example:</p>
<pre><code class="lang-auto">import torch
from torch.fx import symbolic_trace

class TestModule(torch.nn.Module):
    def __init__(self):
        super(TestModule, self).__init__()
        class ActFun(torch.autograd.Function):
            @staticmethod
            def forward(ctx,input):
                output = input.gt(0.0).float()
                return output
            @staticmethod
            def backward(ctx,grad_output):
                return grad_output
        self.act = ActFun()
        self.func = self.act.apply

    def forward(self, x):
        return self.func(x)

mfunc = TestModule()
mfunc = symbolic_trace(mfunc)
a = torch.ones(1, requires_grad=True)
b = mfunc(a)
b.backward()
print(b, a.grad)
</code></pre>

username: bosskwei
cooked: <p>similar problem, and autowrap_function not working on autograd.Function.apply</p>
No_grad to submodule

username: tsp_t
cooked: <p>I want to set submodule as no_grad for reduce memory.<br>
Specifically, I consider the following example</p>
<pre><code class="lang-auto">import timm
model = timm.create_model("convnext_small_in22k")
stem = model.get_submodule("stem")
model.__setattr__("stem", torch.no_grad()(stem))
</code></pre>
<pre><code class="lang-auto">TypeError: cannot assign 'torch.nn.modules.container._DecoratorContextManager.__call__.&lt;locals&gt;.decorate_context' as child module 'stem' (torch.nn.Module or None expected)
</code></pre>
<p>Please tell me how to modify submodel to have no_grad attribute<br>
(requires_grad = False is not a solution)</p>
<p>+ I don’t want to use a wrapper class that runs no_grad forward.<br>
+ After applying no_grad , model.named_parameters() should return the same name.</p>
<p>+ The current makeshift is to use a wrapper class.</p>
<pre><code class="lang-auto">class no_grad_wrapper(nn.Module):
    def __init__(self, module):
        super().__init__()
        self.M = module

    @torch.no_grad()
    def forward(self, x):
        return self.M(x)
</code></pre>
<p>But this is not good because it creates a letter <code>M</code> in named_parameters that doesn’t exist</p>
<p>How to handle it without creating a variable, or is it possible to apply no_grad statically?</p>
How to use torch::nn in LibTorch for Android

username: vujadeyoon
cooked: <p>Dear all,</p>
<p>I have develop a deep-learning model with the LibTorch which is C++ based Torch for an android application. To be specific, I use JNI in order to call C++ LibTorch code.<br>
You can refer to the <a href="https://pytorch.org/tutorials/recipes/android_native_app_with_custom_op.html" rel="noopener nofollow ugc">LibTorch_Android</a> for LibTorch on the Android.</p>
<p>I hope to use below functions, but I cannot load the namespace, torch::nn.</p>
<ul>
<li>torch::nn::functional::pad</li>
<li>torch::nn::functional::PadFuncOptions</li>
<li>torch::nn::functional::grid_sample</li>
<li>torch::nn::functional::GridSampleFuncOptions</li>
</ul>
<p>I attach my CMakeLists.txt and native-lib.cpp which is called by JNI.<br>
CMakeLists.txt</p>
<pre><code class="lang-bash"># LibTorch
file(GLOB PYTORCH_INCLUDE_DIRS "${PATH_BUILD}/pytorch_android*.aar/headers")
file(GLOB PYTORCH_LINK_DIRS "${PATH_BUILD}/pytorch_android*.aar/jni/${ANDROID_ABI}")
set(BUILD_SUBDIR ${ANDROID_ABI})
target_include_directories(${PROJECT_NAME} PRIVATE ${PYTORCH_INCLUDE_DIRS} ${PYTORCH_INCLUDE_DIRN})
find_library(PYTORCH_LIBRARY pytorch_jni PATHS ${PYTORCH_LINK_DIRS} NO_CMAKE_FIND_ROOT_PATH)
</code></pre>
<p>native-lib.cpp</p>
<pre><code class="lang-cpp">#include "torch/script.h"
auto temp = torch::nn::functional::PadFuncOptions({1, 1, 1, 1});
</code></pre>
<p>Could I use torch::nn on Android? What should I do?</p>
<p>Best regards,<br>
Vujadeyoon</p>
Tensorboard writer add_histogram bug

username: calwritescode
cooked: <p>Since I have a problem with vanishing gradients while training my model I added a writer to monitor the models gradients</p>
<pre><code class="lang-auto">for name, param in self.model.named_parameters():
    self.writer.add_histogram(f"{name}.grad", param.grad, iteration)
</code></pre>
<p>this lead to <code>ValueError("The histogram is empty, please file a bug report.")</code></p>
<p>even after changing that to</p>
<pre><code class="lang-auto">for name, param in self.model.named_parameters():
    if torch.isnan(param.grad).any():
        self.logger.warn(
            f"Gradient {name}.grad is NaN in iteration {iteration}"
        )
    else:
        self.writer.add_histogram(f"{name}.grad", param.grad, iteration)
</code></pre>
<p>I still get the same error.</p>
<p>This issue <a href="https://github.com/pytorch/pytorch/issues/31855" class="inline-onebox" rel="noopener nofollow ugc">The histogram is empty · Issue #31855 · pytorch/pytorch · GitHub</a> was closed as resolved but multiple user still seem to have the same issue.</p>
<p>Is there something I am missing?</p>
<p>Thanks in advance</p>

username: ptrblck
cooked: <p>The warning is raised from <a href="https://github.com/pytorch/pytorch/blob/4d3f7df33fa49a80b20ae328d5a5d71c655e3634/torch/utils/tensorboard/summary.py#L401-L402">here</a>. Could you check if the passed <code>param.grad</code> value might be empty?</p>

username: calwritescode
cooked: <p>I added</p>
<pre><code class="lang-auto">for name, param in self.model.named_parameters():
    try:
        self.writer.add_histogram(f"{name}.grad", param.grad, iteration)
    except:
        self.logger.warn(
            f"{name} with min:{torch.min(param.grad)} and max:{torch.max(param.grad)} is {param.grad}"
        )
</code></pre>
<p>For some batches the gradients are printed, e.g.:<br>
<code>D with min:4.3589534497664784e+26 and max:4.3589534497664784e+26 is tensor([4.3590e+26], device='cuda:0')</code></p>
<p>This does not look empty to me but the try block still fails.</p>

username: ptrblck
cooked: <p>I guess trying to create a histogram from a single value fails (which would be expected) so you might need to check if the gradient contains multiple values.</p>

username: calwritescode
cooked: <pre><code class="lang-auto">if param.grad.size(dim=0) &gt;= 2:
    self.writer.add_histogram(f"{name}.grad", param.grad, iteration)
</code></pre>
<p>solved it. Thank you!</p>
Put lazily created parameters on the correct device

username: TriKri
cooked: <p>I have a custom module that creates some parameters lazily. These parameters are initialized as <code>None</code>, but when <code>forward(input)</code> is called, these parameters are “created,” that is, changed to instances of the <code>torch.nn.parameter.Parameter</code> class. The reason I do this is because the shapes of those parameters depend on the shape of the input to the module, which I don’t know until I have used it once. (As a side note, for this reason, I also need to pass an example through the network once before creating the optimizer for the lazily created parameters to be registered in the optimizer.)</p>
<p>But how do I know which device to put the parameters on? I can pass a <code>device</code> parameter to the module’s<code> __init__</code> method, but if the module, <code>a</code>, is a child module of another module, <code>b</code>, and the programmer that uses the module does <code>b.to(other_device)</code>, then the <code>to</code> method of <code>a</code> will not be called so <code>a</code> will not know that all parameters should now be located on <code>other_device</code> rather than on <code>device</code>, and consequently the parameters will end up on the wrong device when created.</p>
<p>Any ideas for how to solve this?</p>

username: ptrblck
cooked: <p>Since you are already using an init forward pass to create the parameters and register them, you might also want to use the <code>.device</code> attribute of the input to move the parameters to the sme device. I don’t know how well this approach would work with a data parallel approach, but it might be fine for single GPU use cases.<br>
Note that PyTorch also provides lazy modules via <code>nn.Lazy*</code> (e.g. <code>nn.LazyLinear</code>) so have you checked them as they might be a clean replacement for your custom approach.</p>

username: TriKri
cooked: <p>Using the same device as the input to the layer seems like a valid approach; I didn’t think of that! Thank you very much!</p>
Is it possible to speed up the inference on multi-core CPU machine with DDP

username: MathewAlexander
cooked: <p>I have a pre-trained transformer model (say LayoutLMv2). I am trying to build a real time API where I have to do about 50 separate inferences on this model (50 images<br>
from a document).<br>
I am trying to speed up the API without having to deploy it on GPU.<br>
Is it possible to parallelize this with DDP  and have a better response time if I am using a multi-core CPU machine?<br>
Are there any practical examples for speeding up inference alone (of any torch models) on a CPU machine with DDP?</p>

username: eqy
cooked: <p>For a CPU machine, you may want to check the existing utilization of the machine without DDP as it is likely that it is already parallelized across the batch dimension by e.g., OneDNN/MKLDNN. If this is the case I would not expect DDP to offer any speedup as it would essentially add a layer of communication and indirection when one is not needed. You can likely also tweak the amount of parallelization via e.g., <code>OMP_NUM_THREADS</code> and I believe it is a common practice to set this to the number of physical CPU cores (rather than total core count because vector units are typically shared across SMT lanes) to maximize throughput.</p>