akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Whisper for C2C rather then Seq2Seq

P-Sood: <p>Hello, as some of you may know, Whisper is a Seq2Seq model, whereas Wav2Vec2ForC2C is a C2C model. The main difference here is that Whisper needs decoder_input_ids, whereas Wav2Vec2 does not.</p>
<p>My question is, can I just use Whispers Encoder model and attach a C2C head over the top of that to leverage both Whispers generality and C2C to get the output I desire? Or would this clearly not work?</p>
<p>Any help would be appreciated, but a simple yes or no will suffice as well. Thank you in advance!</p>
Application to interactively test pytorch data augmentation

diegulio: <p>Hello! We all know that Data Augmentation is very useful to improve our model quality. Here I want to share an application to interactively test different data augmentations in your images. So you will decide more confidently which ones are better to your use case.</p>
<p>App: <a href="https://pytorch-transform-illustrations.streamlit.app" rel="noopener nofollow ugc">https://pytorch-transform-illustrations.streamlit.app</a></p>
Caffe Ops vs ATen Ops

Jerome_Ku: <p>What is the difference between Caffe <a href="https://github.com/pytorch/pytorch/blob/main/caffe2/operators" rel="noopener nofollow ugc">ops</a> and those in ATen <a href="https://github.com/pytorch/pytorch/tree/main/aten/src/ATen" rel="noopener nofollow ugc">native</a>?</p>

ptrblck: <p>Caffe2 was merged into PyTorch a while ago with the idea to combine development efforts into one code base. However, Caffe2 is deprecated and you might want to ignore these ops.</p>
Connecting units to each other

alfredl7: <p>Hi there,</p>
<p>I’m working on developing a attractor network that requires the units of a layer to connect to each other, in addition to the output and input connections. I’m not quite sure how it can implemented without using a recurrent layer, or am I restricted to using a recurrent layer with some modifications? Thanks.</p>

ptrblck: <p>I’m not sure I understand the use case correctly. Could you post a drawing showing the schema you are working with?</p>

alfredl7: <p>Thank you for your prompt response.</p>
<p>The purpose of the network is to simulate how human acquire language (i.e., cognitive modelling). Schema as follows:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/7/3782ba7075c82f36cc9c6777acbb15ed0a023e49.png" data-download-href="https://discuss.pytorch.org/uploads/default/3782ba7075c82f36cc9c6777acbb15ed0a023e49" title="Capture"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/7/3782ba7075c82f36cc9c6777acbb15ed0a023e49_2_639x500.png" alt="Capture" data-base62-sha1="7V4lnS0NpQ5sO5PIh7rgxcmNC4x" width="639" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/7/3782ba7075c82f36cc9c6777acbb15ed0a023e49_2_639x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/3/7/3782ba7075c82f36cc9c6777acbb15ed0a023e49_2_958x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/3/7/3782ba7075c82f36cc9c6777acbb15ed0a023e49.png 2x" data-dominant-color="ECECEC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Capture</span><span class="informations">988×772 208 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>In particular, I’m having trouble trying to connect the Phonological Units to one another.</p>

HernandoR: <p><a class="mention" href="/u/alfredl7">@alfredl7</a>  Any success?</p>
Tensor error while applying MobileViTV2

nijatmursali: <p>I’m trying to apply <a href="https://arxiv.org/abs/2206.02680" rel="noopener nofollow ugc">MobileViTV2 mentoned in this document</a> using Timm. I found model from HuggingFace and modified the <code>config</code> file, and when training starts all is good with it, but I get</p>
<pre><code class="lang-auto">RuntimeError: The size of tensor a (4165) must match the size of tensor b (1045) at non-singleton dimension 1
</code></pre>
<p>I found the issue where in <a href="https://github.com/RangiLyu/nanodet/blob/main/nanodet/model/head/nanodet_plus_head.py" rel="noopener nofollow ugc">/nanodet/model/head/nanodet_plus_head.py</a></p>
<pre><code class="lang-auto">        dis_preds = self.distribution_project(reg_preds) * center_priors[..., 2, None]
</code></pre>
<p>where <code>center_priors</code> has <code>torch.Size([32, 1045, 4])</code> and <code>reg_preds</code> has <code>torch.Size([32, 4165, 32])</code></p>
<p>and <code>distribution_project </code> is just a Integral  <code>self.distribution_project = Integral(self.reg_max)</code></p>
<p>How can I solve this issue?</p>
<p>My config file looks like:</p>
<pre><code class="lang-auto">model:
  weight_averager:
    name: ExpMovingAverager
    decay: 0.9998
  arch:
    name: NanoDetPlus
    detach_epoch: 10
    backbone:
      name: TIMMWrapper
      model_name: mobilevitv2_100
      features_only: True
      pretrained: True
      out_indices: [1, 2, 3]
    fpn:
      name: GhostPAN
      in_channels: [128, 256, 384]
      out_channels: 128
      kernel_size: 5
      num_extra_level: 1
      use_depthwise: True
      activation: SiLU
    head:
      name: NanoDetPlusHead
      num_classes: 80
      input_channel: 128
      feat_channels: 128
      stacked_convs: 2
      kernel_size: 5
      strides: [ 8, 16, 32, 64 ]
      activation: SiLU
      reg_max: 7
      norm_cfg:
        type: BN
      loss:
        loss_qfl:
          name: QualityFocalLoss
          use_sigmoid: True
          beta: 2.0
          loss_weight: 1.0
        loss_dfl:
          name: DistributionFocalLoss
          loss_weight: 0.25
        loss_bbox:
          name: GIoULoss
          loss_weight: 2.0
    aux_head:
      name: SimpleConvHead
      num_classes: 80
      input_channel: 256
      feat_channels: 256
      stacked_convs: 4
      strides: [ 8, 16, 32, 64 ]
      activation: SiLU
      reg_max: 7
</code></pre>
How to treat shared part of a model and its ema version in train

LuoXin-s: <p>Hi, I have a pretrained frozen model <code>net</code>, then I’d like to train a another model <code>pre_net</code>, so it will process input <code>x</code> as <code>net(pre_net(x))</code>. My training loss is constructed similar to contrastive learning: <code>d(net(pre_net(x)), net(pre_net_ema(x_t)))</code>. <code>pre_net_ema</code> is ema version of <code>pre_net</code>, and <code>x_t</code> is a transformation of <code>x</code>.</p>
<p>My question is which one should I choose between <code>.detach()</code> or <code>torch.no_grad()</code> when calculate <code>net(pre_net_ema(x_t))</code>, I 'm sure <code>.detach()</code> is a correct choice, while <code>torch.no_grad()</code> do not construct graph and will be faster than <code>.detach()</code> theoretically. However, I’m not sure where or not using <code>torch.no_grad()</code> is a correct choice for my setting.</p>
Struct type as a class argument

robinho: <pre><code class="lang-auto">class people:
    def __init__(self,a,b):
    self.a=a
    self.b=
</code></pre>
<p>I want self.b to be a struct type in C/matlab so that self.b.field1=field1, self.b.field2=field2. How to do this?</p>

ptrblck: <aside class="quote no-group" data-username="robinho" data-post="1" data-topic="190828">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/r/c6cbf5/48.png" class="avatar"> robinho:</div>
<blockquote>
<p>I want self.b to be a struct type in C/matlab</p>
</blockquote>
</aside>
<p>I don’t understand this requirement, since you are creating a Python class, which cannot directly interact with C/Matlab. You could transform raw Matlab objects, but still I’m unsure if that’s your use case.</p>
<aside class="quote no-group" data-username="robinho" data-post="1" data-topic="190828">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/r/c6cbf5/48.png" class="avatar"> robinho:</div>
<blockquote>
<p>so that self.b.field1=field1, self.b.field2=field2</p>
</blockquote>
</aside>
<p>You can directly assign attributes in Python. Also, your question doesn’t seem to be related to PyTorch, so a Python-specific discussion board might be a better place to ask.</p>

robinho: <p>Thank you, I’ll try finding a Python forum. BTW,  my intention is not to interact with C/Matlab, but to mimic them to create self.b variable in this class so that self.b.field1=field1, self.b.field2=field2 etc can be created afterwards</p>
Customized efficient implementation of scaled_dot_product_attention

deJQK: <p>I would like to implement one customized function similar to <code>scaled_dot_product_attention</code>, as shown in <a href="https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html" rel="noopener nofollow ugc">this doc</a>. Specifically, suppose I want to replace the softmax with some modified function with learned parameter <strong>w</strong>, but maintaining the potential efficient implementation from the lower end of CUDA. I notice from <a href="https://discuss.pytorch.org/t/i-cant-find-the-source-code-of-scaled-dot-product-attention/169389">this discussion</a> about <a href="https://github.com/pytorch/pytorch/blob/51c6c5e156c64d84ff0cd06a559fa6786c96f128/aten/src/ATen/native/transformers/attention.cpp#L692" rel="noopener nofollow ugc">the cuda function code</a>.</p>
<p>What could be the best practice to implement what I expect? I know how to implement this with pytorch but I am afraid this might be not efficient during training compared to the default efficient implementation.</p>
<p>Or is there recommended doc of customized CUDA function for pytorch? Maybe <a href="https://pytorch.org/tutorials/advanced/cpp_extension.html" rel="noopener nofollow ugc">this one</a>?</p>
<p>Thanks!</p>

ptrblck: <p>Yes, C++/CUDA extensions are a supported approach to write custom implementations and to pipe them to PyTorch as fucntions or layers.</p>

deJQK: <p>If I would like to customize <code>nn.functional.scaled_dot_product_attention</code>, is implementing the three functions <code>at::_scaled_dot_product_flash_attention</code> <code>at::_scaled_dot_product_efficient_attention</code> <code>at::_scaled_dot_product_attention_math</code> enough, or I also need to implement other I/O functions? BTW, where are the definition of <code>at::_scaled_dot_product_flash_attention</code> and <code>at::_scaled_dot_product_efficient_attention</code>?</p>
FSDP: Different learning rate for different weights

dome272: <p>Hey there,<br>
is it possible to have different learning rates for different weights of a model wrapped in FSDP?</p>
<pre><code class="lang-auto">    model = FSDP( 
        model, auto_wrap_policy=fsdp_auto_wrap_policy, sharding_strategy=fsdp_sharding_strategy, 
        mixed_precision=fsdp_mixed_precision, cpu_offload=fsdp_cpu_offload, device_id=device,
        limit_all_gathers=fsdp_limit_all_gathers,
    )

    # SETUP OPTIMIZER, SCHEDULER &amp; CRITERION
    normal_params = nn.ParameterList()
    clip_params = nn.ParameterList()
    for k, v in model.named_parameters():
        print(k)
        if k.startswith("_fsdp_wrapped_module.clip"):
            clip_params.append(v)
        else:
            normal_params.append(v)
    print(f"Normal Parameters: {sum([p.numel() for p in normal_params])}")
    print(f"Clip Parameters: {sum([p.numel() for p in clip_params])}")


    optimizer = optim.AdamW([{"params": normal_params}, {"params": clip_params, "lr": lr_clip}], lr=lr) 
</code></pre>
<p>This only seems to get a fraction of the parameters and as a result the clip_params are empty. Is there a way to make this work?</p>
<p>FSDP config:</p>
<pre><code class="lang-auto">fsdp_auto_wrap_policy = ModuleWrapPolicy([ResBlock, AttnBlock, TimestepBlock, FeedForwardBlock])
fsdp_sharding_strategy = ShardingStrategy.SHARD_GRAD_OP 
fsdp_fullstate_save_policy = FullStateDictConfig(offload_to_cpu=True, rank0_only=True)
fsdp_cpu_offload = None
fsdp_mixed_precision = MixedPrecision(
    param_dtype=torch.bfloat16,
    reduce_dtype=torch.bfloat16,
    buffer_dtype=torch.bfloat16,
)
fsdp_limit_all_gathers = True # False
</code></pre>
Use MPS trained model on Windows

MidKnightXI: <p>Hello I trained a model with MPS on my M1 Pro but I cannot use it on Windows or Linux machines using x64 processors.</p>
<p>I was wondering if that was possible in general to do that, because I need to distribute it to these types of machines and my macbook is the most powerful machine I have currently and training it on the CPU is taking me way too much time to make it worth doing it.</p>
<p>I am a beginner in AI and don’t know really much about pytorch, sorry about my ignorance.</p>
<p>For more informations about my project, here is the link to the code I’m using to train the model: <a href="https://github.com/MidKnightXI/BlurWarp/blob/main/trainer.py" rel="noopener nofollow ugc">https://github.com/MidKnightXI/BlurWarp/blob/main/trainer.py</a></p>

ptrblck: <p>Why can’t you use the <code>state_dict</code> on other systems? Do you see any specific error messages?</p>

MidKnightXI: <p>This is the error I’m having:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/3/a36cc4fb4f561efeff1f60555aaf79c3ff3a56fd.png" data-download-href="https://discuss.pytorch.org/uploads/default/a36cc4fb4f561efeff1f60555aaf79c3ff3a56fd" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/3/a36cc4fb4f561efeff1f60555aaf79c3ff3a56fd.png" alt="image" data-base62-sha1="njIRDJCvOfCQOscVcXqXAFrruhL" width="690" height="469" data-dominant-color="1B1B1B"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">782×532 15.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>But on macos it works just fine</p>

KMT62: <p>device=“mps” is Apple-only (metal). You’ll need to change your device to whatever you have on your PC, e.g. “cuda”, “xmp”, etc.</p>

ptrblck: <p>As <a class="mention" href="/u/kmt62">@KMT62</a> pointed out it seems the device is stored in the actual data. Try to use <code>map_location="cpu"</code> in the <code>torch.load</code> operation or move your model to the CPU before saving its <code>state_dict</code>.</p>

MidKnightXI: <p>Sorry if I wasn’t clear but I already know that, I just want to know if it is possible to train via the metal API and then make it compatible with other devices when saving the model</p>
Pytorch profiler does not record CUDA stacktrace

Monkey.Py: <p>I am using the following code from the tutorial : <a href="https://pytorch.org/tutorials/recipes/recipes/profiler_recipe.html" class="inline-onebox" rel="noopener nofollow ugc">PyTorch Profiler — PyTorch Tutorials 2.2.0+cu121 documentation</a></p>
<pre><code class="lang-auto">import torch
import torchvision.models as models
torch.cuda.empty_cache()
gc.collect()


model = models.resnet18().cuda()
print(model.device)
inputs = torch.randn(5, 3, 224, 224).cuda()

with profile(activities=[ProfilerActivity.CPU, ProfilerActivity.CUDA]) as prof:
    model(inputs)

prof.export_chrome_trace("trace.json")
</code></pre>
<p>Yet the chrome trace only contains CPU trace and not GPU (while cuda() seems to work properly).</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/b/3ba6583ae76ef53d92054b55b68e135ee3109f06.png" data-download-href="https://discuss.pytorch.org/uploads/default/3ba6583ae76ef53d92054b55b68e135ee3109f06" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/b/3ba6583ae76ef53d92054b55b68e135ee3109f06_2_690x87.png" alt="image" data-base62-sha1="8vGySkXjR6b3DUuWMxhU4fDsYXs" width="690" height="87" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/b/3ba6583ae76ef53d92054b55b68e135ee3109f06_2_690x87.png, https://discuss.pytorch.org/uploads/default/optimized/3X/3/b/3ba6583ae76ef53d92054b55b68e135ee3109f06_2_1035x130.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/3/b/3ba6583ae76ef53d92054b55b68e135ee3109f06_2_1380x174.png 2x" data-dominant-color="E4D9E8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1895×241 17.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Also my operation conv2d and convolution are described as cpu_op, I think it should be on gpu right ?</p>
<p>Can anyone explain why I have a different output, I would expect the GPU trace to be there.</p>
<p>Thank you !</p>
Running Llama2 locally on my device (MacBookPro Intel)

MaWieKA: <p>Hello I am Trying to run Llama2 with 70B-chat model but receive a PyTorch error or better said warning:</p>
<p>(textgen) maric@MacBook-Pro-7 llama-main % torchrun --nproc_per_node 8 example_chat_completion.py<br>
–ckpt_dir ~/llama-main/llama-2-70b-chat<br>
–tokenizer_path ~/llama-main/tokenizer.model<br>
–max_seq_len 512 --max_batch_size 6<br>
[2023-10-28 14:44:25,029] torch.distributed.elastic.multiprocessing.redirects: [WARNING] NOTE: Redirects are currently not supported in Windows or MacOs.<br>
[2023-10-28 14:44:25,065] torch.distributed.run: [WARNING]<br>
[2023-10-28 14:44:25,065] torch.distributed.run: [WARNING] *****************************************<br>
[2023-10-28 14:44:25,065] torch.distributed.run: [WARNING] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.<br>
[2023-10-28 14:44:25,065] torch.distributed.run: [WARNING] *****************************************</p>
<p>Did anyone face the same issue and how to solve it?<br>
I wanted to run Llama2 locally but facing error after error.</p>
Implementation of Faster RCNN with Resnet50 + FPN as backbone

Manu_Johny: <p>Hello everyone,</p>
<p>I have a question regarding the implementation of Faster RCNN with ResNet50 + FPN as backbone. I am using the implementation given by Pytorch:</p>
<pre><code class="lang-auto">model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True)
in_features = model.roi_heads.box_predictor.cls_score.in_features
model.roi_heads.box_predictor = FastRCNNPredictor(in_features, 3)
</code></pre>
<p>I have also heard that the standard anchor sizes are 32, 64, 128, 256, 512 and the aspect ratios are 0.5, 1, 2.</p>
<p>But when I look at the RPN header, it says the following:</p>
<blockquote>
<p>RPNHead(<br>
(conv): Sequential(<br>
(0): Conv2dNormActivation(<br>
(0): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))<br>
(1): ReLU(inplace=True)<br>
)<br>
)<br>
(cls_logits): Conv2d(256, 3, kernel_size=(1, 1), stride=(1, 1))<br>
(bbox_pred): Conv2d(256, 12, kernel_size=(1, 1), stride=(1, 1))<br>
)</p>
</blockquote>
<p>However, if I use the same anchor sizes and ratios but with the following program, cls_logit’s output size increases from 3 to 15 and bbox_pred’s output increases from 12 to 60:</p>
<pre><code class="lang-auto">model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True)
sizes = ((64,), (156,), (300,), (512,), (800,))
 aspect_ratios = ((0.5, 1.0, 2.0),) * len(sizes)
 anchor_generator = AnchorGenerator(sizes=sizes, aspect_ratios=aspect_ratios)
 model.rpn.anchor_generator = anchor_generator
 model.rpn.head = RPNHead(256, anchor_generator.num_anchors_per_location()[0])
 in_features = model.roi_heads.box_predictor.cls_score.in_features
 model.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_classes=3)
</code></pre>
<p>The second program is used to test other sizes. My question is: why is the num_anchors only 3 in the given implementation of FasterRCNN?</p>
<p>I would be very happy if you can help me.</p>
Specifying Kaiming initialization nonlinearity in case of softmax, low rank or a stack of linear layers, and layer norm

R-N: <p>I read that I should be initializing the weights instead of letting torch do it, and I want to initialize linear layers with Kaiming.  I have a few questions about specifying its nonlinearity.</p>
<ol>
<li>There’s no nonlinearity option for softmax, but since it ranges from 0 to 1, I assume it’s fine to use sigmoid nonlinearity?</li>
<li>If I’m using low rank linear layer, I’ll have two linear layers before activation. Should both have the nonlinearity of the activation function or just the second (last) one?</li>
<li>Should I still be using the nonlinearity of the activation function if I will have layer norm just right after the activation? If not, what should I use instead? Maybe tanh? I mean, normalization alters the range, but it’s not a nonlinear operation, or is it?</li>
</ol>
How to lazy_init for other backends

feifan: <p>hello,<br>
I’m using privateuse1 to add our backend. My customer find the following code is working in cuda, but not working in my backend.</p>
<pre><code class="lang-auto">import torch
tensor_a  = torch.rand(2).to("cuda")
</code></pre>
<p>In my backend, it will raise RuntimeError in <code>getDevice</code> in func <code>ensure_has_index</code>, because my device was not initialized.<br>
CUDA is ok is because of the <code>cuda_lazy_init</code> in <code>THPVariable_to</code>.<br>
So I want to discuss how privateuse1 to support this case</p>
Why would GradientScaler work

gradient: <p>According to this page <a href="https://pytorch.org/docs/stable/amp.html#gradient-scaling" class="inline-onebox" rel="noopener nofollow ugc">Automatic Mixed Precision package - torch.amp — PyTorch 2.1 documentation</a>, we have to scale the loss so that the gradient can be represented without underflow. I get this part, but when we call unscale_, it would essentially undo-the scaling as the name suggests. So wouldn’t unscale_ make the gradient underflow again?</p>

soulitzer: <p>Great question. To take the example of a linear, the weight is implicitly broadcasted in order to bmm with the input. Hence, there’s the corresponding accumulation/reduction/summation happening when you gradient of the weight using the gradient of the output of linear. Since we unscale after doing reduction, the underflow would not happen (because the reduction would increase the magnitude of the values).</p>

gradient: <p>Thanks for the reply! However, I am still a bit confused, let’s just say our batch_size=1 and the network is only one linear layer. Then how would the scaler work in this case? And moreover, what is the difference between accumulation and summation?</p>

soulitzer: <p>The other aspect is one which I maybe wouldn’t describe as “underflow”? I think it would be better thought of as “avoiding executing the function in a regime where you have lower precision”. This is idea that there is value in performing the function in higher precision and then unscaling versus performing the function in low precision and. And this effect can be compounding if you have many layers.</p>
How pytorch propagates gradient for LSTM

zabulon: <p>Hello!</p>
<p>When using an LSTM, and feeding it a sequence all in one learning step, how does the backprop through time work?</p>
<p>For example, if I use NLLLoss, and I have an input sequence of length 12 (“time steps”), like “I like torch” (embedded), I pass the outputs of LSTM and the targets and those also have length 12. NLLLoss reduces that to a number, from which backprop can start, but what does the graph for backprop look like? How does backprop know to iterate over the 12 “time steps” here?</p>

J_Johnson: <p>From what I can tell, the timesteps are now handled via for loop in C++ on the backend.</p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/main/aten/src/ATen/native/RNN.cpp">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/main/aten/src/ATen/native/RNN.cpp" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/main/aten/src/ATen/native/RNN.cpp" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/main/aten/src/ATen/native/RNN.cpp</a></h4>


      <pre><code class="lang-cpp">#define TORCH_ASSERT_ONLY_METHOD_OPERATORS
#include &lt;ATen/native/RNN.h&gt;

#include &lt;ATen/core/Tensor.h&gt;
#include &lt;ATen/core/List.h&gt;
#include &lt;ATen/Context.h&gt;
#include &lt;ATen/TensorOperators.h&gt;
#include &lt;ATen/mps/MPSDevice.h&gt;
#include &lt;ATen/native/quantized/PackedParams.h&gt;
#include &lt;ATen/native/quantized/cpu/fbgemm_utils.h&gt;
#include &lt;ATen/native/quantized/cpu/QnnpackUtils.h&gt;
#include &lt;c10/core/GradMode.h&gt;
#include &lt;c10/macros/Macros.h&gt;
#include &lt;c10/util/irange.h&gt;
#include &lt;torch/custom_class.h&gt;
#include &lt;torch/library.h&gt;
#include &lt;ATen/Config.h&gt;

#ifndef AT_PER_OPERATOR_HEADERS
#include &lt;ATen/Functions.h&gt;
</code></pre>



  This file has been truncated. <a href="https://github.com/pytorch/pytorch/blob/main/aten/src/ATen/native/RNN.cpp" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
Using same variable name in model calculation

Gears_Gears: <p>Hi!</p>
<p>I’m wondering if it makes a difference to reuse a variable name (from coder’s point of view) in model calculation.</p>
<p>For example, (1)</p>
<pre><code class="lang-auto">x = nn.module_1(input)
x = nn.module_2(x)
x = nn.module_3(x)
</code></pre>
<p>Versus (2)</p>
<pre><code class="lang-auto">x1 = nn.module_1(input)
x2 = nn.module_2(x1)
x3 = nn.module_3(x2)
</code></pre>
<p>Ignoring the case where one needs (I think) to use (2) (let’s say we need x2 but not x3 for another calculation), is (1) better?<br>
I’ve been using (2) to improve readability (I use longer variable names that explain what’s going on)</p>
<p>Many Thanks!</p>

soulitzer: <p>(1) is better because with (2) you’re explicitly keeping all the intermediates alive. If you don’t want to reuse variable names, you’d need to equivalently do:</p>
<pre><code class="lang-auto">x1 = nn.module_1(input)
x2 = nn.module_2(x1)
del x1
x3 = nn.module_3(x2)
del x2
</code></pre>
<p>(how much more memory you end up using in practice depends on the specific module and whether they are already save the inputs for backward)</p>

Gears_Gears: <p>Thanks very much for your help soulitzer! This cleared things up for me!</p>
Does Vision Transformer throw away information?

pumplerod: <p>I think I’ve wrapped my head around, mostly, how the vision transformer (ViT) works, but I’m a bit confused regarding how it makes use of the full embedding layer.</p>
<p>According to the paper, <a href="https://arxiv.org/abs/2010.11929" rel="noopener nofollow ugc">An image is worth 16x16 words</a>, A class token is attached at the head of our dimensional embedding, and for classification, that class token is extracted and passed through a MLP layer block in order to then output a final classification.  This part makes sense to me, however what I don’t understand is that if we are stripping off the <code>[class]</code> token to pass through the MLP, how is the rest of the information in the embedded dimension ever seen or used when training with back propagation?</p>
<p>pseudo code…</p>
<pre><code class="lang-auto">x = PatchAndEmbed( image)    # breaks up image into patches and creates embed 
x = torch.cat( class_token, x, dim=1)    # Places a pre-declared class token parameter at the 0'th position of the embedding
x = x + PositionalEmbedding()   # add sinusoidal positional embedding
x = TransformerEndoder( x)      # Passes batch of embeddings through the transformer/encoder
class_token = x[:, 0]    # Extracts class token from the embedding
output = mlp( class_token)   # passes the class token through mlp layer block for categorical logits output.
</code></pre>
<p>I’ve created my model, which works, but I’m not sure how.  I would expect that the back propagation process only ever passes back through the class token part of the embedding.  How is the rest of that information ever used/trained?</p>
Pytorch softmax classificatoin

Mohamed_Farag: <p>Hi everyone,</p>
<p>I am a little bit confused now, I have a small toy problem in binary classification and I am use CrossEntropyLoss() as an criterion without attaching LogSoftmax() at the end of the architecture, since it was written that the loss is calculating it implicitly. The question now, during testing, should I add LogSoftmax() or Softmax() ?</p>

ptrblck: <p>It depends which information and output you want to create during testing.<br>
If you are only interested in the predicted classes you can directly use <code>preds = torch.argmax(output, dim=1)</code> on the raw logits, since the order won’t change if <code>(log)softmax</code> is applied.<br>
On the other hand, if you want to see the probabilities for some reason you could apply a <code>softmax</code> to the output. Note however, that the loss calculation via <code>nn.CrossEntropyLoss</code> would still expect to receive raw logits.</p>
What is the impact of using x+=y in the forward method of Pytorch?

pytorchzzz: <p>I encountered the following error in backpropagation during model training (at the bottom). After changing x+=y to x=x+y, I solved this problem. What I know is that x+=y is modified on the original value of x, and x=x+y will generate new values. So, what is the impact of generating new values on the backpropagation of the model? There is another question, if the value cannot be modified in place, then my previous code had x+=y. Why can it run correctly? I cannot run it after adding some activation functions.<br>
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch. cuda. FloatTensor [256, 128, 310]], which is output 0 of ReluBackward0, is at version 1; Expected version 0 installed Hint: the backend further above shows the operation that failed to compute its gradient The variable in question was changed in there or anywhere later Good luck!</p>

J_Johnson: <p>If using inplace operations, from what I understand, PyTorch autograd cannot track the operation history to the graph and thus cannot determine the gradients. Please see here:</p>
<p><a href="https://pytorch.org/docs/stable/notes/autograd.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/notes/autograd.html</a></p>

ptrblck: <p>Autograd only disallows inplace operations on tensors when their original values are needed for the gradient computation, which means Autograd understands and accepts inplace ops only if they are mathematically correct.<br>
<a class="mention" href="/u/kfrank">@KFrank</a> explained it in more detail with a great example <a href="https://discuss.pytorch.org/t/when-inplace-operation-are-allowed-and-when-not/169583/3">here</a>.</p>
How important is ECC memory?

AlbertZeyer: <p>On the A10, we can enable or disable ECC. Disabling it would give us a bit more available memory and make it faster. I think by default, it is enabled, so I assume, most people use this. So what should we do?</p>

J_Johnson: <p>Error correction is ideal for very precise tasks where being off by a percent would devastate results.</p>
<p>That said, neural networks benefit from a lack of precision in the mantissa and it can act as an indirect form of regularization. Hence bfloat being a preferred dtype.</p>
<p>Additionally, it’s standard practice to use Dropout layers that just replace a given parameter with a zero 10-50% of the time. Similar effect without ECC but more equivalent to a model wide Dropout with p value of 0.004.</p>
<p>On the other hand, on rare occasions, you might get exploding gradients, such as if the error is in the exponent. However, from experience, that only occurs maybe 1/100 training runs.</p>

AlbertZeyer: <p>I assume that most people use ECC enabled, as this is the default, and they would not change it. So if this is basically some more memory and faster speed for everyone doing NN training, why isn’t this then more widely known to disable ECC? When I search for this, rarely anything shows up. So either people don’t care too much about getting a few more percent of speed and memory (but why?), or it’s not so safe as you say.</p>
<p>Did you directly compare ECC enabled vs disabled?</p>
<p>1% of training runs, this is very relative: On a single GPU? Trained for how long, what kind of model, what kind of data, how big is the model?</p>
<p>Exploding gradient is not necessarily from ECC. You can get the same also without just by the nature of NN training.</p>

J_Johnson: <p>When I was messing with ECC on/off, it was on a server I built with 6 Tesla K80s(12GB per unit x 6 = 72GB). Model size was around 300-500m parameters, but large batches,(in fact the model size is less relevant than the number of flops) and a training run was over the course of a day and a half. So I made sure periodic model saves took place regularly. As I recall, it was a noticeable speedup, although I don’t have any exact numbers.</p>
<p>You could probably avoid exploding gradients by making use of gradient clipping, data clipping, and parameter clipping.</p>
How do I fill the black or "blank" corners after I use rotation data augmentation?

inkplay: <pre><code class="lang-auto">transform = transforms.Compose([
        transforms.Resize(64),
        transforms.RandomRotation(360, resample=Image.BICUBIC),
        transforms.ToTensor(),
        transforms.Normalize(mean=(0.5, 0.5, 0.5), std=(0.5, 0.5, 0.5))
]) 
</code></pre>
<p>After I call RandomRotation I get this image</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/2X/e/e2fa62666c6ac083bc13997abeaf0e05d16dbf8d.png" alt="Screenshot_43" data-base62-sha1="wnWlYcGgf0wxJ4JZieSq7fnTcnP" width="339" height="345"></p>
<p>Its screwing up the GAN I am trying to train.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/2X/6/6d2c2c6ce6671b53db708623f622527ca42fa386.png" alt="Screenshot_44" data-base62-sha1="fzMGvCm32ta6kD6ervbk53GlH4a" width="513" height="315"></p>
<p>Whats the easiest way to fill the corners white?</p>

SimonW: <p>some fill with nearest pixels, but that may be hard to implement. alternatively, you can increase the size (e.g., by using nearest pixels to fill in blank areas. this is easy because it is shifting in axes-aligned directions), and then rotate.</p>

inkplay: <p>So let me get this straight I increase the white border size but not the content so when I rotate/crop I would still get to keep the corners? If so how do I increase the size is there a direct function you can call in pytorch?</p>

SimonW: <p>Ideally you would want to do that in <code>dataset.__getitem__</code>, where you can operate on np arrays and/or PIL images. There are plenty of useful functions there.</p>

stherdy: <p>Your images seem to have a white background. In this case you can set the fill parameter of torchvision.transforms.RandomRotation with transforms.RandomRotation(360, fill=255) in order to fill the black corners. For more information please have a look at the official documentation: <a href="https://pytorch.org/vision/main/generated/torchvision.transforms.RandomRotation.html" class="inline-onebox" rel="noopener nofollow ugc">RandomRotation — Torchvision main documentation</a></p>
<p>I hope this helps.<br>
All the best,<br>
Stefan</p>
Custom faster-rcnn builder

mg.group: <p>Dear Support,<br>
can you provide some insights in the following function:</p>
<p><a href="https://github.com/pytorch/vision/blob/a63f98add5d67f85a5eac6ede8ca850d3b2bbfd3/torchvision/models/detection/faster_rcnn.py#L567" rel="noopener nofollow ugc">builder</a></p>
<p>I want to implement my own builder to build a faster rcnn using my custom Resnet50. Maybe it is not enough to assign my resnet50 to the backbone of the faster rcnn as suggested in some posts elsewhere. I mean, the builder applies some processing to the original resnet50 I suppose I need to replicate them, is that correct?</p>
<p>Can you point to the strictly necessary lines of code within the builder I need to include?</p>
<p>I apologize for the topic maybe I failed to explain plainly.</p>
<p>Many thanks</p>
Is there a convenient method of making sure all tensors have been moved to the correct device on a CPU only machine?

Jbartlett6: <p>When debugging my code on a CPU only machine, is there a convenient way to check all required tensors have been moved to the correct device prior to using the code to train the model on a machine with GPU access? I was wondering if there was maybe some kind of ‘dummy’ CPU device I could move the tensors to that would throw a similar error.</p>
<p>For context I am developing a model on my local machine which will be trained on a HPC cluster I have access to. However, while possible, its inconvenient to debug the code on this cluster.</p>
How to apply c++ code after successful pytorch android build?

Jin_Pyo_Jun: <p>pytorch build for android was completed as follows.</p>
<pre><code class="lang-bash">$ ANDROID_NDK=.../android-ndk-r21e
$ ANDROID_ABI="arm64-v8a"
$ cd scripts
$ ./build_android.sh
... (long time) ...
[100%] Linking CXX static library ../lib/libtorch.a
[100%] Built target torch
Install the project...
-- Install configuration: "Release"
Installation completed, now you can copy the headers/libs from /mnt/HDD_2TB/greenfish/pytorch_build/work.trial9/pytorch/build_android/install to your Android project directory.
</code></pre>
<p>the install dir is as follows.</p>
<pre><code class="lang-auto">/mnt/HDD_2TB/greenfish/pytorch_build/work.trial9/pytorch/build_android/install
├── include
│   ├── ATen
│   ├── TH
│   ├── THCUNN
│   ├── c10
│   ├── caffe2
│   ├── fp16
│   ├── pybind11
│   └── torch
├── lib
└── share
    ├── ATen
    └── cmake
</code></pre>
<p>example code:<br>
<strong>/mnt/HDD_2TB/greenfish/pytorch_build/work.trial9/src/test.cpp</strong></p>
<pre><code class="lang-c++">#include &lt;torch/torch.h&gt;
#include &lt;iostream&gt;

int main() {
  torch::Tensor tensor = torch::eye(3);
  std::cout &lt;&lt; tensor &lt;&lt; std::endl;
}
</code></pre>
<p><strong>/mnt/HDD_2TB/greenfish/pytorch_build/work.trial9/src/CMakeLists.txt</strong></p>
<pre><code class="lang-auto">cmake_minimum_required(VERSION 3.0 FATAL_ERROR)
project(test)

find_package(Torch REQUIRED)

add_executable(test test.cpp)
target_link_libraries(test "${TORCH_LIBRARIES}")
set_property(TARGET test PROPERTY CXX_STANDARD 14)
</code></pre>
<p><strong>build</strong></p>
<pre><code class="lang-auto">src$ mkdir build
src$ cd build
build$ cmake ..
...
...
...
-- Detecting CXX compile features - done
CMake Error at CMakeLists.txt:4 (find_package):
  By not providing "FindTorch.cmake" in CMAKE_MODULE_PATH this project has
  asked CMake to find a package configuration file provided by "Torch", but
  CMake did not find one.

  Could not find a package configuration file provided by "Torch" with any of
  the following names:

    TorchConfig.cmake
    torch-config.cmake

  Add the installation prefix of "Torch" to CMAKE_PREFIX_PATH or set
  "Torch_DIR" to a directory containing one of the above files.  If "Torch"
  provides a separate development package or SDK, be sure it has been
  installed.
</code></pre>
<p>Passing the <code>CMAKE_MODULE_PATH</code> and <code>CMAKE_PREFIX_PATH</code> values ​​did not work.<br>
How can I pass the install directory?</p>
Passing Python scalars to compiled functions

cbcase: <p>Another simple test case to understand the boundary conditions for compile:</p>
<pre><code class="lang-auto">@torch.compile()
def affine(x, a, b):
    return a * x + b

alst = [3., 4., 5., 6.]
blst = [2., 3., 7., -3.2]
nelems = 1000
x = torch.randn(nelems, dtype=torch.float32, device="cuda")
for a, b in zip(alst, blst)
    affine(x, a, b)
</code></pre>
<p>My observation is that this code will generate 4 compiled kernels, each one specialized on the constant input values for <code>a</code> and <code>b</code>. This is in contrast with a dynamically-sized <code>x</code>, in which case you get the first size-specialized kernel, followed by a variable-sized one used for subsequent invocations.</p>
<p>It’s possible to avoid recomputation for every new a/b by making those into cuda tensors and <code>copy_</code>ing the value in, but that’s pretty kludgey. I’m guessing this is a dynamo issue – is it possible to get it to generate a variable-scalar-input graph for <code>a</code> and <code>b</code>? I understand that it can’t in the case of cuda graph execution, but for eager execution, this is basically the same thing as dynamic shapes.</p>
Pre-trained models in pytorch.models

aavishkar_gautam: <p>Hey everyone. I have a question. Are the pre-trained models available in torch-vision.models trained using the imagenet dataset? If not, how can i get the ones trained on the imagenet dataset ?</p>

ptrblck: <p>Yes, a lot of pretrained <code>torchvision.models</code> were pretrained on ImageNet as seen <a href="https://pytorch.org/vision/stable/models.html">here</a>. The docs also show how to access the pretrained weights which now also include the dataset name, e.g. <code>ResNet50_Weights.IMAGENET1K_V1</code> etc.</p>
Jupyter Notebooks for Beginners in NLP

vdw: <p>I hope this doesn’t violate any community guidelines; it’s just about sharing free resources.</p>
<p>I’m co-teaching an undergrad NLP course at the National University of Singapore. As part of our course materials, we provide our students with a collection of supplementary Jupyter notebooks for a more hands-on experience. By now, these notebooks are polished enough to put them on <a href="https://github.com/chrisvdweth/nus-cs4246x/tree/master" rel="noopener nofollow ugc">Github</a>.</p>
<p>So if you’re new to NLP, these notebooks might be useful. They are organized into 3 parts:</p>
<ul>
<li>
<p><strong>Foundations:</strong> Regular Expressions, Data Preprocessing (tokenization, normalization, stemming/lemmatization), N-gram Language Models, Vector Space Model, Naive Bayes Classification</p>
</li>
<li>
<p><strong>Core Tasks:</strong> POS Tagging, Constituency and Dependency Parsing, Word Semantics, Keyword Extraction, Basic Text Summarization</p>
</li>
<li>
<p><strong>Neural NLP:</strong> Multi-Layer Perceptrons (MLPs), Recurrent Neural Networks (RNNs), Word Embeddings (Word2Vec: CBOW + Skip-gram), Transformers, Applications (sentiment analysis, machine translations, language models)</p>
</li>
</ul>
<p>All examples in Neural NLP are using <strong>PyTorch</strong>. I already posted some individual notebooks in recent posts of mine here. So why not link the whole repository.</p>
<p>Again, these notebooks are for <strong>beginners</strong> and focus on learning and understanding the basics, and not about the state-of-the-art in NLP research.</p>
<p>Please let me know if you have any questions or maybe even suggestions for improvements.</p>

ptrblck: <p>Thanks for sharing these great resources!</p>
Breaking down compute and communication in loss.backward()

pnnaman: <p>I understand that the <strong>loss.backward()</strong> when training using DDP overlaps compute and communication by sharing the gradients in a bucketed way i.e. the all-reduce communication takes place for each bucket of gradients that have been calculated. Is there a way to break down the time taken for these 2 compute (gradient calculation) and communication (syncing gradients using all-reduce) phases?</p>

pnnaman: <p>Any pointers regarding this would be extremely helpful!</p>

ptrblck: <p>You could profile the workload with e.g. Nsight Systems and check the timeline to see the communication and compute kernels. This should allow you to estimate the times of these workloads.</p>
Huge performance decrease when matrix multiplication goes from float32 to float64

cinna: <p>I noticed that matrix multiplication with torch.float32 is more than 50 times faster than with torch.float64.</p>
<p>import torch<br>
torch.set_default_device(‘cuda:0’) # cuda:0 is A6000</p>
<h2><a name="def-timer_torchxy-z-x-y-torchcudasynchronize-return-1" class="anchor" href="#def-timer_torchxy-z-x-y-torchcudasynchronize-return-1"></a>def timer_torch(x,y):<br>
z = x @ y<br>
torch.cuda.synchronize()<br>
return</h2>
<h1><a name="float32-2" class="anchor" href="#float32-2"></a>float32</h1>
<p>x = torch.randn(100,1000000)<br>
y = torch.randn(1000000,100)</p>
<h2><a name="timeit-timer_torchxy-168-ms-451-s-per-loop-mean-std-dev-of-7-runs-1000-loops-each-3" class="anchor" href="#timeit-timer_torchxy-168-ms-451-s-per-loop-mean-std-dev-of-7-runs-1000-loops-each-3"></a>timeit timer_torch(x,y)<br>
1.68 ms ± 4.51 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)</h2>
<h1><a name="float64-4" class="anchor" href="#float64-4"></a>float64</h1>
<p>x = torch.randn(100,1000000).to(torch.float64)<br>
y = torch.randn(1000000,100).to(torch.float64)</p>
<h2><a name="timeit-timer_torchxy-68-ms-325-ns-per-loop-mean-std-dev-of-7-runs-10-loops-each-5" class="anchor" href="#timeit-timer_torchxy-68-ms-325-ns-per-loop-mean-std-dev-of-7-runs-10-loops-each-5"></a>timeit timer_torch(x,y)<br>
68 ms ± 325 ns per loop (mean ± std. dev. of 7 runs, 10 loops each)</h2>
<p>So on my A6000 with torch 2.1 and cuda 12.1, float32 is 40 times faster than float64.<br>
Is this the expected behavior? I want to make sure this is not a result of some mismatch in libraries that would produce such dramatic difference.<br>
If the result is expected, are there strategies to leverage off GPU when using big matrices where a higher 64 bit precision is necessary?</p>

ptrblck: <p>The result seems to be expected based on the spec sheet claiming the A6000 achieves a peak FP32 performance of ~38.7TFLOPS and a peak FP64 performance of ~1.25TFLOPS.</p>

cinna: <p>Thank you very much for the reply - now I don’t have to worry about debugging my configuration looking for library/version inconsistencies. Do you have any suggestion for the second part of my question: If the result is expected, are there strategies to leverage off GPU when using big matrices where the higher 64 bit precision is necessary?</p>

ptrblck: <p>Assuming you really need <code>float64</code> precision you could check some data center devices, such as A100 or H100, which improve the FP64 performance significantly.</p>
How to use grid_sample to do image deformation

jshcht: <p>Hi everyone, I wonder how to use grid_sample to deform an image? The deformation is not necessarily affine transform. I already have the deformed grid but I’m not sure if the deformed grid is the one I should put in grid_sample. For example, in the following case, I expect the strip between y=10~100 shift to the right because the x coordination of the deformed grid move along the positive direction. However, the image shifts to the left.</p>
<pre><code class="lang-auto">import torch
from skimage import data
from torch.nn.functional import grid_sample
import matplotlib.pyplot as plt

# read the image with OpenCV
image = data.camera().astype('float32')
image = image[None]
img = torch.as_tensor(image)
angle = torch.deg2rad(45. * torch.ones(1))
x = torch.linspace(-1, 1, 512)
Y, X = torch.meshgrid(x, x)

X1 = X.clone()
X1[10:100] += 50/512
Y1 = Y.clone()
grid_deform = torch.stack([X1, Y1], dim=-1).unsqueeze(0)
img1 = grid_sample(img[None], grid_deform)[0,0]
plt.imshow(img1)
</code></pre>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/5/954ab73d987b6f21ec355f30c348848d73d33915.jpeg" alt="image" data-base62-sha1="liHdHeCPklUjGIs4aJAuL51vyhT" width="452" height="435"></p>