akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
DataLoader for a LSTM Model with a Sliding Window

skiddles: <p>I am working on a LSTM model and trying to use a DataLoader to provide the data.  I am using stock price data and my dataset consists of:</p>
<ul>
<li>Date (string)</li>
<li>Closing Price (float)</li>
<li>Price Change (float)</li>
</ul>
<p>Right now I am just looking for a good example of LSTM using similar data so I can configure my DataSet and DataLoader correctly.</p>
<p>To test my <code>DataLoader</code> I have the following code:</p>
<pre><code class="lang-auto">    for i, d in enumerate(dataloader):
        print(i, d)
</code></pre>
<p>Using the following definition of <code>dataloader</code>  the test works.</p>
<pre><code class="lang-auto">    dataloader = DataLoader(pricedata,
                            batch_size=30,
                            shuffle=False,
                            num_workers=4)
</code></pre>
<p>This give me <em>x</em> batches  of size 30, which makes sense.</p>
<p>However, I need to use a sliding window of size <em>n</em> so, assuming there are <em>k</em> instances in the dataset, I would like <em>k-n</em> batches with <em>n</em> instance in each batch.</p>
<p>So I redefined <code>dataloader</code> as:</p>
<pre><code class="lang-auto">    dataloader = DataLoader(pricedata,
                            batch_sampler=torch.utils.data.sampler.SequentialSampler(pricedata),
                            shuffle=False,
                            num_workers=4)
</code></pre>
<p>With this change I get the following error message:<br>
<code>TypeError: 'int' object is not iterable</code><br>
when the code hits the test:<br>
<code>for i, d in enumerate(dataloader):</code></p>
<p>Based on this I have two questions:</p>
<ol>
<li>Is <code>torch.utils.data.sampler.SequentialSampler</code> the appropriate sampler to use for a sliding window?</li>
<li>Can anyone point me to a good example of configuring an LSTM using a <code>DataLoader</code> to load numeric data?  All the examples I have seen are for NLP.</li>
</ol>
<p>Thanks</p>

ptrblck: <p>The <code>SequantialSampler</code> samples your data sequentially in the same order.<br>
To use a sliding window, I would create an own <code>Dataset</code> and use the <code>__getitem__</code> method to get the sliding window.<br>
Here is a small example (untested):</p>
<pre><code class="lang-python">class MyDataset(Dataset):
    def __init__(self, data, window):
        self.data = data
        self.window = window

    def __getitem__(self, index):
        x = self.data[index:index+self.window]
        return x

    def __len__(self):
        return len(self.data) - self.window
</code></pre>

duckbill: <p>thank you! that’s awesome!</p>
How to resume with powerSGD-enabled training?

Rui_Wang: <p>Hi, I have found powerSGD as very helpful in my training scenario where the bandwidth has been playing a limiting factor in training speed and powerSGD works very well in reducing the time while maintaining performance.</p>
<p>However, it is my understanding that powerSGD is a stateful hook, and hence if I somehow pause the training and resume later, I need to save the state as well, and could anyone show me how to do that? I’ve been playing around with it and it seems that this would work:</p>
<pre><code class="lang-auto">class MyOwnPowerSGDState(PowerSGDState):
    def state_dict(self):
        return {
            k: getattr(self, k) for k in self.__slots__
        }

    def load_state_dict(self, state_dict: Dict[str, Any]):
        for k in self.__slots__:
            setattr(self, k, state_dict[k])

</code></pre>
<p>However, I have not been able to resume the run since it gives me some shape error for the tensors. Could anyone help me with this? Thanks!</p>

kumpera: <p>I don’t believe that there’s a good solution here.</p>
<p>I managed to get the following to work.</p>
<p>Say you’re doing it in the following way:</p>
<pre><code class="lang-auto">import torch.distributed.algorithms.ddp_comm_hooks as hooks

state = hooks.powerSGD.PowerSGDState(dist.group.WORLD)
register_ddp_comm_hook(DDPCommHookType.POWER_SGD, state)
</code></pre>
<p>The way to add state to your state dict is the following:</p>
<pre><code class="lang-auto">state: PowerSGDState = ...
pg = state.process_group
state.process_group = None

state_dict = {
   'powerSGD_state':  state,
   'model' = ....
}
torch.save(state_dict, "/tmp/my_model")

#NB restore pg in `state` otherwise you won't be able to keep training
state.process_group = pg
</code></pre>
<p>The when loading, you’d do this instead:</p>
<pre><code class="lang-auto">state_dict = torch.load("/tmp/my_model")
state = state_dict['powerSGD_state']
state = dist.group.WORLD
register_ddp_comm_hook(DDPCommHookType.POWER_SGD, state)

</code></pre>
<p>I believe <a class="mention" href="/u/rohan_varma">@Rohan_Varma</a> might have a better idea on how to address this issue.</p>

Rui_Wang: <p>Hi, thanks for this. So you are saying that I have to detach the process groups before saving and reattach  after loading?</p>

kumpera: <p>Yes, you have to do so because it’s part of the PowerSGDState object and it’s not serializable.</p>
<p>This is a shortcoming of the current design. I filed a feature request to explore addressing it: <a href="https://github.com/pytorch/pytorch/issues/75666" class="inline-onebox" rel="noopener nofollow ugc">checkpointing of DDP comms hook · Issue #75666 · pytorch/pytorch · GitHub</a></p>

rvarm1: <p>Thanks Rodrigo! Agreed that we don’t have a great way to fix this at the moment, we’ll track the fix in <a href="https://github.com/pytorch/pytorch/issues/75666" class="inline-onebox" rel="noopener nofollow ugc">checkpointing of DDP comms hook · Issue #75666 · pytorch/pytorch · GitHub</a></p>

Rui_Wang: <p>Great! Thanks for the help!</p>
Hardware configuration for running YoloV5 model

Sreekanth: <p>Hi,</p>
<p>We are using YoloV5x model for object detection, we would like to know the hardware configuration for running the model in a local machine.</p>
<p>Thanks,<br>
Sreekanth</p>

Sreekanth: <p><a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>Do you have any idea on the hardware configuration required for running the Yolov5x model.</p>

ptrblck: <p>No, I don’t have any configurations ready, but the <a href="https://github.com/ultralytics/yolov5">repository</a> mentions the training was performed on a V100 16GB, so you might use this device as a reference.</p>

Sreekanth: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>Thanks. We are not planning to train the model in local machine. When we run the model in local machine (OS: Ubuntu) the prediction time is more with dedicated GPU.</p>
<p>Wanted to know which GPU will suit to run the model in local machine.</p>
Difference between GPU memory usage pattern in training/inference

hyhuang00: <p>During a typical training step for a neural network model, it seems like we need to keep all the <em>activation</em> we have so that we can easily calculate the gradient when doing the backward step. However, for inference (say, under <code>torch.no_grad()</code> or <code>torch.inference_mode()</code>), does this statement holds true? It seems like at any moment during a forward step, we only need to keep track of the activations that are relevant for future layers, and memory that are reserved for previous activation can be freed or reused.</p>
Torch.scatter_() with scalar src works differently parameter name specified & not specified

jh_shim: <p>It seams that <code>torch.scatter_()</code> with scalar src parameter name specified works differently from that the name is not specified.</p>
<p>For example:</p>
<pre><code class="lang-auto">x = torch.ones(2, 10)
idx = torch.tensor([[3, 4], [5, 6]]).long()
</code></pre>
<pre><code class="lang-auto"># 1. when parameter name `src` is not specified
x.scatter_(-1, idx, 0)
"""
x is:
tensor([[1., 1., 1., 0., 0., 1., 1., 1., 1., 1.],
        [1., 1., 1., 1., 1., 0., 0., 1., 1., 1.]])
"""
</code></pre>
<pre><code class="lang-auto"># 2. when parameter name `src` is specified
x.scatter_(-1, idx, src=0)
"""
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Input In [35], in &lt;cell line: 1&gt;()
----&gt; 1 x.scatter_(-1, idx, src=0)

TypeError: scatter_() received an invalid combination of arguments - got (int, Tensor, src=int), but expected one of:
 * (int dim, Tensor index, Tensor src)
      didn't match because some of the arguments have invalid types: (int, Tensor, !src=int!)
 * (int dim, Tensor index, Tensor src, *, str reduce)
 * (int dim, Tensor index, Number value)
      didn't match because some of the keywords were incorrect: src
 * (int dim, Tensor index, Number value, *, str reduce)
"""
</code></pre>
<pre><code class="lang-auto"># 3. test case
x.scatter_(-1, idx, value=0)
"""
x is:
tensor([[1., 1., 1., 0., 0., 1., 1., 1., 1., 1.],
        [1., 1., 1., 1., 1., 0., 0., 1., 1., 1.]])
"""
</code></pre>
<p>Yes, it’s not a bug and it’s because the parameter <code>src</code> and <code>value</code> are separated in the valid combination of arguments. But even though it’s not a bug, it could confuse users and I cannot deduce why these two parameter names should be separated. If it is internally necessary one, then I think the <a href="https://pytorch.org/docs/stable/generated/torch.Tensor.scatter_.html" rel="noopener nofollow ugc">documentation</a> should mention about <code>value</code> parameter.</p>
<p>Thank you.</p>

ptrblck: <p>Thanks for raising the potentially confusing docs! A contribution to improve the documentation is more than welcome, so would you be interested in improving it? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
I want to make 2D into 3D

YuhskeHujisaki: <p>I am processing signals, but the signals are arranged in an array of 432 horizontal and 8750 vertical.  I’m trying to classify this with cnn.  Therefore, I would like to make this array into data such as image classification with 432 horizontal and 50 batch vertical 175.  How do I convert an array?<br>
The tensor type is now [128 (batch), 1 (vertical because we bring one data at a time), 423 (horizontal)], but I want to change this to [175,1 (input), 50,423].</p>

Andrei_Cristea: <p>Hello Yuhske, is this what you’re looking for:</p>
<pre><code class="lang-auto">x = torch.ones(432, 8750)
x_newshape = x.view(175, 1, 50, 432)
print(x_newshape.shape)

Output: 
torch.Size([175, 1, 50, 432])
</code></pre>

YuhskeHujisaki: <p>Thank you!!! I’m a beginner, so I’m grateful that I want to learn pytorch while asking questions.</p>
Training Model SSD300_VGG16 In Torchvision.Models

Duc_Nguyen: <p>Hi,<br>
I want to use the model SSD300_VGG16 from Torchvision.models.detection.ssd300_vgg16 with my custom dataset.<br>
But when I trained this model, the loss didn’t decrease. And I checked inside the model.<br>
I found out there are problems here.</p>
<p>Follow this code, torch.arange() is a problem because I have too many boxes with 0, and 1 box with another index. So location loss is not right.</p>
<p>Please check and help me. Thank you.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/8/680f72b03e75742414a1e3ca533411159621169f.png" data-download-href="https://discuss.pytorch.org/uploads/default/680f72b03e75742414a1e3ca533411159621169f" title="Capture"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/8/680f72b03e75742414a1e3ca533411159621169f_2_690x310.png" alt="Capture" data-base62-sha1="eQyKKPoK276MxgHS1IHuprqQSAL" width="690" height="310" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/8/680f72b03e75742414a1e3ca533411159621169f_2_690x310.png, https://discuss.pytorch.org/uploads/default/optimized/3X/6/8/680f72b03e75742414a1e3ca533411159621169f_2_1035x465.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/6/8/680f72b03e75742414a1e3ca533411159621169f_2_1380x620.png 2x" data-dominant-color="242525"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Capture</span><span class="informations">1918×864 97.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Why does pytorch prompt: RuntimeError: Legacy autograd function with non-static forward method is deprecated. Please use new-style autograd function with static forward method

johansmithy: <p>Hi I have the following depreciated torch.autograd.Function code</p>
<pre><code class="lang-auto">class AdversarialLayer(torch.autograd.Function):
  def __init__(self, high_value=1.0):
    self.iter_num = 0
    self.alpha = 10
    self.low = 0.0
    self.high = high_value
    self.max_iter = 10000.0
    
  def forward(self, input):
    self.iter_num += 1
    output = input * 1.0
    return output

  def backward(self, gradOutput):
    self.coeff = np.float(2.0 * (self.high - self.low) / (1.0 + np.exp(-self.alpha*self.iter_num / self.max_iter)) - (self.high - self.low) + self.low)
    return -self.coeff * gradOutput
</code></pre>
<p>I am not sure how to update this so that I no longer get a Runtime Error. The example on the pytorch <a href="https://pytorch.org/tutorials/beginner/examples_autograd/two_layer_net_custom_function.html" rel="noopener nofollow ugc">website</a> does not address code relying on using an init function and self.</p>

johansmithy: <aside class="quote no-group" data-username="johansmithy" data-post="1" data-topic="149079">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/3e96dc/48.png" class="avatar"> johansmithy:</div>
<blockquote>
<p><code>torch.autograd.Function</code></p>
</blockquote>
</aside>
<p>Found a solution: <a href="https://discuss.pytorch.org/t/custom-autograd-function-must-it-be-static/14980/3" class="inline-onebox">Custom autograd.Function: must it be static? - #3 by Megh_Bhalerao</a></p>
Debugging Strategies

mikeb127: <p>Hi Everyone,</p>
<p>I have recently made my first contribution to pytorch(small bug fix) and am looking to do some more of these in the future.</p>
<p>I was hoping that some of the more experienced pytorch developers might be able to share their debugging techniques/strategies for the C++/CUDA side of things so I can learn and make my life a little easier going forward?</p>
<p>Thanks in advance</p>
<p>Michael</p>
The Hutchinson’s estimator (Trace of Fisher Information Matrix)

BartekK: <p>Hi,<br>
sorry for inconvenience, this is my first post.</p>
<p>I am trying to implement an FIM Trace estimator from the paper <a href="https://arxiv.org/pdf/2012.14193.pdf" rel="noopener nofollow ugc">https://arxiv.org/pdf/2012.14193.pdf</a></p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/3/9303d917056bb4d104d3e1857239f10c4b8da199.png" alt="hutchinson" data-base62-sha1="kYyqOqGMhO9nqSSLWObksOV05lv" width="499" height="255"></p>
<p>Such a function came out of an attempted implementation.</p>
<pre><code class="lang-auto">def fim_trace(loss, model, m):
    fim = {}
    for name, module in model.named_modules():
        if isinstance(module, torch.nn.Conv2d) or isinstance(module, torch.nn.Linear):
            if module.weight.requires_grad:
                fim[name] = 0.
                dloss_w = torch.autograd.grad(loss, module.weight, create_graph=True)[0]
                for _ in range(m):
                    dloss_w = dloss_w.flatten()
                    z = torch.rand_like(dloss_w)
                    loss2_w = dloss_w.T @ z
                    dloss2_w = torch.autograd.grad(loss2_w, module.weight, retain_graph=True)[0]
                    loss3_2 = z.T @ dloss2_w.flatten()
                    fim[name] += loss3_2.item()
                fim[name] /= m
                
    fim_trace = 0.
    for name in fim:
        fim_trace += fim[name]
    return fim, fim_trace
</code></pre>
<p>Is there anyone here who can determine if it is well implemented, and want to help me? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>Greetings</p>
PyTorch profiler and Nvidia GPU memory consumption

Margo: <p>Hello !</p>
<ul>
<li>
<p>Can Pytorch Profiler profile GPU memory consumption during inference ?<br>
<a href="https://pytorch.org/tutorials/recipes/recipes/profiler_recipe.html" class="inline-onebox" rel="noopener nofollow ugc">PyTorch Profiler — PyTorch Tutorials 1.11.0+cu102 documentation</a></p>
</li>
<li>
<p>Does ProfilerActivity.CUDA only measures the GPU time during a processus ?</p>
</li>
<li>
<p>Is it possible to profile (GPU time + GPU memory consumption) a function ? Like with a decorator ?</p>
</li>
<li>
<p>What is PyTorch autograd profiler ? What are the differences compared to <a href="https://github.com/pytorch/pytorch/blob/master/torch/profiler/profiler.py" rel="noopener nofollow ugc">torch.profile.profiler</a> ?</p>
</li>
</ul>
<p>Thanks in advance</p>
Timing in Torchscript

knitvoger: <p>Below code is not working in torch.jit.script</p>
<pre><code class="lang-auto">    def forward(self, x):
        start = time.time() * 1000
        y = self.compute_conv1d((x))
        print("cost %d ms" % (time.time() * 1000 - start))
        return y
</code></pre>
<p>error</p>
<pre><code class="lang-auto">RuntimeError: 
Python builtin &lt;built-in function time&gt; is currently not supported in Torchscript:
</code></pre>
<p>Since time.time() is not supported in Torchscript, is there any way to measure each module’s cost inside script model? I’m running on a cpu machine.</p>
<p>Thanks</p>

googlebot: <p>something like</p>
<pre><code class="lang-auto">@jit.ignore
def jtime() -&gt; float:
  return time.time()
</code></pre>
<p>autograd.profiler.profile + export_chrome_trace is another method</p>

knitvoger: <p><a class="mention" href="/u/googlebot">@googlebot</a> thank you! It works!</p>

i03fesea: <p>I am getting the following exception when I call the function suggested by knitvoger’s code:</p>
<pre><code class="lang-auto">@jit.ignore
def jtime() -&gt; float:
  return time.time()
</code></pre>
<p>ValueError: Unknown type annotation: ‘float’ at</p>
Run inference with model trained using nn.DataParallel

jpcenteno: <p>I am working on lung segmentations using a Fully Convolutional DenseNet that has both batchnorm and dropout layers. The model was trained using <code>nn.DataParallel</code> on two GPUs. When I run inference, I load the weights after first wrapping the model in <code>nn.DataParallel</code>. Inference results without the flag <code>model.eval()</code> look as expected. However, when I run inference using <code>model.eval()</code>, the segmentation fails and I get random clusters of pixels inside the lungs. I have checked the <code>.training</code> properties for both the <code>nn.BatchNorm2d</code> and <code>nn.Dropout2d</code> layers in my parallel model using <code>model.train()</code> and <code>model.eval()</code>, and the properties look correct (True for <code>model.train()</code> and False for <code>model.eval()</code>):</p>
<pre><code class="lang-auto">model = nn.DataParallel(FCDN()).cuda()
model.eval()
for m in model.modules():
    if isinstance(m, nn.BatchNorm2d):
        print('bn', m.training) # bn False
    if isinstance(m, nn.Dropout2d):
        print('do', m.training)  # do False
</code></pre>
<p>Seems the problem could be related to this: <a href="https://github.com/pytorch/pytorch/issues/1051" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/1051</a>, but I’m not sure I fully understand the post.</p>
<p>Since segmentations look acceptable without using <code>model.eval()</code>, it is not a big issue for me, but I wonder if the results could be better if I knew how to use <code>model.eval()</code> correctly.</p>

hristo-vrigazov: <p>I stumbled upon the exact same problem - there is a dramatic decrease of performance when using <code>.eval()</code> on <code>DataParallel</code>. I do not understand why though.</p>

ptrblck: <p><a class="mention" href="/u/jpcenteno">@jpcenteno</a>, <a class="mention" href="/u/hristo-vrigazov">@hristo-vrigazov</a><br>
Did you observe the same accuracy drop on your validation set during training or using a single GPU?</p>

BenPark: <p>I am also stuck in this problem… Anybody solve the problem??</p>
<p>When i do not call model.eval() and assign the identical batch number as what used in training, then the estimation seems reasonable.</p>

SofiaCP: <p>Have you solved this issue?</p>
How to write a sequence-to-trees LSTMs in pytorch?

Jay_Gala: <p>I am currently working on deep semantic parsing for conversational systems. I am planning to use <a href="http://fb.me/semanticparsingdialog" rel="noopener nofollow ugc">TOP dataset</a> for semantic parsing. I was wondering if there are any open-sourced implementations of tree LSTMs with both top-down and bottom-up approaches. How to support batching with tree-based decoding (top-down)?</p>
<p>Thank you!</p>
PyTorch and ROCm 5: What ROCm packages are required for PyTorch?

humanplayer2: <p>I’m hoping to use PyTorch with ROCm to speed up some SVD using an AMD GPU. I’m new to GPU computing, ROCm and PyTorch, and feel a bit lost.</p>
<p>I’m pretty sure I need ROCm &gt;= 5.0 to support the 6800 RX GPU, which means the PyTorch <em>Get Started Locally</em> command doesn’t quite work for me. Further, I’d like to test on a laptop with a Vega 8 iGPU which some ROCm packages do not support (HID, I believe).</p>
<p>Hence, I need to install ROCm differently, and due to my OS, I can’t use the AMD script (PopOS 21.10), but go through the <a href="https://docs.amd.com/bundle/ROCm-Installation-Guide-v5.1/page/How_to_Install_ROCm.html" rel="noopener nofollow ugc">Using Package Manager on Ubuntu</a> method, which is all fine.</p>
<p>In the very last step, I’m asked to install ROCm meta-packages—and here I’m unsure what I need for PyTorch. AMD has a <a href="https://docs.amd.com/bundle/ROCm-Installation-Guide-v5.1/page/Meta-packages_in_ROCm_Programming_Models.html" rel="noopener nofollow ugc">neat list of ROCm meta-packages</a>, but I don’t know which are necessary.</p>
<p>I’ve been trying to prod the URL from the PyTorch Get Started Locally command, but I didn’t get anything out of that.</p>
<h3>
<a name="so-in-short-which-of-the-rocm-meta-packages-should-i-install-to-make-use-of-pytorch-1" class="anchor" href="#so-in-short-which-of-the-rocm-meta-packages-should-i-install-to-make-use-of-pytorch-1"></a><strong>So, in short: Which of the ROCm meta-packages should I install to make use of PyTorch?</strong>
</h3>
<p>Thank you for your time.</p>

nothingbutausername: <p>As far as I am aware (which I admit isn’t a lot) we don’t yet have Torch packages for ROCm 5.x and RDNA2 cards are only supported with ROCm 5.0 and above.</p>
<p>Although standalone Torch packages only support 4.5.2 at the moment there is a docker package from AMD which supports ROCm 5 but I’ve not experimented at all with this: <a href="https://www.amd.com/en/technologies/infinity-hub/pytorch" rel="noopener nofollow ugc">https://www.amd.com/en/technologies/infinity-hub/pytorch</a></p>

humanplayer2: <p>Ah. I didn’t realize there were specific PyTorch packages for the different ROCm versions. I thought that if I had the necessary ROCm packages installed, PyTorch would “just work” with them. Thank you for pointing this out!</p>
<p>Then I will sit curb my enthusiasm and wait until the “Start Locally” page is updated for ROCm 5.0+</p>
Passing padded tensors as input to non-recurrent models

justaquestion: <p>Hi,</p>
<p>I am using the output from a transformer encoder as input to an MLP network. The input to the encoder is padded with zeroes so all sequences are the same length, and I pass src_key_padding_mask=mask to the encoder. The MLP operates on the output from each time-step from the encoder, i.e. taking as input sequence_length * batch size vectors with shape encoder_dim:</p>
<p>Padded Batch: [ seq_length, batch_size, dim ] → <br>
Encoder output: [ seq_length, batch_size, encoder_dim ] → <br>
MLP output: [ seq_length, batch_size, output_dim ]</p>
<p>Since the encoder output includes padded values, the MLP is producing output from those values, which would affect the loss. What is the best way to avoid this? I have been zeroing out the predicted values with a binary mask, and calculating the loss as follows:</p>
<pre><code class="lang-auto">def sum_mask_loss(mask, output, labels, loss_func):

    newmask = torch.unsqueeze((~mask).int(), 2).repeat(1, 1, output.shape[2])
    output = newmask * output
    loss = loss_func(output, labels) / newmask.sum()

    return loss
</code></pre>
<p>But I am not sure this is the best way. Any advice would be appreciated!</p>
Modify intermediate layer

mshooter: <p>Hi,<br>
I wondered how would you modify the out_channels of an intermediate layer of a pre-trained ResNet?<br>
I am not talking about the fully connected layer, but for example a conv2 layer?</p>
<p>Thank you!</p>

mxahan: <aside class="quote no-group" data-username="mshooter" data-post="1" data-topic="148933">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mshooter/48/32208_2.png" class="avatar"> mshooter:</div>
<blockquote>
<p>I wondered how would you modify the out_channels of an intermediate layer of a pre-trained ResNet?<br>
I am not talking about the fully connected layer, but for example a conv2 layer</p>
</blockquote>
</aside>
<p>Does <a href="https://discuss.pytorch.org/t/how-can-i-replace-an-intermediate-layer-in-a-pre-trained-network/3586/8">it</a> work?</p>

tom: <p>So there are several aspects to it:</p>
<ul>
<li>The for the forward to go through, the layer following the layer for which you change out_channels would need to be adapted, too, to expect the new number of channels.</li>
<li>The typical thing to change the last layer is to replace it and train the weights of the new one from scratch. The more or less direct analogue would be to train the changed layer and everything following it on your task.</li>
<li>In principle, it also should be possible to try to only the two layers that need to be modified and train them to produce the same outputs as the two layers they are replacing. This is could be considered to be in the family of techniques known as “model distillation”.</li>
</ul>
<p>Best regards</p>
<p>Thomas</p>

mshooter: <aside class="quote no-group" data-username="tom" data-post="3" data-topic="148933">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tom/48/3162_2.png" class="avatar"> tom:</div>
<blockquote>
<p>The typical thing to change the last layer is to replace it and train the weights of the new one from scratch. The more or less direct analogue would be to train the changed layer and everything following it on your task.</p>
</blockquote>
</aside>
<p>The second advice did it for me, thank you!</p>
Getting this Error while running the code in PyTorch ver 1.8.1 and CUDA 10.1

Sajid_Ahmed: <p>(base) sajid@sajid-pc:~/Desktop/dctts-pytorch$ python3 main.py --action train --module Text2Mel<br>
train: Text to Mel</p>
<p>loop 0<br>
▝[1/60] |gs: 0, mels: 0.384276, bd1: 0.694295, atten: 0.783801, scale: 99.999000| [0.0% eta ETA unknown]Traceback (most recent call last):<br>
File “/home/sajid/Desktop/dctts-pytorch/main.py”, line 62, in <br>
main()<br>
File “/home/sajid/Desktop/dctts-pytorch/main.py”, line 47, in main<br>
train(args.module, args.load)<br>
File “/home/sajid/Desktop/dctts-pytorch/pkg/train.py”, line 54, in train<br>
train_text2mel(load_trained)<br>
File “/home/sajid/Desktop/dctts-pytorch/pkg/train.py”, line 198, in train_text2mel<br>
plot_spectrum(mels[0].cpu().data, “mel_true”, gs, dir=logdir)<br>
File “/home/sajid/Desktop/dctts-pytorch/pkg/utils.py”, line 110, in plot_spectrum<br>
im = ax.imshow(np.flip(spectrum, 0), cmap=“jet”, aspect=0.2 * spectrum.shape[1] / spectrum.shape[0])<br>
File “&lt;<strong>array_function</strong> internals&gt;”, line 5, in flip<br>
File “/home/sajid/anaconda3/lib/python3.9/site-packages/numpy/lib/function_base.py”, line 241, in flip<br>
return m[indexer]<br>
ValueError: step must be greater than zero</p>

ptrblck: <p>The issue seems to be raised by <code>numpy</code> in:</p>
<pre><code class="lang-python">im = ax.imshow(np.flip(spectrum, 0), cmap=“jet”, aspect=0.2 * spectrum.shape[1] / spectrum.shape[0])
&gt; ValueError: step must be greater than zero
</code></pre>
<p>so maybe you are passing a non-contiguous tensor to this method.<br>
Call <code>spectrum.numpy()</code> or <code>spectrum.contiguou().numpy()</code> before using it in <code>flip</code>.</p>

Sajid_Ahmed: <p>Thanks a lot, it worked and fixed the issue, however, I have found another issue.</p>
<pre><code class="lang-auto">(dctts-pytorch) sajid@sajid-pc:~/Desktop/dctts-pytorch$ python3 main.py --action train --module Text2Mel
train: Text to Mel
loop 0
▝[16/60] |gs: 15, mels: 0.257367, bd1: 0.689199, atten: 0.787459, scale: 99.984001| [26.7% eta 00:01:22]Traceback (most recent call last):
  File "/home/sajid/Desktop/dctts-pytorch/main.py", line 62, in &lt;module&gt;
    main()
  File "/home/sajid/Desktop/dctts-pytorch/main.py", line 47, in main
    train(args.module, args.load)
  File "/home/sajid/Desktop/dctts-pytorch/pkg/train.py", line 54, in train
    train_text2mel(load_trained)
  File "/home/sajid/Desktop/dctts-pytorch/pkg/train.py", line 168, in train_text2mel
    atten_guide * graph.attention * atten_mask,
RuntimeError: The size of tensor a (300) must match the size of tensor b (328) at non-singleton dimension 2

</code></pre>
Titanic - Neural Network - Two hidden layers – 5 units ReLU and 3 units tanh

dipakpatadia: <p>Want to build a model neural network model using <strong>PyTorch</strong> library.</p>
<p>The model should use two hidden layers: the first hidden layer must contain <strong>5 units using the ReLU</strong> activation function; the second layer must contain <strong>3 units using tanh</strong> activation function.</p>
<p>Want to use the Titanic train dataset I have. Want to use ONLY the train dataset (.CVV) and split the dataset (i.e., train.csv) to a training set (80% samples) and a testing set<br>
(20% samples)</p>
<p>Don’t know how to define model definition for 2 hidden layers which are ReLU and tanh with 5 and 3 units (nucleus) respectively.</p>
<p>A sample code would really help. Step by step code including standard methods implemented (data cleansing, normalizing, Model training, testing, etc.) would really help me to learn this.</p>
<p>I am using a Jupyter notebook.</p>
<p>Regards,<br>
Dipak</p>

ptrblck: <p>The <a href="https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html">CIFAR10 tutorial</a> might be a good starter as it walks you through the data loading pipeline, defining a custom model, and training it.</p>

dipakpatadia: <p>Thank you so much for referring good start.</p>

dipakpatadia: <p>I am getting the below error. How this can be resolved?</p>
<h1>
<a name="lets-split-the-train-dataset-into-training-set-80-and-testing-set-20-1" class="anchor" href="#lets-split-the-train-dataset-into-training-set-80-and-testing-set-20-1"></a>Lets Split the Train Dataset into training set (80%) and Testing set (20%)</h1>
<p>X = train.drop(‘Survived’, axis=1).values<br>
y = train[‘Survived’].values</p>
<p>X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)</p>
<p>X_train = torch.FloatTensor(X_train)<br>
X_test = torch.FloatTensor(X_test)<br>
y_train = torch.LongTensor(y_train)<br>
y_test = torch.LongTensor(y_test)</p>
<p>TypeError                                 Traceback (most recent call last)<br>
C:\Users\JAYPAT~1\AppData\Local\Temp/ipykernel_5072/502746771.py in <br>
6 X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)<br>
7<br>
----&gt; 8 X_train = torch.FloatTensor(X_train)<br>
9 X_test = torch.FloatTensor(X_test)<br>
10 y_train = torch.LongTensor(y_train)</p>
<p>TypeError: can’t convert np.ndarray of type numpy.object_. The only supported types are: float64, float32, float16, complex64, complex128, int64, int32, int16, int8, uint8, and bool.</p>
Cannot trace all op of Alexnet when using FX

Weizhe_Zhu: <p>I used “torch.fx.symbolic_trace” to trace the pretrained Alexnet, and then I got a graph of Alexnet. But when I print the node of graph, I only got “feature_0. feature_1…”. But actually I want “conv, relu, maxpooling…”</p>
<p>Here is my code:</p>
<pre><code class="lang-auto">traced = symbolic_trace(module)
graph = traced.graph
for node in graph.nodes:
    print(node.name)

</code></pre>
<p>Ang my results:</p>
<pre><code class="lang-auto">x
features_0
features_1
features_2
features_3
features_4
features_5
features_6
features_7
features_8
features_9
features_10
features_11
features_12
avgpool
flatten_1
classifier_0
classifier_1
classifier_2
classifier_3
classifier_4
classifier_5
classifier_6
output
</code></pre>
<p>I want to print op name of each node like:</p>
<pre><code class="lang-auto">Conv
Relu
MaxPool
Conv
Relu
MaxPool
Conv
Relu
Conv
Relu
Conv
Relu
MaxPool
AveragePool
Flatten
Dropout
Linear
Relu
Dropout
Linear
Relu
Linear
</code></pre>
<p>I found the when the model based on nn.Sequential and it cannot trace the op name. Could anyone help me with this problem?</p>

jinfagang: <p>I can trace resnet50 via fx, so alexnet should not have problem.</p>
Help me to Implement BYOL

como: <p>Hi, Im a master’s student in South Korea.</p>
<p>Im implementing BYOL in PyTorch-lightning style and testing using CIFAR10 dataset and ResNet-18.</p>
<p>Im using the official SimSiam and MoCo repository as baseline.</p>
<p>My BYOL’s performance of linear classification is low (maximum 69.3), but I cannot find any reason why the performance is low.</p>
<p>Can anyone help me?</p>
<p>I have attached my GitHub repository and my some experiment below.</p>
<p>Repository: <a href="https://github.com/comojin1994/byol" class="inline-onebox" rel="noopener nofollow ugc">GitHub - comojin1994/byol: Bootstrap Your Own Latent A New Approach to Self-Supervised Learning</a></p>
<p>Experiment</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/4/c4cdd76e3b9653dd920b02208c12abfd32d54995.png" alt="Screen Shot 2022-04-13 at 4.03.22 PM" data-base62-sha1="s50B3l4nPYzv4FHcNIM2bv57gqh" width="420" height="391"></p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/8/7/87f2f35a03e5ffcc9b2af3d2a945b128bdc3d112.png" alt="Screen Shot 2022-04-13 at 4.03.46 PM" data-base62-sha1="joEYCu91kMTewk7CVSvzAEKbu6u" width="421" height="381"></p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/1/b18ad29fdf504bd45c7528bfd941c24072b004b7.png" alt="Screen Shot 2022-04-13 at 4.03.58 PM" data-base62-sha1="pkBWc96IirkdZdivNIZWUmQF167" width="420" height="442"></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/f/bf27774f84ad777fab03b83d33c9b5450e9eaed5.png" data-download-href="https://discuss.pytorch.org/uploads/default/bf27774f84ad777fab03b83d33c9b5450e9eaed5" title="Screen Shot 2022-04-13 at 4.04.12 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/f/bf27774f84ad777fab03b83d33c9b5450e9eaed5_2_498x500.png" alt="Screen Shot 2022-04-13 at 4.04.12 PM" data-base62-sha1="rh1KEAbxRRamkjYAfj0yV5HinYN" width="498" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/f/bf27774f84ad777fab03b83d33c9b5450e9eaed5_2_498x500.png, https://discuss.pytorch.org/uploads/default/original/3X/b/f/bf27774f84ad777fab03b83d33c9b5450e9eaed5.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/b/f/bf27774f84ad777fab03b83d33c9b5450e9eaed5.png 2x" data-dominant-color="B6ABAA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2022-04-13 at 4.04.12 PM</span><span class="informations">560×562 304 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>singular value<br>
          <a href="https://user-images.githubusercontent.com/46745325/163086193-6838c74a-782f-40a4-b48f-be44540c2ac8.png" target="_blank" rel="noopener nofollow ugc" class="onebox">
            <img src="https://user-images.githubusercontent.com/46745325/163086193-6838c74a-782f-40a4-b48f-be44540c2ac8.png" width="607" height="303">
          </a>
</p>
Memory issues in computer vision

gummz00: <p>Hello,</p>
<p>I’m running a training loop consisting of around 1700 images of 512x512 resolution. The batch size is 8. I have 15 GB of GPU memory, and yet, I cannot use a higher batch size than 8 since I will get an out of memory error.</p>
<p>Is there something I’m doing wrong in the code or do those numbers look right?</p>

ptrblck: <p>The images will take a small portion of the device memory and the majority will be used by the actual model training, i.e. parameters, forward activations, gradients, etc.<br>
The general memory requirement thus depends on the actual model as well as the input shape.</p>
Runtime Error: Expected 2 to 3 dimensions, but got 4-dimensional tensor for argument # 1 I get the error'self'

YuhskeHujisaki: <p>When building CNN with pytorch<br>
RuntimeError: Expected 2 to 3 dimensions, but got 4-dimensional tensor for argument # 1’self’ (while checking arguments for max_pool1d)<br>
What does the error mean?<br>
class Net (nn.Module):</p>
<p>def <strong>init __ (self):<br>
super (Net, self) .</strong> init__ ()<br>
self.conv1 = nn.Conv1d (1, 60, kernel_size = (1, 15), stride = (1, 3))<br>
self.conv2 = nn.Conv1d (60, 60, kernel_size = (1, 4), stride = (1, 2))<br>
self.conv3 = nn.Conv1d (60, 60, kernel_size = (1, 17), stride = (1,3))<br>
self.conv4 = nn.Conv1d (60, 90, kernel_size = (1, 3), stride = (1, 1))<br>
self.conv5 = nn.Conv1d (90, 120, kernel_size = (1, 1), stride = (1, 1))<br>
self.pool = nn.MaxPool1d ((1,2), stride = (1,2))<br>
self.soft = nn.Softmax ()<br>
self.li = nn.Linear (11520,95)</p>
<p>def forward (self, x):<br>
x = self.conv1 (x)<br>
x = self.pool (x)<br>
x = self.conv2 (x)<br>
x = self.pool (x)<br>
x = self.conv3 (x)<br>
x = self.conv4 (x)<br>
x = self.pool (x)<br>
x = self.conv5 (x)<br>
x = self.li (x)<br>
x = self.soft (x)<br>
return x<br>
The cnn class looks like this:</p>

ptrblck: <p><code>nn.MaxPool1d</code> expect a 3-dimensional tensor in the shape <code>[batch_size, channels, seq_len]</code> while you are apparently passing a 4-dimensional tensor to this layer.</p>
Pytorch profiler `add_metadata` api

raejaf: <p>Greetings,</p>
<p>I want to add some extra information when using the PyTorch profiler, and I found the <code>add_metadata_json</code> <a href="https://pytorch.org/docs/master/profiler.html#torch.profiler._KinetoProfile.add_metadata_json" rel="noopener nofollow ugc">API</a> in the official documentation of pytorch. But the doc did not explain how this function works and whether it’s possible to draw some self-defined charts on the TensorBoard. Could anyone give me some suggestions about how to use this API?</p>

ptrblck: <p>Based on <a href="https://github.com/pytorch/pytorch/blob/c0a6add7eef8ab3715c0f4281b3b7a2a0f1e24b5/test/test_profiler.py#L754-L768">this small unit test</a> it seems that this metadata will be directly stored in the trace and you should be able to read it afterwards.</p>

raejaf: <p>Hi, ptrblck! Thank you for your reply! I am developing a new tensorboard plugin for our <a href="https://github.com/hpcaitech/ColossalAI" rel="noopener nofollow ugc">project</a>, and I really want to get it compatible with PyTorch Profiler. From your reply, I guess I can append some new JSON snippets and parse them in my plugin?</p>

ptrblck: <p>Yes, I think this is the case based on the posted unit test, but let us know if this approach is not working.</p>
Problem about nn.Linear(16 * 5 * 5, 120)

hhh: <p>Hello,everyone,<br>
I am new to PyTorch, and I feel confused about the reason of 16×5×5 in the codes below. Can anyone tell me why  we choose 16×5×5  here? Thank you:)</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.nn.functional as F

class Net(nn.Module):

    def __init__(self):
        super(Net, self).__init__()
        # 1 input image channel, 6 output channels, 5x5 square convolution
        # kernel
        self.conv1 = nn.Conv2d(1, 6, 5)
        self.conv2 = nn.Conv2d(6, 16, 5)
        # an affine operation: y = Wx + b
        **self.fc1 = nn.Linear(16 * 5 * 5, 120)**
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        # Max pooling over a (2, 2) window
        x = F.max_pool2d(F.relu(self.conv1(x)), (2, 2))
        # If the size is a square you can only specify a single number
        x = F.max_pool2d(F.relu(self.conv2(x)), 2)
        x = x.view(-1, self.num_flat_features(x))
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

    def num_flat_features(self, x):
        size = x.size()[1:]  # all dimensions except the batch dimension
        num_features = 1
        for s in size:
            num_features *= s
        return num_features


net = Net()
input = torch.randn(1, 1, 32, 32)
out = net(input)
print(out)
</code></pre>

ptrblck: <p>The number of input features to your linear layer is defined by the dimensions of your activation coming from the previous layer.<br>
In your case the activation would have the shape <code>[batch_size, channels=16, height=5, width=5]</code>. In order to pass this activation to <code>nn.Linear</code>, you are flattening this tensor to <code>[batch_size, 16*5*5]</code>.<br>
The <code>16</code> is defined by the number of <code>out_channels</code> (i.e. number of filter kernels in the previous conv layer), while <code>5x5</code> is the spatial size defined by the conv and pooling operations performed on your input data.</p>
<p>Let me know, if you need more information!</p>

hhh: <p>Thank you! Is the meaning of 5×5 the same as 28×28 in the MINST example? If so, why 5×5 here?</p>

hhh: <p>Got it! Because the input is 1×32×32,  after the conv and pooling operations, the size becomes 16×5×5. Thank you very much for your help!</p>

jhoanmartinez: <p>Is the out_channels * kernel_size[width] * kernel_size[hight]  from the previous layer right?</p>

ptrblck: <p>Yes, the <code>out_channels</code> are defined by the previous layer. No, the kernel size of the previous layer won’t be used directly, but the spatial size of the output activation of the previous layer.</p>
Faster RCNN model makes only 100 predictions on every image. How to make the model predict dorrectly when the image has more than 100 objects?

deiv_dh: <p>I use Faster RCNN Predictor</p>
<p>My build model function</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/6/969b4f3f061cf4ca3ea46456a13900d2718c9893.png" data-download-href="https://discuss.pytorch.org/uploads/default/969b4f3f061cf4ca3ea46456a13900d2718c9893" title="1"><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/6/969b4f3f061cf4ca3ea46456a13900d2718c9893.png" alt="1" data-base62-sha1="lukmVl1lmuHE3sFzueOSLDoYa3N" width="690" height="123" data-dominant-color="F0EFEE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">1</span><span class="informations">718×128 14.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>predictions = model(imgs)</p>
<blockquote>
<blockquote>
<blockquote>
<p>predictions[0][“boxes”].<strong>len</strong>()<br>
100</p>
</blockquote>
</blockquote>
</blockquote>

ptrblck: <p>It seems you might want to change the <code>box_detections_per_img</code> argument in <code>FasterRCNN</code>.</p>
Torch geometric - modify the graphs nodes on the fly

Thibault712: <p>Hi I was wondering if it was possible for my data augmentation to be done on the fly during my epoch, this way it would be convenient to store less data.<br>
I created a function let’s say f that does a slight modification of a graph in input (Data(nodes, edges, edges_attributes, y))<br>
when I charge my batch let’s say hypothetically 5 x1 … x5 I would instead input f(x1) … f(x5) but I noticed that with torch geometric the loading of a batch is kinda a black box and is loading different batches.</p>
Clearing cache in Nvidia xavier agx improves speed of copy from cpu to gpu for pytorch

SM19: <p>I am using Pytorch + tensorrt (using torch2trt) to run inference on the Nvidia Xavier AGX development kit.</p>
<p>The problem is, data loading from numpy array on CPU to torch cuda tensor using <em><strong>torch.from_numpy().float().to(“cuda:0”).unsqueeze_(0).permute(0, 3, 1, 2)</strong></em><br>
is very slow.</p>
<p>This copy operation takes a lot of time. (Nearly 70% of all time spent in one cycle of pre-process + inference + post-process).</p>
<p>I used python <strong>timeit</strong> module to measure time spent by each of pre-processing, inference and post-processing.</p>
<p>Surprisingly, when I do a cache clear using the jtop tool (<a href="https://github.com/rbonghi/jetson_stats" rel="noopener nofollow ugc">jetson-stats tool</a>), the copy time reduces drastically and my full cycle time now reduces as a result of that.</p>
<p>I am trying to understand:</p>
<ol>
<li>Why is memory copy from numpy to cuda tensor so slow? I am copying a 1024x1024x5 input. It takes about 60ms.</li>
<li>Why cache clear boosts speed? (After cache clear I see &lt; 20ms)</li>
</ol>
<p>The entire system runs as a ROS node.</p>
<p>Please help me understand.</p>
<p>Best Regards<br>
Sambit</p>

ptrblck: <p>Are you seeing different times by properly synchronizing the code before starting and stopping the timers?<br>
If so, could you check if some kind of cache is being used to offload the data?</p>

SM19: <p>I know that I need to call torch.cuda.synchronize(), but this only ensures that the copy is finished before moving on, right?</p>
<p>My problem is that the copy operation seems to take unnaturally long on the Xavier AGX devkit.</p>
<p>And then even more strangely, we I do torch.cuda.empty_cache(), things speed up a lot.<br>
So why is it like this?</p>
<p>Also,<br>
<em><strong>test_x = torch.from_numpy(test_x_npy).float().unsqueeze_(0).permute(0, 3, 1, 2).pin_memory()</strong></em></p>
<p>This seems to make no difference in speed.</p>
<p>Please note that the function runs in a loop inside a ROS subscriber node.<br>
So test_x get’s allocated in each iteration.</p>
<p>Sumary: How to make numpy array to cuda tensor copy fast?</p>

ptrblck: <aside class="quote no-group" data-username="SM19" data-post="3" data-topic="148816">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sm19/48/48889_2.png" class="avatar"> SM19:</div>
<blockquote>
<p>I know that I need to call torch.cuda.synchronize(), but this only ensures that the copy is finished before moving on, right?</p>
</blockquote>
</aside>
<p>Yes, and thus the timers are measuring the actual copy. If you are not synchronizing the code you might be measuring e.g. the kernel launches etc. which is wrong and wouldn’t mean that the actual copy was already done.</p>