akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Indexing a tensor with unique indices along each element of a batch

Pietro_Cicalese: <p>Lets say that we have a tensor of shape [N, C, D], where N = Batch, C = Channels, and D = H*W. For this example we can use a tensor of shape [10, 64, 9].</p>
<p>Lets say I want to index each input element along the D dimension with 4 unique indices per element (for example, [2, 5, 7, 8] for batch element one, [1, 2, 4, 6] for batch element two, … , [0, 7, 2, 8] for batch element ten) to yield an output tensor of size [10, 64, 4].</p>
<p>Is there a way to accomplish this without looping? I believe I would have to make use of torch.index_select somehow, but am not entirely certain. It would also be great if I could accomplish this in-place. Here is a code snippet of me solving this for just one batch element (and because I could really use the help and dont want people to think this is a low effort post):</p>
<pre><code class="lang-auto"># Where we get our indices
&gt;&gt;&gt;temp.shape
torch.Size([32768, 9])
# Our tensor of interest
&gt;&gt;&gt;temp2.shape
torch.Size([32768, 64, 9])
# Generating our indices
&gt;&gt;&gt;torch.topk(temp, 4).indices.shape
torch.Size([32768, 4])
# Generating our outputs for just one batch element.
&gt;&gt;&gt;torch.index_select(temp2[1], 1, torch.topk(temp, 4).indices[1]).shape
torch.Size([64, 4])
# How can we do this for all of the batch elements without looping/out-of-place operations?


</code></pre>

Pietro_Cicalese: <p>This appears to have been solved <a href="https://discuss.pytorch.org/t/batched-index-select/9115">here.</a> This is however not in-place as far as I can tell.</p>
<p>The solution:</p>
<pre><code class="lang-auto">def batched_index_select(input, dim, index):
    for ii in range(1, len(input.shape)):
        if ii != dim:
            index = index.unsqueeze(ii)
    expanse = list(input.shape)
    expanse[0] = -1
    expanse[dim] = -1
    index = index.expand(expanse)
    return torch.gather(input, dim, index)
</code></pre>
<p>If anyone can figure out an in-place solution it would be deeply appreciated, I will also keep trying and report back here if I make any progress.</p>
RuntimeError: running_mean should contain 32 elements not 1024?

Michael_Moran: <p>I get the error described above when attempting to perform batch norm on my network</p>
<p>The isolated code for the section that fails to run is</p>
<pre><code>            print(layer_f)
            print(layer_b)
            x = self.af(layer_f(x))
            print(x.shape)
            x = layer_b(x)
</code></pre>
<p>The print out for the objects are</p>
<p>Linear(in_features=84, out_features=1024, bias=True)<br>
BatchNorm1d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
torch.Size([11, 32, 1024])</p>
<p>From what I understand I am passing my 11,32,84 tensor into the linear layer to get a 11,32,1024 tensor then pushing that tensor through to a batch norm layer of size 1024. I do not understand what has gone wrong here?</p>
<p>The input dimensions are [Batch,channel,element] and the number of channels changes for each input.</p>

ptrblck: <p>As you’ve already described the number of channels in the input activation to the batchnorm layer doesn’t match the expected channels, so you would have to permute the activation via:</p>
<pre><code class="lang-python">x = x.permute(0, 2, 1)
</code></pre>
<p>before passing it to <code>layer_b</code>.</p>

Michael_Moran: <p>If I do this wouldn’t things be normalized with respect to the channels rather than to the features or have I misunderstood something?</p>

ptrblck: <p>Permuting the output of the linear layer would assign the <code>out_features</code> dimension to <code>dim1</code>, which is the “channels” dimension in batchnorm layers.<br>
I assumed this is the expected use case, since <a class="mention" href="/u/michael_moran">@Michael_Moran</a> defined the size of these dimensions both as <code>1024</code>. <code>dim1</code> in the original output of the linear layer (size of <code>32</code>) is the “additional” dimension and is often used e.g. for the temporal dimension.</p>
<p>If this approach is correct or not depends on the use case and might be wrong for you so you would have to explain a bit what you are trying to achieve.</p>

Michael_Moran: <p>In the end I got the desired behaviour by doing the following snippet</p>
<p>i,j,k = x.shape<br>
x = x.view(i*j,k)<br>
x = batch_layer(x)<br>
x = x.view(i,j,k)</p>
<p>This seems to have worked where the usage case was that I have a minibatch of sets / unordered sequences and I wanted the embedding of each token in these sets / sequences to be normalized in the same way with the same parameters.</p>
<p>This isn’t an NLP task but the language is appropiate so I will use it.</p>
<p>I wanted to normalize each word token consistently with the same parameters. In the context of the normalization the batch of sentences where each sentence contained words was abstracted down to just being a large batch of words. I was unsure how batch_norm1d would actually handle things using the permutation solution.</p>
<p>I apologise if I’m unclear as I’m unsure how batch_norm1d behaves with 3d tensors.</p>
Why is nn.Embedding behaving differently on CPU and GPU?

amogh112: <p>I created an embedding and passing out of index tensors doesn’t work on CPU, but on GPU it is returning a tensor(all except first embedding keep changing with every call, and even the first tensor is not equal to embedding.weight).<br>
Torch version = 1.5.0</p>
<pre><code class="lang-auto">a = torch.nn.Embedding(1,768)
&gt;&gt;&gt; a(torch.LongTensor([1,2,3]))
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "&lt;path&gt;/envs/vl-bert/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "&lt;path&gt;/envs/vl-bert/lib/python3.6/site-packages/torch/nn/modules/sparse.py", line 114, in forward
    self.norm_type, self.scale_grad_by_freq, self.sparse)
  File "&lt;path&gt;/vl-bert/lib/python3.6/site-packages/torch/nn/functional.py", line 1724, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
IndexError: index out of range in self

a.cuda()(torch.LongTensor([1,2,3]).cuda())
tensor([[1.4013e-45, 0.0000e+00, 2.8026e-45,  ..., 0.0000e+00, 0.0000e+00,
         0.0000e+00],
        [0.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 0.0000e+00, 0.0000e+00,
         0.0000e+00],
        [0.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 0.0000e+00, 0.0000e+00,
         0.0000e+00]], device='cuda:0', grad_fn=&lt;EmbeddingBackward&gt;)
a.cuda()(torch.LongTensor([1,2,3]).cuda())
tensor([[ 1.4013e-45,  0.0000e+00,  2.8026e-45,  ...,  1.0869e+00,
         -1.7131e+00, -6.9908e-01],
        [-5.6997e-01,  1.6486e+00,  1.7096e+00,  ...,  1.0869e+00,
          1.7131e+00,  6.9908e-01],
        [ 5.6996e-01,  1.6486e+00,  1.7096e+00,  ...,  1.7427e+00,
          2.0000e+00,  1.7774e+00]], device='cuda:0',
       grad_fn=&lt;EmbeddingBackward&gt;)
a.cuda()(torch.LongTensor([1,2,3]).cuda())
tensor([[1.4013e-45, 0.0000e+00, 2.8026e-45,  ..., 0.0000e+00, 0.0000e+00,
         0.0000e+00],
        [0.0000e+00, 2.0000e+00, 0.0000e+00,  ..., 1.0869e+00, 1.7131e+00,
         6.9908e-01],
        [6.5202e+06, 4.3207e+00, 8.6057e-02,  ..., 1.7427e+00, 2.0000e+00,
         1.7774e+00]], device='cuda:0', grad_fn=&lt;EmbeddingBackward&gt;)

&gt;&gt;&gt; a.weight
Parameter containing:
tensor([[ 0.7804,  1.5051,  0.0861, -0.9269, -0.8105, -2.7018, -1.2860, -0.4517,
          0.6019,  1.2832,  2.1942,  0.3216,  1.9599,  0.8146,  0.0085,  0.6976,
          1.9618,  0.0783,  1.3515,  0.8830,  0.8101, -2.4665,  2.6164,  1.1543,
         -0.8128, -0.9217,  1.3534, -0.3387,  0.1712,  1.1185, -0.5681,  0.2406,
          1.8387,  0.7704,  1.6712,  0.4060, -1.2792, -0.3
</code></pre>

ptrblck: <p>Some CUDA assert statements were accidentally disabled in PyTorch <code>1.5.0</code>, so you should update to the latest stable version, as it was fixed in <code>1.5.1</code>.</p>
File "/home/jake/detectron2/detectron2/engine/train_loop.py", line 280, in _write_metrics

Sangwon_Jake: <p>I’m doing detection problem with  cascade rcnn with detectron2.<br>
Testing the augmentation right now.</p>
<pre><code class="lang-auto">T.RandomFlip(prob=0.5, horizontal=True, vertical=False), 
T.ResizeShortestEdge(short_edge_length=(640, 672, 704, 736, 768, 800), max_size=1333, sample_style='choice')
</code></pre>
<p>with this two options it has no problem.<br>
but when I add the</p>
<p><code> </code>Preformatted text<code> ,T.RandomCrop('relative_range', (0.4, 0.5))</code></p>
<p>the error occured</p>
<pre><code class="lang-auto">FloatingPointError: Loss became infinite or NaN at iteration=1!
loss_dict = {'loss_cls_stage0': 1.613979458808899, 'loss_box_reg_stage0': 0.011088866740465164, 'loss_cls_stage1': 1.668488621711731, 'loss_box_reg_stage1': 0.00575869157910347, 'loss_cls_stage2': 1.4664241075515747, 'loss_box_reg_stage2': 0.0028667401056736708, 'loss_rpn_cls': 1.469098448753357, 'loss_rpn_loc': inf}
[11/21 06:53:37 d2.engine.hooks]: Total training time: 0:00:00 (0:00:00 on hooks)
[11/21 06:53:37 d2.utils.events]:  iter: 1  total_loss: 4.9  loss_cls_stage0: 1.609  loss_box_reg_stage0: 0.005077  loss_cls_stage1: 1.685  loss_box_reg_stage1: 0.002878  loss_cls_stage2: 1.464  loss_box_reg_stage2: 0.002352  loss_rpn_cls: 0.09238  loss_rpn_loc: 0.03937  data_time: 0.1260  lr: 2e-05  max_mem: 5386M
</code></pre>
<p>My initial value of parameter is as follow</p>
<pre><code class="lang-auto">cfg.DATALOADER.NUM_WORKERS = 4
cfg.MODEL.WEIGHTS = model_zoo.get_checkpoint_url("Misc/cascade_mask_rcnn_R_50_FPN_3x.yaml")  # Let training initialize from model zoo
cfg.SOLVER.IMS_PER_BATCH = 6#6
cfg.SOLVER.MAX_ITER = 15000    # Need to train longer
#cfg.SOLVER.CHECKPOINT_PERIOD = 1000
cfg.MODEL.RESNETS.DEPTH = 50 #RESNET 50,101
#RESNEXT parameters
cfg.SOLVER.CHECKPOINT_PERIOD = 200
cfg.MODEL.ROI_HEADS.NAME= "CascadeROIHeads"
cfg.MODEL.ROI_HEADS.NUM_CLASSES = 4  # 4 classes
cfg.MODEL.MASK_ON = False
cfg.OUTPUT_DIR = save_path
cfg.SOLVER.LR_SCHEDULER_NAME = 'WarmupCosineLR'
cfg.SOLVER.BASE_LS = 0.0001
</code></pre>
How to have deterministic dropout?

seyeeet: <p>I want to use dropout on input1 and input2 (where they are both tensors with similar sizes) but I want it to be the same dropout for both of them. meaning that it zeros(drops) the same elements on both of them. how can i do that?</p>

ptrblck: <p>I would sample a <code>mask</code> manually and apply it to both tensors (with the scaling, if needed).<br>
Alternatively, you could also try to seed the dropout call before its usage, but I would rather avoid these kind of seed-hacks.</p>

seyeeet: <p>I see thank you, would that be possible to see what elements the dropout will zero out if we know the size of tensor?<br>
I mean I can check the output of tensor to see where it is zero but that is kinda risky (if the tensor has zero elements then that would cause issue, though it is not too difficult to check I guess).<br>
if I can get the mask from the dropout then the problem is solved.</p>

ptrblck: <p>I think you can use your mentioned approach, but I would rather sample the mask manually instead of checking the input and output for zeros.</p>

seyeeet: <p>just to confirm is this a right way of creating mask:<br>
let say I want to do dropout with p=0.1</p>
<pre><code class="lang-auto">import torch
input1 = torch.rand(5,2,2)
mask = torch.bernoulli(input1.data.new(input1.data.size()).fill_(1-p))
input_after_dropout = mask*input1
</code></pre>

ptrblck: <p><code>torch.bernoulli</code> expects an <code>input</code> tensor containing the probabilities of drawing a <code>1</code> value. So depending how <code>p</code> is defined your code should be correct.<br>
Note that you would have to take care of the scaling in dropout layers (either the inverse scaling during training or the vanilla scaling during evaluation).<br>
To switch the behavior between training and evaluation you could create a custom <code>nn.Module</code> and use the internal <code>self.training</code> flag to switch between the behaviors.<br>
The <code>self.training</code> flag will be changed through calls into <code>model.train()</code> and <code>model.eval()</code>.</p>
Advanced/Fancy indexing across batches

Glerk: <p>Hi,</p>
<p>Suppose I have a batch b consisting of BxCxHxW, and I have two tensors y and x of size BxHxW. my goal is to return a tensor b_sampled consists of B tensors which have been sampled according to the entries of x and y.</p>
<p>In other wants I want, b_sampled[0] = b[0,[x[0,:,;], y[0,:,:]]], b_sampled[1] = [1,[x[1,:,:],y[1,:,:]]],…</p>
<p>I can do this in a loop but there must be a better way.</p>

Pietro_Cicalese: <p>You can do this using:</p>
<pre><code class="lang-auto">def batched_index_select(input, dim, index):
    for ii in range(1, len(input.shape)):
        if ii != dim:
            index = index.unsqueeze(ii)
    expanse = list(input.shape)
    expanse[0] = -1
    expanse[dim] = -1
    index = index.expand(expanse)
    return torch.gather(input, dim, index)
</code></pre>
<p>Taken from <a href="https://discuss.pytorch.org/t/batched-index-select/9115">here.</a></p>
USE_CUDA=0 has no effect with cmake build

John_J_Watson: <p>I am trying to build libtorch like so:</p>
<pre><code class="lang-auto">USE_CUDA=0 cmake -DBUILD_SHARED_LIBS:BOOL=ON -DCMAKE_BUILD_TYPE:STRING=Release -DPYTHON_EXECUTABLE:PATH=`which python3` -DCMAKE_INSTALL_PREFIX:PATH=../pytorch-install ../pytorch &amp;&amp;\
USE_CUDA=0 cmake --build . --target install
</code></pre>
<p>However, I see that it is still building with CUDA <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"> not sure why this is…</p>

albanD: <p>Hi,</p>
<p>Why are you building with CMake directly? If you want to do a python install, you should be using <code>setup.py</code> and the <code>USE_CUDA</code> flag will be properly picked up in that case.</p>
<p>If you are trying to do a libtorch install (I am not sure what is the process supposed to be), it might be that the flag is not the same for it.</p>

John_J_Watson: <p>my goal is to build a minimalist cpu build of libtorch.</p>
<p>I now see what you mean that I can use python to build libtorch:<br>
</p><aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/blob/master/docs/libtorch.rst" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/master/docs/libtorch.rst" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/master/docs/libtorch.rst</a></h4>
<pre><code class="lang-rst">libtorch (C++-only)
===================

The core of pytorch does not depend on Python. A
CMake-based build system compiles the C++ source code into a shared
object, libtorch.so.

Building libtorch using Python
------------------------------

You can use a python script/module located in tools package to build libtorch
::
   cd &lt;pytorch_root&gt;

   # Make a new folder to build in to avoid polluting the source directories
   mkdir build_libtorch &amp;&amp; cd build_libtorch

   # You might need to export some required environment variables here.
   Normally setup.py sets good default env variables, but you'll have to do
   that manually.
</code></pre>

  This file has been truncated. <a href="https://github.com/pytorch/pytorch/blob/master/docs/libtorch.rst" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>but I think the problem is I am not sure if I can link to through through other C++ programs.</p>
<p>Anyways, I will try it now.</p>
Why dose tensor.argmax return last index of maximum values?

xinsss: <p>Based on the documentation, the tensor.argmax should returns the index of the first occurrence of multiple largest values.</p>
<p>But when the tensor contains all same values, it returns the index of the last value.</p>
<p>For example, <code>torch.tensor([0, 0, 0, 0]).argmax(dim=0)</code>, outputs is <code>tensor(3)</code>. Shouldn’t it be 0?</p>

albanD: <p>Hi,</p>
<p>Which version of pytorch are you using?<br>
This was fixed in 1.7+ versions only IIRC</p>

xinsss: <p>Got it. Thanks. I was using 1.6.0.</p>
Wrapping All Parameters of a Model with a Nonlinearity

kko: <p>I want to wrap all of my convolution parameters (.weight and .bias) with a sigmoid function which I want to make part of the graph i.e. sigmoid’s should also be part of the backprop. In other words, weight and bias values should never be out of range [0, 1].</p>
<p>What is the suggested way of doing this?</p>
<p>Thanks.</p>

Abhilash_Srivastava: <p>Try using <a href="https://pytorch.org/docs/stable/generated/torch.clamp.html" rel="noopener nofollow ugc">torch.clamp</a> over the model parameters.</p>

kko: <p>Thanks, I am aware of torch.clamp. But, ignoring [0,1] range what if I want to use a specific function like sigmoid, square of the weights. Is there a way to this via hooks etc.?</p>
How to use sigmoid on only positive values?

Richard_S: <p>I have a binary classification model, that in the latest linear layer, it outputs only positive values (don’t ask why, that’s a different matter), now when i give the final layer’s output to torch.sigmoid, all the results are above 50%, because the final linear layer is only outputting positive values, how can i fix this and output probability? is there any “positive only” sigmoid in pytorch?</p>

KFrank: <p>Hi Richard!</p>
<aside class="quote no-group" data-username="Richard_S" data-post="1" data-topic="103495" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/r/3be4f8/48.png" class="avatar"> Richard_S:</div>
<blockquote>
<p>I have a binary classification model, that in the latest linear layer, it outputs only positive values</p>
</blockquote>
</aside>
<p>This is the core issue you need to address – tweaking <code>sigmoid()</code> to<br>
undo whatever damage you’ve done would be a sideshow.</p>
<aside class="quote no-group">
<blockquote>
<p>(don’t ask why, that’s a different matter)</p>
</blockquote>
</aside>
<p>Well, it actually does matter, because whatever you’ve done is<br>
breaking the interpretation of the output of your model as reasonable<br>
logits (that become reasonable probabilities when passed through<br>
<code>sigmoid()</code>).</p>
<p>My advice:  Tell us what you are actually doing and what your<br>
motivation is.  My guess is that you’ll be able to use standard<br>
techniques to build your classifier.  But if your problem has some<br>
unusual properties that prevent you form using standard techniques,<br>
you’ll likely get better advice from the forum if you describe your<br>
problem and what makes it atypical.</p>
<aside class="quote no-group">
<blockquote>
<p>the final linear layer is only outputting positive values, how can i fix this and output probability? is there any “positive only” sigmoid in pytorch?</p>
</blockquote>
</aside>
<p>(As an aside, and against my better judgment, I will comment on some<br>
mathematical structure.  But please don’t actually try doing this.  You<br>
want to convert “positive-only” things that look sort of like logits into<br>
“normal” logits that range from <code>-inf</code> to <code>inf</code>, and that therefore become<br>
“normal” probabilities that range from <code>0</code> to <code>1</code> when passed through<br>
<code>sigmoid()</code>.  The <code>log()</code> function maps the positive half real line to the<br>
whole real line, so you could use the following conversion:<br>
<code>normal_logit = log (positive_only_logit)</code>.)</p>
<p>Good luck.</p>
<p>K. Frank</p>
Error occurs when i install pytorch on windows by pip or conda

ninesun127: <ol>
<li>when i run pip3 command   or similiar command,it said;</li>
</ol>
<p>torch-1.0.1-cp37-cp37m-win_amd64.whl is not a supported wheel on this platform.<br>
when i run</p>
<p>2.when i run conda install pytorch-cpu torchvision-cpu -c pytorch<br>
it said<br>
Solving environment: failed</p>
<p>CondaHTTPError: HTTP 000 CONNECTION FAILED for url An HTTP error occurred when trying to retrieve this URL.<br>
HTTP errors are often intermittent, and a simple retry will get you on your way.</p>
<p>If your current network has <a href="https://www.anaconda.com" rel="nofollow noopener">https://www.anaconda.com</a> blocked, please file<br>
a support request with your network engineering team.</p>
<p>SSLError(MaxRetryError(‘HTTPSConnectionPool(host=‘<a href="http://repo.anaconda.com" rel="nofollow noopener">repo.anaconda.com</a>’, port=443): Max retries exceeded with url: /pkgs/main/noarch/repodata.json.bz2 (Caused by SSLError(“Can’t connect to HTTPS URL because the SSL module is not available.”))’))</p>

ptrblck: <p>It seems to be a conda bug. Could you try the suggestions posted in <a href="https://github.com/conda/conda/issues/6064#issuecomment-334087725" rel="nofollow noopener">this issue</a>?</p>

karynaur: <p>deactivate conda and<br>
conda update anaconda-navigator</p>
<p>should work.</p>
Reducing the file size of libtorch_cpu.so

John_J_Watson: <p>I am attempting to use libtorch on aws lambda which has a limit of 250MB. However, my <code>libtorch_cpu.so</code> is 290MB and so I am not able to upload it on lambda <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"></p>
<p>I have a cpu build from here:<br>
<a href="https://download.pytorch.org/libtorch/cpu/libtorch-cxx11-abi-shared-with-deps-1.7.0%2Bcpu.zip" class="onebox" target="_blank" rel="noopener nofollow ugc">https://download.pytorch.org/libtorch/cpu/libtorch-cxx11-abi-shared-with-deps-1.7.0%2Bcpu.zip</a></p>
GRU hidden size vs input size

Pranavan_Theivendira: <p>Hi All,</p>
<p>When we use GRU with multiple layers, the previous hidden state is used as the input to the upper layers of GRU. [As per this doc - <a href="https://pytorch.org/docs/stable/generated/torch.nn.GRU.html" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.nn.GRU.html</a>]</p>
<p>Suppose my input size is 8 and hidden size is 64. How can we use the hidden state as the input to the next layer? With this setting, the model learns. Does that mean PyTorch truncate the hidden size to match the input size (or some other mechanism to change dimension)? I am not really clear in this scenario. Looking forward to a good explanation.</p>
<p>Thanks</p>

vdw: <p>Only the first layer has the input size of your original data, e.g., 8. The upper layers use the size of the hidden state as input size.</p>
Mean columnwise log loss implementation

IamSparky: <p>My main concern is how &amp; where should I keep the log_loss_metric in my pytorch training as well as for my evaluation loop to calculate the mean column wise log_loss value? As I am doing multilabel binary classification there are 206 prediction columns in total.</p>
<p>This is how I am writing my evaluation loop,</p>
<pre><code class="lang-auto">def eval_loop_fn(data_loader, model, device):
    running_loss = 0.0
    score = 0.0
    
    model.eval()
    
    for batch_index,dataset in enumerate(data_loader):
        tabular_data = dataset["tabular_data"]
        output = dataset["output"]
        
        tabular_data = tabular_data.to(device, dtype=torch.float)
        targets = output.to(device, dtype=torch.float)

        outputs = model(tabular_data)
        
        loss = loss_fn(outputs , targets)

        running_loss += loss.item()
    
    valid_loss = running_loss / float(len(val_data))

    xm.master_print('validation Loss: {:.4f} '.format(valid_loss))
</code></pre>
ERROR: Could not find a version that satisfies the requirement torchaudio

kang-im: <p>I just tried to install PyTorch in mac with python 64bit in Pycharm,</p>
<blockquote>
<p>pip install torch torchvision torchaudio</p>
</blockquote>
<p>the result <img src="https://discuss.pytorch.org/images/emoji/apple/expressionless.png?v=12" title=":expressionless:" class="emoji" alt=":expressionless:" loading="lazy" width="20" height="20"></p>
<p>ERROR: Could not find a version that satisfies the requirement torchaudio (from versions: none)<br>
ERROR: No matching distribution found for torchaudio</p>
<p>Is there something wrong with the setting or what ?</p>

kang-im: <p>I found to install Torch <img src="https://discuss.pytorch.org/images/emoji/apple/expressionless.png?v=9" title=":expressionless:" class="emoji" alt=":expressionless:"></p>
<p>pip install --pre torch -f “<a href="https://download.pytorch.org/whl/nightly/%24%7BWHEEL_DIR%7Dtorch_nightly.html" rel="noopener nofollow ugc">https://download.pytorch.org/whl/nightly/${WHEEL_DIR}torch_nightly.html</a>”</p>
What is the correct place to apply dropout at Conv and Linear layers?

jarvico: <p>Hi,</p>
<p>I am a bit confused about where to exactly apply dropout in CNN network.</p>
<p>In the below model I applied dropout in both of the Conv layers and also in the linear layer.</p>
<p>But I am not sure whether I need to apply it. After ReLu? or before ReLu ? in linear layers.</p>
<p>And also I am not sure if I implemented dropout in correct place in Conv layers.</p>
<p><strong>I am experimenting on dropout mc outputs of the CNN model : uncertainty metrics</strong></p>
<p>I got different mean confidence values and uncertainty values, when I used dropout before or after the F.relu for fc1.</p>
<pre><code class="lang-auto">class CNN_dropout(nn.Module):
    def __init__(self):
        super(CNN_dropout, self).__init__()
        self.conv1 = nn.Conv2d(1, 10, kernel_size=5)
        self.conv2 = nn.Conv2d(10, 20, kernel_size=5)
        self.fc1 = nn.Linear(320, 50)
        self.fc2 = nn.Linear(50, 10)
        self.drop_layer = nn.Dropout(p=0.5)

    def last_hidden_layer_output(self, x):
        x = F.max_pool2d(F.relu(self.drop_layer(self.conv1(x))), 2)
        x = F.max_pool2d(F.relu(self.drop_layer(self.conv2(x))), 2)
        x = x.view(-1, 320)
        x = F.relu(self.drop_layer(self.fc1(x)))
        return x

    def forward(self, x):
        x = self.last_hidden_layer_output(x)
        x = self.fc2(x)
        return x
</code></pre>
<p>My experiment results differ when I switch</p>
<p>from</p>
<pre><code class="lang-auto">    def last_hidden_layer_output(self, x):
        x = F.max_pool2d(F.relu(self.drop_layer(self.conv1(x))), 2)
        x = F.max_pool2d(F.relu(self.drop_layer(self.conv2(x))), 2)
        x = x.view(-1, 320)
        x = F.relu(self.drop_layer(self.fc1(x)))
        return x
</code></pre>
<p>to</p>
<pre><code class="lang-auto">    def last_hidden_layer_output(self, x):
        x = F.max_pool2d(F.relu(self.drop_layer(self.conv1(x))), 2)
        x = F.max_pool2d(F.relu(self.drop_layer(self.conv2(x))), 2)
        x = x.view(-1, 320)
        x = F.relu(self.fc1(x))
        x = self.drop_layer(x)
        return x
</code></pre>
<p>If I am sure where exactly to apply dropout in linear layer, maybe I also need to change the place of dropout ın conversation layers too?</p>
<p>To me, these seem to be a better choice but I am not sure unfortunately.</p>
<pre><code class="lang-auto">x = F.relu(self.drop_layer(self.fc1(x)))
</code></pre>
<pre><code class="lang-auto">x = F.max_pool2d(F.relu(self.drop_layer(self.conv1(x))), 2)
</code></pre>

Abhilash_Srivastava: <p>I usually go for this:<br>
x -&gt; conv -&gt; relu -&gt; maxpool -&gt; dropout -&gt; linear</p>

jarvico: <p>or is it like?</p>
<pre><code class="lang-auto">        x = self.drop_layer(F.max_pool2d(F.relu(self.conv1(x)), 2))
</code></pre>
<p>and</p>
<pre><code class="lang-auto">        x = self.drop_layer(F.relu(self.fc1(x)))
</code></pre>
When torchtext was used to load the GLOVE data set, 503 errors occurred. What should I do?

ZhangSheng: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/0/4012d5e0b651ec932e2489e728d1a0020149d9a3.png" data-download-href="https://discuss.pytorch.org/uploads/default/4012d5e0b651ec932e2489e728d1a0020149d9a3" title="5OPWA{UO~0EGPTVLQ(4Q"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/0/4012d5e0b651ec932e2489e728d1a0020149d9a3_2_690x257.png" alt="5OPWA{UO~0EGPTVLQ(4Q" data-base62-sha1="98OUBaDWHiF1ZKB9hG9TghgaSjx" width="690" height="257" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/0/4012d5e0b651ec932e2489e728d1a0020149d9a3_2_690x257.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/0/4012d5e0b651ec932e2489e728d1a0020149d9a3_2_1035x385.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/0/4012d5e0b651ec932e2489e728d1a0020149d9a3_2_1380x514.png 2x" data-dominant-color="F5F4F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">5OPWA{UO~0EGPTVLQ(4Q</span><span class="informations">1382×516 36 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

Abhilash_Srivastava: <p>Network error. Try again, with good network!</p>
Pointers to analyze overfitting

aswamy: <p>I need help in analyzing the overfitting behaviour, I have tried a lot of possible solutions, but nothing was successful.<br>
To explain a bit about the problem:<br>
Basically, I have a model that is trained for the binary classification of 3D points.<br>
Single training pair looks like this: (an image, N 3D points and N labels(binary labels)<br>
The goal is to classify points corresponding to the input image. Model: Each 3D points is linear transformed using single layer MLP and then fed through resnet18 blocks conditioned upon images encoded using pre-trained imagenet model to decode the probability of two classes. (Using this <a href="https://github.com/autonomousvision/occupancy_networks" rel="noopener nofollow ugc">repository</a> for as base code)</p>
<p>Model is overfitting heavily(validation class 0 accuracies are stuck at around 45%)</p>
<p>Note: There is an imbalance in points labels(class 0 labels are 10% and class 1 labels are 90%). But while sampling points, I tried to sample both classes equally.</p>
<p>I tried data augmentations(random crop, flip) and decaying weights but nothing helped.</p>
<p>Could you give any pointers to better analyze and understand the reason for this overfitting?</p>
<p>Thank you in Advance!</p>
Connecting a backbone to a head

nscotto: <p>Hello,</p>
<p>how would I automate the connection between a backbone and a head?<br>
I’m interested into developing some code for replacing the backbone of any model, if its backbone is known by the code.</p>
<p>So let’s say I have a model using a resnet backbone, I detected the backbone and the head, and I would like to replace resnet by mobilenet.<br>
The problem I see is mainly the mismatch between dimensions, I guess I need a layer in between to make the connection.</p>
ARM processor offloading

111421: <p>Hi, I’m a newbie in PyTorch.<br>
I’ve been wondering if there is any reference or project going on or done already about offloading task to ARM processor.<br>
I’ve wondered this by the reason below.<br>
As far as I’m aware of, target devices, such as GPU, FPGA and etc, are used for offloading computation of some NN models.<br>
The target devices are assumed to be connected to the mother board via PCIe interfaces.</p>
<p>My project is about offloading computation workload of a NN model to ARM processor currently and other DSP in the future.<br>
So I’m working on building PyTorch to detect ARM processor on PCIe bus and include library for ARM processor offloading execution.</p>
<p>In summary, my questions are these.</p>
<ol>
<li>Is there any project or references that I can refer to which is about offloading a task/computation to a ARM processor via PCIe interfaces.</li>
<li>If else, what about NEON architecture? I mean I can find some source code on PyTorch source code of NEON execution. Thus I wonder if it is possible to offload task to a NEON processor on a board which is connected to a mother board via PCIe interface?</li>
</ol>
Transformer model prediction same every time

utkuumetin: <p>Hello, i tried make masked word prediction model, but every time i give input to model masked word prediction is same. And weird way, its change with epochs, example with all inputs its predict “the” token, then i increase epoch and its predict “we” token.</p>
<p>Example:</p>
<pre><code class="lang-auto">Epoch 2:
Input:
[2, 42, 43, 44, 45, 46, 28, 35, 47, 48, 8, 3]
 sss of morning at blazing star and the settlement awoke to eee
[2, 42, 1, 44, 45, 46, 28, 35, 47, 48, 8, 3]
 sss of unnk at blazing star and the settlement awoke to eee
Output
[2, 42, 35, 44, 45, 46, 28, 35, 47, 48, 8, 3]
 sss of the at blazing star and the settlement awoke to eee

Epoch 10:
Input:
[2, 42, 43, 44, 45, 46, 28, 35, 47, 48, 8, 3]
 sss of morning at blazing star and the settlement awoke to eee
[2, 42, 1, 44, 45, 46, 28, 35, 47, 48, 8, 3]
 sss of unnk at blazing star and the settlement awoke to eee
Output
[2, 42, 100, 44, 45, 46, 28, 35, 47, 48, 8, 3]
 sss of we at blazing star and the settlement awoke to eee
</code></pre>
<p>What will cause this? Model doesn’t overfitting.</p>
Calculate accuracy of autoencoder

Uma: <p>First I train the autoencoder for number of epochs for normal images and save the model. Then I load it and evaluate it for validation dataset consist of normal and abnormal images and calculate the reconstruction loss for each image in validation set. Then I add the each reconstruction loss and the label of that image (normal or abnormal) to two arrays. Then I use sklearn, svm to to fit the validation set. Then I predict the test images using svm and calculate the accuracy.</p>
<p>for data in dataloader:<br>
images, label, paths = data<br>
x = images.to(device)<br>
with torch.set_grad_enabled(False):<br>
x_reconstructed = model(x)<br>
for i in range(len(x)):<br>
reconstructed_loss = reconstruct_loss_fn(x_reconstructed[i], x[i])<br>
if test == “abnormal”:<br>
label = 0<br>
if test == “normal”:<br>
label = 1<br>
trainDdata.append(reconstructed_loss)<br>
targets.append(label)</p>
<p>model1 = svm.SVC()<br>
trainDdata= (np.array(trainDdata)).reshape(-1, 1)<br>
targets= (np.array(targets)).reshape(-1, 1)<br>
model1.fit(trainDdata, targets.ravel())</p>
<p>testDdata= (np.array(testDdata)).reshape(-1, 1)<br>
testTargets= (np.array(testTargets)).reshape(-1, 1)<br>
targets2 = model1.predict(testDdata)<br>
targets2= (np.array(targets2)).reshape(-1, 1)<br>
equals = targets2 == testTargets.reshape(*targets2.shape)<br>
print(f"Accuracy = {np.mean(equals.astype(int))*100:.3f}%")<br>
I’m new to pytorch and I have a doubt whether this is correct?</p>
Convert deform_conv2d to ONNX

Tuyen_Vo_Quang: <p>I would like to convert models with deform_conv2d to ONNX format, but currently it’s only defined in torchvision: <a>https://github.com/pytorch/vision/blob/master/torchvision/ops/deform_conv.py</a></p>
<p>So i add code to torchvision like this:</p>
<pre><code class="lang-auto">    @parse_args('v', 'v', 'v', 'v', 'is', 'is', 'is')
    def deform_conv(g, input, offset, weight, bias, stride, padding, dilation):
        return g.op('DeformConv', input, offset, weight, stride, padding, dilation)

    from torch.onnx import register_custom_op_symbolic
    register_custom_op_symbolic('torchvision::nms', symbolic_multi_label_nms, _onnx_opset_version)
    register_custom_op_symbolic('torchvision::roi_align', roi_align, _onnx_opset_version)
    register_custom_op_symbolic('torchvision::roi_pool', roi_pool, _onnx_opset_version)
    register_custom_op_symbolic('torchvision::deform_conv', deform_conv, _onnx_opset_version)
</code></pre>
<p>However, it does not work. Could you suggest me any idea to fix. What is the purpose of g.op()?</p>
How to use torch.onnx.export with customed input datatype, like SparseTensor?

nan0755: <p>In this repo <a href="https://github.com/mit-han-lab/torchsparse" rel="noopener nofollow ugc">torchsparse</a>, there is a customed datatype <a href="https://github.com/mit-han-lab/torchsparse/blob/d2a5817c1b30565ffdfcd191b171a0957db408a8/torchsparse/sparse_tensor.py#L6" rel="noopener nofollow ugc">SparseTensor`</a>.</p>
<pre data-code-wrap="python"><code class="lang-python">class SparseTensor:
    def __init__(self, feats, coords, cur_tensor_stride=1):
        self.F = feats
        self.C = coords
        self.s = cur_tensor_stride
        self.coord_maps = {}
        self.kernel_maps = {}

    def check(self):
        if self.s not in self.coord_maps:
            self.coord_maps[self.s] = self.C

    def cuda(self):
        assert type(self.F) == torch.Tensor
        assert type(self.C) == torch.Tensor
        self.F = self.F.cuda()
        self.C = self.C.cuda()
        return self

    def detach(self):
        assert type(self.F) == torch.Tensor
        assert type(self.C) == torch.Tensor
        self.F = self.F.detach()
        self.C = self.C.detach()
        return self

    def to(self, device, non_blocking=True):
        assert type(self.F) == torch.Tensor
        assert type(self.C) == torch.Tensor
        self.F = self.F.to(device, non_blocking=non_blocking)
        self.C = self.C.to(device, non_blocking=non_blocking)
        return self

    def __add__(self, other):
        tensor = SparseTensor(self.F + other.F, self.C, self.s)
        tensor.coord_maps = self.coord_maps
        tensor.kernel_maps = self.kernel_maps
        return tensor
</code></pre>
<p>And I want to export to ONNX model, but when I ran <code>torch.onnx.export</code>, I got this ERROR:</p>
<pre><code class="lang-auto">RuntimeError: Only tuples, lists and Variables supported as JIT inputs/outputs. 
Dictionaries and strings are also accepted but their usage is not recommended. 
But got unsupported type SparseTensor
</code></pre>
<p>This problem may be same to other custome data types.</p>
<p>I also noticed this line in <a href="https://github.com/pytorch/pytorch/blob/6da26fe79b7045fac743c81ca8d38c5340de17ab/torch/onnx/__init__.py#L45" rel="noopener nofollow ugc">torch.onnx.<strong>init</strong>.py</a><br>
What do you mean by this ?</p>
<blockquote>
<p>Any non-Tensor arguments (including None) will be hard-coded into the exported model</p>
</blockquote>
<p>Thanks in advance for any help!</p>
How to retrieve learning rate from ReduceLROnPlateau scheduler

kirk86: <p>Hi there,<br>
I was wondering if someone could shed some light on the following questions:</p>
<ol>
<li>
<p>Why is ReduceLROnPlateau the only object without <code>get_lr()</code> method among all schedulers?</p>
</li>
<li>
<p>How to retrieve the learning rate in this case? Previously without scheduler I would do <code>optimizer.param_groups[0]['lr']</code> but now after using the scheduler and printing <code>optimizer.param_groups[0]['lr']</code> I see no change in the learning rate?</p>
</li>
<li>
<p>How to save the optimizer state if you’re using a scheduler. Previous examples I’ve seen show <code>optimizer.state_dict()</code> in order to save the optimizer state should we replace that with <code>scheduler.state_dict()</code>?</p>
</li>
</ol>
<p>Thanks!</p>

ptrblck: <ol>
<li>
<p>The scheduler might track multiple <code>param_groups</code> as described <a href="https://github.com/pytorch/pytorch/issues/2829" rel="nofollow noopener">here</a>.</p>
</li>
<li>
<p>It should still work as shown in this example:</p>
</li>
</ol>
<pre><code class="lang-python">model = nn.Linear(10, 2)
optimizer = optim.Adam(model.parameters(), lr=1e-3)
scheduler = optim.lr_scheduler.ReduceLROnPlateau(
    optimizer, patience=10, verbose=True)

for i in range(25):
    print('Epoch ', i)
    scheduler.step(1.)    
    print(optimizer.param_groups[0]['lr'])
</code></pre>
<ol start="3">
<li>No, you should still save the optimizer’s <code>state_dict</code> (and also call <code>optimizer.step()</code>, as the scheduler is not a replacement for the optimizer). Additionally, you could also store the scheduler’s <code>state_dict</code>.</li>
</ol>

Cognac: <p>Thanks for the great answer!</p>
<p>By the way, is there anyway to obtain the learning rate from the <code>ReduceLROnPlateau</code> scheduler?<br>
When I’m using other schedulers they all have <code>get_last_lr()</code> function; but since <code>ReduceLROnPlateau</code> is not inherited from <code>_LRScheduler</code> this function is not defined. Any equivalent way with <code>get_last_lr()</code> with this scheduler?</p>
<p>Thanks a lot!</p>

ptrblck: <p>You could use the internal <code>scheduler._last_lr</code> attribute, the <code>scheduler.state_dict()</code> or alternatively you could check the learning rate in the optimizer via <code>optimizer.param_groups[0]['lr']</code>.<br>
Note that the first two approaches would only work after the first <code>scheduler.step()</code> call.</p>

Cognac: <p>Thank you so much! Your response is very helpful as always.</p>
Missing values in a train sequence during training LSTM

farhad-bat: <p>Hello,</p>
<p>I have work with LSTM in PyTorch, but I have faced a new challenge.</p>
<p>Assume you have some sequences during training that each of contains 10 time step like this:<br>
a0, a1, a2, a3, a4, a5, a6, a7, a8, a9</p>
<p>in this form we easily give these sequence to LSTM, and we train it.</p>
<p>Now my problem is this. In some of sequences, some time steps are not available. For example, above sequece can be like this:<br>
a0, a1, a2, a3,  , a5, a6,  , a8, a9</p>
<p>In this form, for this sequence, a4 and a7 are not available during training.</p>
<p>How should I handle this?</p>

vdw: <p>It depends on your exact data and task.</p>
<p>In text processing one often has to deal with “missing” words, i.e., words that are not available in the index (e.g., rare words, typos, named entities). Such missing words are generally represented by a special word or token. For example, “the loudd noise woke me up” becomes “the  noise woke me up”. Maybe such a special “not available” token/value works for you as well.</p>
<p>Alternatively, is it a problem if the sequence is simply shorter. RNNs can handle sequences of different lengths.</p>
DistributedDataParallel() Hanfing

gnadaf: <p>I have used DDP in my Transformer model, but when I execute, init_process_group is hanging.</p>
<p>command used: python -m torch.distributed.launch --nnodes=1 --node_rank=1 --nproc_per_node=1 --use_env standard.py</p>
<p>With the above command, my goal is to run my model on a single node with a single GPU.<br>
the system has 8 GPUs, but I would like to use a single GPU, just to make DDP API is working</p>

ptrblck: <p>Was DDP working on this machine before and are you able to use e.g. all 8 GPUs or are all calls hanging?</p>

gnadaf: <p><a class="mention" href="/u/ptrblck">@ptrblck</a><br>
Issue was with launching processes, now resolved.</p>
<p>Thanks</p>
How does backward work for EmbeddingBag?

iaagi: <p>Let’s assume we have</p>
<pre><code class="lang-auto">params = torch.cuda.FloatTensor(16384, 256).cuda()
table = nn.EmbeddingBag.from_pretrained(params, sparse=True)
table.requires_grad_(True)
idx = torch.randint(low=0, high=16384,size=(8192, 64), dtype=torch.int64).cuda()
ugrad = torch.cuda.FloatTensor(8192, 256)
tmp = table(idx)
tmp.backward(ugrad)
</code></pre>
<p>I assume calling the backward should behave as follows:</p>
<ol>
<li>Read the upstream gradient (8192 x 256 x 4 (Bytes) = 8.3 MB read.)</li>
<li>Write the corresponding gradient value for each element of embedding bag that is looked up. Hence,  8192 x 64 x 256 x 4 (Bytes) = 536 MB write.</li>
</ol>
<p>However, looking at the behavior of the function in Nsight suggests that my understanding is not accurate. In the profiler running the backward launches 2 major kernels:</p>
<ol>
<li>
<code>indexSelectLargeIndex</code> that reads 12 MB and writes 516 MB - similar to what I expect.</li>
<li>
<code>unrolled_elementwise_kernel</code> that reads 539 MB and writes 539 MB - I’m confused what this kernel does.</li>
</ol>
<p>I would appreciate it if someone could shed some light on how backward on <code>embedding_bag</code> operates and where these numbers come from.</p>
<p>Thank you!</p>

albanD: <p>Hi,</p>
<p>I am not a specialist of EmbeddingBag but from what I remember, the default will perform averaging on common entries. So that second kernel would correspond to the backward of that averaging no?</p>

iaagi: <p>Thank you <a class="mention" href="/u/alband">@albanD</a> for your response.<br>
I think the first kernel corresponds to the averaging. That’s why it reads a rather small amount of data (upstream gradients) and writes a large amount of data (gradient values w.r.t to the all of the vectors whose average was computed).</p>
Format of weight parameters in KLDivLoss

mromaniuk: <p>KLDivLoss can take a weight parameter but the docs don’t specify how it should be formatted. What should the format be?</p>

apaszke: <p>It should be a 1D tensor having as many elements as you have classes. We’ll have to add that to the docs, thanks for reporting that!</p>

nofreewill: <p>Can it? I can see nothing weight related in the docs, nor in the source.</p>
GPU tensor access time overhead after resnet feature extraction

Markos-NTUA: <p>Hello everyone,<br>
while using pytorch’s <em>fasterrcnn_resnet50_fpn</em> I noticed that <strong>after</strong> passing a list of images from resnet’s backbone there is a time interval (e.g. for a batch of 8 images its ~0.22 sec) where any following tensor gpu operation will have to wait in order to be completed. All tensors and models are on the GPU. Here is code for reproduction:</p>
<pre><code class="lang-auto">from torchvision.models.detection import fasterrcnn_resnet50_fpn
import torch
import time

_backbone = fasterrcnn_resnet50_fpn(pretrained=True).cuda()
for name, param in _backbone.named_parameters():
    if name.startswith('backbone') or name.startswith('roi_heads.box_head'):
        param.requires_grad = False
backbone = _backbone.backbone
transform = _backbone.transform
images = [torch.rand(3, 768, 1024) for _ in range(8)]
foo = torch.tensor([0, 1, 2, 3]).cuda()

images, _ = transform(images, None)

base_features = backbone(images.tensors.cuda());
t = time.time()
# torch.max(foo) # max has no problem
torch.unique(foo) #unique has
print(time.time()-t)

t = time.time()
torch.unique(foo)
print(time.time()-t)
</code></pre>
<p>The output of this code for the setup described below is:</p>
<p>0.22019076347351074<br>
0.0008745193481445312</p>
<p>Setup:<br>
python: 3.7<br>
torch: 1.7.0<br>
torchvision: 0.8.1<br>
cudatoolkit: 10.1<br>
GPU: nvidia 20180Ti<br>
Ubuntu: 18.4.4 LTS</p>
<p>I also tried on 1080Ti and TitanX and with previous python/pytorch/torchvision releases having almost same results.</p>
<p>Thanks in advance!</p>

albanD: <p>Hi,</p>
<p>This happens because the whole CUDA API is actually asynchronous and only blocks when the work queue on the GPU is full and you have to wait for it to process or because you want some data on the cpu side.<br>
You can use <code>torch.cuda.synchronize()</code> to force synchronization of the GPU so that it waits on the resnet computations to finish.</p>

Markos-NTUA: <p>Thanks a lot for your answer!<br>
I thought that passing e.g. 8 images through resnet would not be so much slower than passing just 1, since they are processed in parallel,  but this does not seem to be the case. Do you have any idea how could I avoid this overhead?</p>

albanD: <p>This is not really an “overhead”, it is the time the GPU takes to actually perform the compute <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>