akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Fusion schema using PyTorch framework

username: Nicolo_Bogliolo
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/a/5a692ae089e56bfb222fc2f536fa25ac96771c67.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/5a692ae089e56bfb222fc2f536fa25ac96771c67" title="photo6003592328612262219"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/a/5a692ae089e56bfb222fc2f536fa25ac96771c67_2_690x379.jpeg" alt="photo6003592328612262219" data-base62-sha1="cTOi8VG7mqnLl0t5TEqXhaLcigL" width="690" height="379" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/a/5a692ae089e56bfb222fc2f536fa25ac96771c67_2_690x379.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/5/a/5a692ae089e56bfb222fc2f536fa25ac96771c67_2_1035x568.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/a/5a692ae089e56bfb222fc2f536fa25ac96771c67.jpeg 2x" data-dominant-color="ECEDEB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">photo6003592328612262219</span><span class="informations">1280×704 94.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<strong>Descpription:</strong><br>
All the light blue layers concer a RGB detector that I have alredy implemented in PyTorch framework using a Mask RCNN framework with a ResNet50 CNN backbone.</p>
<p>All the light green layers concern a thermal detector that, again, I have alredy implemented in PyTorch framework with the same Mask RCNN framework previously mentioned.</p>
<p>the RGB and thermal images are pixel-to-pixel well aligned. it is possible to see the RGB-Thermal pairs as 4-channel images.</p>
<p>The figure represent a slightly different scenario since they have used a Faster RCNN framework with a VGG16 CNN backbone but the concept is the same since I am just interested in bbox regression task and not instance segmentation one.</p>
<p><em>Is it possible to implement in PyTorch this kind of fusion schema?</em> The crucial point is to use just one RPN module that is able to exploit the feature map of both the detector as explained in the figure and treat these 3 different sub-networks as single one.</p>
<p>Then, when the 2 pseudo-independent detectors(that share the RPN module concatenating the feature map) get out, at the end, the class score and bbox prediction for each candidate region for both the color and thermal detector another gated fusion is wanted according to the illumination weight that comes from another CNN as it is possible to see from the figure.</p>
<p><strong>Inputs:</strong><br>
RGB-thermal pixel-to-pixel alligned that come in in two different detectors, the RGB and thermal one.</p>
<p><strong>Outputs:</strong><br>
A class score and bbox prediction pair for each candidate regions. each of these final pairs is the result of fuse each class score and bbox prediction that come out from the 2 detectors according to the weight_c and weight_t that come put from a third CNN.</p>
CUDA out of memory when back-propagating loss

username: Bo_Ni
cooked: <p>Hi,</p>
<p>I have a customized GCN-Based Network and a pretty large graph (40000 x 40000). Basically,  There is no problem with forwarding passing (i.e the GPU memory is enough); but cuda ran out of memory when loss.backward() is executed. The error message is as following</p>
<p><code>Traceback (most recent call last):   File "main.py", line 142, in &lt;module&gt;     main()   File "main.py", line 125, in main     r_gcn.train()   File "/afs/crc.nd.edu/group/dmsquare/vol3/bni/user/e2e/model/R_GCN.py", line 167, in train     loss.backward()	   File "/afs/lib/python3.8/site-packages/torch/tensor.py", line 198, in backward     torch.autograd.backward(self, gradient, retain_graph, create_graph)   File "/afs/lib/python3.8/site-packages/torch/autograd/__init__.py", line 98, in backward     Variable._execution_engine.run_backward( RuntimeError: CUDA out of memory. Tried to allocate 7.72 GiB (GPU 0; 31.75 GiB total capacity; 27.83 GiB already allocated; 2.90 GiB free; 27.86 GiB reserved in total by PyTorch) (malloc at /opt/conda/conda-bld/pytorch_1591914858187/work/c10/cuda/CUDACachingAllocator.cpp:289) frame #0: c10::Error::Error(c10::SourceLocation, std::string const&amp;) + 0x4e (0x2b9c1d781b5e in /afs/lib/python3.8/site-packages/torch/lib/libc10.so) frame #1: &lt;unknown function&gt; + 0x1f39d (0x2b9c1d54339d in /afs/lib/python3.8/site-packages/torch/lib/libc10_cuda.so) frame #2: &lt;unknown function&gt; + 0x2058e (0x2b9c1d54458e in /afs/lib/python3.8/site-packages/torch/lib/libc10_cuda.so) frame #3: at::native::empty_cuda(c10::ArrayRef&lt;long&gt;, c10::TensorOptions const&amp;, c10::optional&lt;c10::MemoryFormat&gt;) + 0x291 (0x2b9bf50eb401 in /afs/lib/python3.8/site-packages/torch/lib/libtorch_cuda.so) frame #4: &lt;unknown function&gt; + 0xdc454b (0x2b9bf339e54b in /afs/lib/python3.8/site-packages/torch/lib/libtorch_cuda.so) frame #5: &lt;unknown function&gt; + 0xe0de37 (0x2b9bf33e7e37 in /afs/lib/python3.8/site-packages/torch/lib/libtorch_cuda.so) ........ frame #24: &lt;unknown function&gt; + 0x7ea5 (0x2b9bca00aea5 in /lib64/libpthread.so.0) frame #25: clone + 0x6d (0x2b9bca31d8cd in /lib64/libc.so.6) </code></p>
<p>Since the error message requests roughly 7G memory on GPU, which is about the size of the graph, and that I need the gradient of graph-entries. I suppose it is because the backward function needs another big matrix to store the computed gradient. However, I am not sure about what the other error outputs mean.</p>
<p>Thank you for your help!</p>

username: ptrblck
cooked: <p>Your assumption sounds reasonable, since the first <code>backward</code> call will use additional memory to store the gradients.</p>
<p>You could reduce the batch size and rerun it or alternatively you could also trade compute for memory via <code>torch.utils.checkpoint</code>.</p>
<p>The <code>malloc</code> stack trace points again to the <code>CUDACachingAllocator</code>, so you can ignore this message, as the <code>RuntimeError</code> was already raised.</p>

username: Bo_Ni
cooked: <p>Thank you! This is very helpful.</p>
How to use torch.nn.GroupNorm function

username: skerlet_flandorle
cooked: <p>How to use torch.nn.GroupNorm function<br>
I don’t understand.<br>
Please tell me who understands</p>

username: ptrblck
cooked: <p>The <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.GroupNorm"><code>GroupNorm</code> docs</a> contain an example including comments for each setup.</p>
<p>Do you have a question for its usage in PyTorch or the method in general?</p>

username: skerlet_flandorle
cooked: <p>m = nn.GroupNorm(1, 6)<br>
output = m(input)</p>
<p>When do you use output = m(input)?<br>
In one sample, x=Relu(GroupNorm(CNN(x)))<br>
However, I do not know whether to do it for each CNN layer or only for the last layer.</p>

username: skerlet_flandorle
cooked: <p>ok It worked properly</p>
Tensorboard visualize graph for a model using autograd functions

username: yifita
cooked: <p>Hi, I’m trying to visualize the graph for the following model:</p>
<pre><code class="lang-python">class Model(torch.nn.Module):
    def __init__(self, ):
        super().__init__()
        self.decoder = Decoder(c_dim=0, dropout_prob=0.2, weight_norm=True, norm_layers=[0,1,2,3,4,5,6,7])

    def forward(self, p):
        # p.requires_grad_(True)

        # sdf = self.decoder(p)
        # normals = autograd.grad(sdf, p, torch.ones_like(sdf), create_graph=True, retain_graph=True)[0]
        normals = torch.autograd.functional.jacobian(self.decoder, p, create_graph=True, strict=False)
        return torch.mean((torch.norm(normals, dim=-1) - 1)**2)
</code></pre>
<p>I try to use tensorboard to visualize as follows:</p>
<pre><code class="lang-auto">writer = SummaryWriter(os.path.join(out_dir))

# create model
model = Model()

# data (1,1,3) point
data = torch.rand((1,1,3))

writer.add_graph(model, data, verbose=True)
</code></pre>
<p>However I keep getting the following error</p>
<pre><code class="lang-bash">RuntimeError: Cannot insert a Tensor that requires grad as a constant. Consider making it a parameter or input, or detaching the gradient
Tensor:
(1,.,.) = 
  0.6029  1.4451  2.1129
[ torch.FloatTensor{1,1,3} ]
</code></pre>
<p>BTW: the code runs if I don’t attempt to add the graph.</p>
Why EfficientNet same model return different predictions

username: sharkdeng
cooked: <p>Hello, when I run the following code I found although the mode and input are same,the  predictions are different.</p>
<pre><code class="lang-auto">!pip install efficientnet_pytorch -q


import torch 
import torch.nn as nn
import efficientnet_pytorch as efn

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
device

model = efn.EfficientNet.from_name('efficientnet-b0')
model = model.to(device)

img = torch.ones((2, 3, 680, 680))*0.5
img = img.to(device)
preds1 = model(img)
preds2 = model(img)
preds3 = model(img)

print(preds1[0][0])
print(preds2[0][0])
print(preds3[0][0])

del model, img, preds1, preds2, preds3
</code></pre>
<p>These are outputs.</p>
<pre><code class="lang-auto">tensor(0.2599, grad_fn=&lt;SelectBackward&gt;)
tensor(0.1364, grad_fn=&lt;SelectBackward&gt;)
tensor(0.1263, grad_fn=&lt;SelectBackward&gt;)
</code></pre>
<p>Do you know why this happens? Thank you for clarificaton.</p>

username: sharkdeng
cooked: <p>Wow, solved! I checked the Efficientnet-pytorch source code and suddendly found I should turn the model to eval() mode.</p>
<p>Now the predictions are same!!</p>
<pre><code class="lang-auto">model = efn.EfficientNet.from_name('efficientnet-b0')
model = model.to(device)
_ = model.eval()  ## hey, look here!!

img = torch.ones((2, 3, 680, 680))*0.5
img = img.to(device)
preds1 = model(img)
preds2 = model(img)
preds3 = model(img)

print(preds1[0][0])
print(preds2[0][0])
print(preds3[0][0])
</code></pre>
Select Tensor by Row

username: Jack_Rolph
cooked: <p>I have two tensors, one with data and one with some value I would like to threshold on, both along row <code>C</code>:</p>
<pre><code class="lang-auto">a = torch.Tensor(B,C,X,Y)
b = torch.Tensor(B,C)
</code></pre>
<p>I would like to find those rows of <code>C</code> in tensor <code>a</code>  that fulfill a criterion based on tensor <code>b</code>. For example:</p>
<pre><code class="lang-auto"># Indices of b where b is zero
idx = (b==0)
</code></pre>
<p>The end result should look like:</p>
<pre><code class="lang-auto">bar(b)
foo(a)
a = torch.Tensor(B,C-N,X,Y)
#N is number of rows that do not match condition. 
</code></pre>

username: ptrblck
cooked: <p>I’m not completely understand the use case.<br>
If you are using a threshold on <code>b</code>, you might only have a single <code>True</code> output in <code>idx</code>.<br>
Indexing <code>a</code> with this <code>idx</code> tensor would return an output of <code>[1, X, Y]</code>, not <code>[B, 1, X, Y]</code>.</p>
<p>Could you explain a bit, how the threshold should be applied to <code>b</code>?<br>
Currently it seems that you are setting the same values for all elements in dim0.</p>

username: Jack_Rolph
cooked: <p>Suppose I have a tensor, <code>a</code></p>
<pre><code class="lang-auto">a = torch.Tensor(B,C,X,Y)
</code></pre>
<p>I am trying to normalize the <code>Y</code> axis to its Z-score taken along statistics in <code>X</code>. However, some of the elements along the channel dimension, <code>C</code>, are degenerate.</p>
<p>This means I am only interested with those <code>C</code> elements that have a non-zero standard deviation along the axis <code>X</code>.</p>
<p>The way I achieved what I wanted was as follows:</p>
<pre><code class="lang-auto"># Find elements with a square-sum standard deviation along the X axis
temp = torch.norm(torch.std(a, axis=-2), dim=-1)
# Find the non-zero indices of this tensor
idx = temp.nonzero()
# Slice the tensor along the indices
a_sliced = a[idx[:,0], idx[:,1], :, :]
# Perform some calculations of a_sliced
a_sliced = foo(a_sliced)
# Slice in the results into a similarly sized tensor as the original tensor.
a_unsliced = torch.zeros_like(a)
a_unsliced[idx[:, 0], idx[:, 1], :, :] = a_unsliced
</code></pre>
Appear RuntimeError when loss.backward()

username: Derek
cooked: <p>During training a RNN-based model, there is a RuntimeError in loss.backward().</p>
<p>"<br>
…<br>
…<br>
File “/usr/local/lib/python2.7/dist-packages/torch/tensor.py”, line 102, in backward<br>
torch.autograd.backward(self, gradient, retain_graph, create_graph)<br>
File “/usr/local/lib/python2.7/dist-packages/torch/autograd/<strong>init</strong>.py”, line 90, in backward<br>
allow_unreachable=True)  # allow_unreachable flag<br>
RuntimeError: select(): index 10 out of range for tensor of size [10, 8, 12020] at dimension 0<br>
"</p>
<p>I wonder why such error appears at loss.backward().<br>
The data flow is convinced to be correct because of the right forward().</p>
<p>How to debug such an error?</p>

username: xelibrion
cooked: <p>To whoever experiences this error:</p>
<p>in my case it was causes by incorrect parameters to <code>rnn.pack_padded_sequence</code>, specifically <code>batch_first</code> wasn’t set to <code>True</code> as my code expected.</p>
Batch processing for custom forward

username: methi1999
cooked: <p>I have a model with only 3 tensors (say p1, p2 and p3) as trainable parameters. The forward function takes in 2 tensors as input, calculates a score and returns it. Given below is the code:</p>
<pre><code class="lang-auto">def forward(self, a, b):
        score = custom_func(a, b, self.p1, self.p2, self.p3)
        return torch.sigmoid(score)
</code></pre>
<p>I am able to backprop and update the parameters.<br>
However, I am currently calling <em>forward()</em> one-by-one for each example, accumulating the gradients using <em>loss.backward()</em> after each <em>forward()</em> and calling <em>optimizer.step()</em> every few examples.<br>
Is there a way to process them in parallel? Instead of passing a, b as arguments to the forward function, can I instead pass a list?<br>
I don’t know CUDA works internally but can I somehow spawn ‘separate CUDA threads’ for each of the examples and process them in parallel?</p>

username: albanD
cooked: <p>Hi,</p>
<p>The short term answer is that you can. You “just” need to modify <code>custom_func</code> to accept input Tensors that have an extra dimension that will be the batch.</p>
<p>We are working on adding constructs to do this automatically but this is not ready yet and will be only in the fall I think.</p>

username: methi1999
cooked: <p>I have a <em>for</em> loop inside <em>custom_func</em> which iterates over each element of the argument <strong>a</strong> along dimension 0 (number of iterations varies from example to example). So, if I insert the batch dimension, I’ll have to add another for loop outside this loop which iterates over the batch dimension, right? but that won’t give any performance benefit, will it?</p>
Why use nn.GRU AND loop over sequence in PyTorch tutorial?

username: rschaefer
cooked: <p>Hi there,</p>
<p>I am referring to the following tutorial: <a href="https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html" rel="nofollow noopener">https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html</a></p>
<p>In this tutorial, a machine translation model is built using nn.GRU. If I am right, nn.GRU can be used on the whole sequence at once, whereas nn.GRUCell is just one cell. In order to use nn.GRUCell you would need to loop over the sequence.</p>
<p>What is confusing me, is that in the tutorial they loop over the sequence AND use the nn.GRU.</p>
<p>Any help to solve this confusion is appreciated.</p>
<p>Best,<br>
Robin</p>

username: vdw
cooked: <p>I think that got asked before. I also don’t see why the encoding is done via a loop and and not giving the GRU layer the full sentence.</p>
<p>I assume one could in principle use a GRUCell. The only difference when it comes to feeding it step by step is that GRU supports multiple layers, but I don’t even know if this is meaningful without the full sequence.</p>
<p>I’m also pretty sure that the first version of the tutorial didn’t use a loop for the encoding since I also used the tutorial as a template back then.</p>

username: rschaefer
cooked: <p>Yes, I saw a topic wondering about the same thing, but it was not really answered, only confirmed, and as it was some time ago I thought I should ask again.</p>
<p>I implemented a model based on the tutorial, too,  and it converged pretty well. So from that perspective, as long as it is working I may stick to that implementation. I was wondering if it’s just about processing time in the end: if you feed only one step of the sequence into nn.GRU the internal loop is redundant and may need some extra time.</p>
Torch capsule-ru giving negative r2 score

username: SamratSengupta
cooked: <p>I am applying bigru-capsule network for stock market forecasting on dow jones industrial average data.R2 score in training itself is coming negative.</p>
<p>code and data files available <a href="https://github.com/SamratSengupta/torch-practice" rel="noopener nofollow ugc">https://github.com/SamratSengupta/torch-practice</a></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/e/ae6696303565051bff0d49d0be23166f598e910b.png" data-download-href="https://discuss.pytorch.org/uploads/default/ae6696303565051bff0d49d0be23166f598e910b" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/e/ae6696303565051bff0d49d0be23166f598e910b_2_690x388.png" alt="image" data-base62-sha1="oSORZk1dr1sQ8WnHnbkGoQ9j7Hd" width="690" height="388" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/e/ae6696303565051bff0d49d0be23166f598e910b_2_690x388.png, https://discuss.pytorch.org/uploads/default/optimized/3X/a/e/ae6696303565051bff0d49d0be23166f598e910b_2_1035x582.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/a/e/ae6696303565051bff0d49d0be23166f598e910b_2_1380x776.png 2x" data-dominant-color="F0ECED"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1920×1080 303 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>really clueless about it…<br>
I am referencing  ‘BinZhou’ s code</p>

username: ptrblck
cooked: <p>As the warning explains, your output and target do not have the same shape and thus will be broadcasted.<br>
Make sure to pass both tensors as <code>[batch_size, 1]</code> to the criterion by unsqueezing the target via:</p>
<pre><code class="lang-python">target = target.unsqueeze(1)
</code></pre>
<p>This might also fix the negative accuracy (although I don’t know, how you are calculating it).</p>
<p>PS: You can post code snippets by wrapping them into three backticks ```, which makes debugging easier. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
Instance norm implement by basic operations has different result comparing to torch.nn.InstanceNorm2d

username: zhousj
cooked: <p>i implement instance norm by pytorch basic operations. but the result is different from torch.nn.InstanceNorm2d. Can anyone help me out? Below is my code:</p>
<p>##########################<br>
import torch<br>
import numpy as np<br>
x = torch.rand((8, 16, 32, 32))<br>
a = torch.nn.InstanceNorm2d(256)<br>
a.eval()<br>
with torch.no_grad():<br>
b = a(x)<br>
x_mean = torch.mean(x, axis=(2,3), keepdims=True)<br>
x_var = torch.var(x, axis=(2,3), keepdims=True)<br>
x_norm = (x - x_mean) / torch.sqrt(x_var + 1e-5)<br>
b_numpy = b.numpy()<br>
x_norm_numpy = x_norm.numpy()<br>
# check if b_numpy and x_norm_numpy close to the torlerance of 1e-3<br>
print(np.allclose(b_numpy, x_norm_numpy, atol=1e-3))<br>
# check if b_numpy and x_norm_numpy close to the torlerance of 1e-3<br>
print(np.allclose(b_numpy, x_norm_numpy, atol=1e-4))</p>
<p>##########################<br>
result:<br>
True<br>
False<br>
##########################</p>
<p>so, the result shows that when precision comes to 1e-4, they are different. i don’t know why. can anyone help me to get a more close result to torch.nn.InstanceNorm2d? Thanks in advance!</p>
<p>BTW. The reason why I do not use formula gamma * x_normalized_numpy + beta in the paper is i find that when the first initialization of torch.nn.InstanceNorm2d, all gamma is initialized to [1.0, 1.0, 1.0, …] and all beta is initialized to [0.0, 0.0, 0.0,…]. So, under this condition, x_normalized_numpy = gamma * x_normalized_numpy + beta</p>

username: ptrblck
cooked: <p>I think <code>nn.InstanceNorm2d</code> might use the biased variance, so changing the variance calculation to:</p>
<pre><code class="lang-python">x_var = torch.var(x, unbiased=False, axis=(2,3), keepdims=True)
</code></pre>
<p>should work (I haven’t checked the source code of <code>nn.InstanceNorm2d</code>, but batchnorm layers use the biased variance, so I just tested your code quickly <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"> ).</p>

username: zhousj
cooked: <p>thank you so much <a class="mention" href="/u/ptrblck">@ptrblck</a> . you are right. this can reduce the difference to the tolerance of  1e-6. I think this result is acceptable.</p>
Multithreaded Matrix Multiplication speed

username: Ausstein
cooked: <p>My code uses a lot of batched matrix multiplications of small matricies, but I get no speedup for using more than one thread on a 4 core machine (also tested on a 6 core and a 64 core cluster node)</p>
<p>So I ran a test and found the following (only on the 4 core)</p>
<pre><code>&gt;import timeit
&gt;print(timeit.timeit("a*b",setup="import torch; torch.set_num_threads(1); torch.set_num_interop_threads(1); a=torch.normal(mean=torch.zeros((1000000,2,2)),std=1.0);  b=torch.normal(mean=torch.zeros((1000000,2,2)),std=1.0)",number=10000))
75.0120605
</code></pre>
<p>cpu usage was at roughly 23% for python</p>
<p>then I ran the following:</p>
<pre><code>&gt;import timeit
&gt;print(timeit.timeit("a*b",setup="import torch; torch.set_num_threads(4); torch.set_num_interop_threads(4); a=torch.normal(mean=torch.zeros((1000000,2,2)),std=1.0);  b=torch.normal(mean=torch.zeros((1000000,2,2)),std=1.0)",number=10000))
61.3748576
</code></pre>
<p>cpu usage was &gt;85% for python and at 100% for the whole system.</p>
<p>I reproduced this with similar results a couple of times and I also tried using different values of threads and interop threads without any improvement.</p>
<p>While this is at least some speedup( which I do not get for my code), this is not what I expected for using 4 cores instead of one.</p>
<p>Is this expected?</p>
<p>I need my code to scale, so that it can use the 64 cores of the cluster node efficiently.<br>
There are some loops that I could manually parallelize, but I would prefer if there was an effiecient way to just run the batched matrix multiplications in parallel.</p>
Run two operations concurrently on one GPU

username: yzz
cooked: <p>Hi all,</p>
<p>I am trying to run two operations concurrently on a single GPU. However, the operations are done sequentially.</p>
<p>I use the sample code shown below. However, my GPU profiler shows that torch.mm() on A and B are computed sequentially on GPU (see attached the picture at the bottom). Is this due to the kernel size? I wonder if anyone might have some clues about what might be the reason. BTW: I am using Volta GPU.</p>
<p>Thanks in advance.</p>
<pre><code class="lang-auto">device = torch.device("cuda" if torch.cuda.is_available() else "cpu") 
s1 = torch.cuda.Stream()
s2 = torch.cuda.Stream()
A = torch.rand(10000, 10000, device=device)
B = torch.rand(10000, 10000, device=device)
torch.cuda.synchronize()
for i in range(10):
  with torch.cuda.stream(s1):
    C = torch.mm(A, A)
  with torch.cuda.stream(s2):
    D = torch.mm(B, B)
torch.cuda.synchronize()
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/e/bee5503eecb14200f493c899819d9d2189765777.png" data-download-href="https://discuss.pytorch.org/uploads/default/bee5503eecb14200f493c899819d9d2189765777" title="Screen Shot 2020-07-03 at 12.53.33 AM"><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/e/bee5503eecb14200f493c899819d9d2189765777.png" alt="Screen Shot 2020-07-03 at 12.53.33 AM" data-base62-sha1="reK1jwq8C3yPB4Yt74MJIzs0cCP" width="690" height="41" data-dominant-color="BEC8E3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2020-07-03 at 12.53.33 AM</span><span class="informations">1259×76 9.43 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>You might use the complete GPU resources, which would disallow overlapping kernel execution.<br>
Try to reduce the workload and check, if you see the overlapping execution.</p>
2-D Tensor: Deleting different columns in different rows

username: Ayush_Mehta
cooked: <p>I have a 2-D Tensor A of shape = [ BS, n*c ]<br>
I need a 2-D Tensor A_out of shape = [ BS, (n-1)*c ]</p>
<p>Essentially, I want to delete c values from each row. These c values will come from different columns for each row.<br>
These specific columns will be decided by the below tensor L:<br>
Shape- [ BS, 1 ] and Values can be integers from ( 0, n-1 )</p>
<p>I want to remove the values between indices c * L[i] and c * ( L[i] + 1 ) for row i</p>
<p>The best I could come up with is as follows. If anyone can help with removing the for loop in this, it’ll be really appreciated:</p>
<p>A_out = torch.zeros( BS, (n-1) * c)</p>
<p>for i in range( BS ):<br>
A_out[i] = torch.cat( [ A[ i , :c * L[i] ], A[ i , c * ( L[i]+1 ):]  ]  )</p>
<p>(‘gather’ seemed like the appropriate option at first, but creating the 2-D tensor of indexes to gather using L also seemed equally costly i.e. for loop needed)</p>
Torch.load()return False

username: KyDlt
cooked: <p>When I test <code>torch.cuda.available()</code>, it returns True.<br>
But when I use <code>torch.load()</code>, it tells me <code>torch.cuda.available()</code> is False:</p>
<p>File “D:\Anaconda3\envs\charm\lib\site-packages\torch\serialization.py”, line 115, in validate_cuda_device<br>
raise RuntimeError('Attempting to deserialize object on a CUDA ’<br>
RuntimeError: Attempting to deserialize object on a CUDA device but torch.cuda.is_available() is False. If you are running on a CPU-only machine, please use torch.load with map_location=torch.device(‘cpu’) to map your storages to the CPU.</p>
<p>Why?</p>

username: ptrblck
cooked: <p>Are you trying to load the model directly or the <code>state_dict</code>?<br>
Could you push the model or <code>state_dict</code> to the CPU before storing it, load it in another script, and try to push it to the device later?<br>
If this is working, the <code>torch.load</code> method might create the issue and would help us to debug it.</p>
RuntimeError: size mismatch, m1: [10 x 3584], m2: [25088 x 10]

username: aatiibutt
cooked: <p>I am trying to train the following model, but size mismatch error occurs in the classifier chunk of code.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/4/94e7e4230ad7bd2eab9455b2063fe78dad575ef3.png" data-download-href="https://discuss.pytorch.org/uploads/default/94e7e4230ad7bd2eab9455b2063fe78dad575ef3" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/4/94e7e4230ad7bd2eab9455b2063fe78dad575ef3.png" alt="image" data-base62-sha1="lfhuo7ebqURuTCC9YHusHKNPZDl" width="520" height="500" data-dominant-color="EFEFEF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">561×539 18.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/d/7db6e2547e041096dd65d7334f8d977d0dde8e3e.png" data-download-href="https://discuss.pytorch.org/uploads/default/7db6e2547e041096dd65d7334f8d977d0dde8e3e" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/d/7db6e2547e041096dd65d7334f8d977d0dde8e3e.png" alt="image" data-base62-sha1="hW7vH0btLL2OlsJRcGqgOR6JxAG" width="690" height="323" data-dominant-color="F7F8F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1093×512 30.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Any help?</p>

username: Dipam_Vasani
cooked: <p>Linear layer consists of matrix multiplication. For matrix multiplication, the number of columns of matrix 1 should be equal to number of rows of matrix 2. In your case the first one has [10 * 3584] while the second one has [25088 * 10] hence you get a mismatch. The 3584 and 25088 should match. I would print the output of avgpool and set the number of nodes in the linear layer according to that.</p>

username: aatiibutt
cooked: <p>Thanks Dear <a class="mention" href="/u/dipam_vasani">@Dipam_Vasani</a>! It worked <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
'torch.FloatTensor' object has no attribute 'requires_grad'

username: Yanglinjun1
cooked: <p>Hello, I am a beginner with PyTorch and just running some basic example code in my jupiter notebook. When I was trying to un the code:<br>
import torch<br>
a = torch.ones(2,2,requires_grad=True);</p>
<p>It reported the error that: TypeError: torch.ones received an invalid combination of arguments - got (int, int, requires_grad=bool), but expected one of:</p>
<ul>
<li>(int … size)<br>
didn’t match because some of the keywords were incorrect: requires_grad</li>
<li>(torch.Size size)</li>
</ul>
<p>I know this is not supposed to happen and there might be some version related issue. I am using conda python 3 and installed the PyTorch following the page suggestion. Thanks in advance.</p>

username: ptrblck
cooked: <p>Try to wrap the size: <code>torch.ones((2, 2), requires_grad=True)</code>.</p>

username: Yanglinjun1
cooked: <p>Hi! Thank you for the response!<br>
But it will report that<br>
torch.ones received an invalid combination of arguments - got (tuple, requires_grad=bool), but expected one of</p>
<p>I am wondering I am using a different PyTorch:(</p>

username: ptrblck
cooked: <p>Which version are you using?<br>
<code>0.4.0</code> was released just yesterday merging <code>Variable</code>s and <code>Tensor</code>s.<br>
Did you update?</p>

username: Yanglinjun1
cooked: <p>Yeah I have updated and the issue was solved. Thanks!</p>

username: austinmw
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>, I’m using  pytorch 0.4.0 (py36_cuda9.0.176_cudnn7.1.2_1  [cuda90]) trying to run the NN tutorial. I ran it from home a week ago on my laptop no problem, but having difficulties running it on a ubuntu machine with cuda. Has it been updated for the newest version of pytorch, or was it just not meant to support pytorch with cuda?</p>
<p>I first got an error on line: <code>outputs = net(inputs)</code> : <code>TypeError: argument 0 is not a Variable</code><br>
Then I wrapped inputs in <code>Variable()</code> and got an error on the next line <code>loss = criterion(outputs, labels)</code> : <code>AttributeError: 'torch.LongTensor' object has no attribute 'requires_grad'</code> Any idea how to fix?</p>

username: ptrblck
cooked: <p>The first error indicates an older version. Could you check it with <code>print(torch.__version__)</code>?</p>

username: austinmw
cooked: <p>Ah weird. Conda says I have pytorch <code>pytorch: 0.4.0-py36_cuda9.0.176_cudnn7.1.2_1</code> (just downloaded today), but although I’m using the Anaconda kernel in jupyter it prints out version 0.3.1.</p>

username: ptrblck
cooked: <p>Try to uninstall it using conda and pip.<br>
Just call <code>conda uninstall pytorch</code> and <code>pip uninstall torch</code> and reinstall it afterwards.<br>
Maybe you have different versions installed.</p>

username: austinmw
cooked: <p>Even though I just dled pytorch I tried  <code>conda update pytorch</code> and it showed <code>pytorch: 0.4.0-py36_cuda9.0.176_cudnn7.1.2_1 pytorch [cuda90] --&gt; 0.4.0-py36hdf912b8_0</code> and now it prints the correct version in jupyter. strange. Thanks for identifying the problem!</p>
How to print gradient for each layer weight/bias in network?

username: cruzas
cooked: <p>Hello there!</p>
<p>I am having an issue printing the computed gradients for the following network:</p>
<pre><code class="lang-auto">Policy()
{
   affine1 = register_module("affine1", torch::nn::Linear(10, 20));
   affine2 = register_module("affine2", torch::nn::Linear(20, 5));
}
</code></pre>
<p>I am trying to print the gradients as follows:</p>
<pre><code class="lang-auto">std::cout &lt;&lt; "Gradient before: " &lt;&lt; policy-&gt;affine1-&gt;weight.data().grad() &lt;&lt; std::endl;
loss.backward();
std::cout &lt;&lt; "Gradient after: " &lt;&lt; policy-&gt;affine1-&gt;weight.data().grad() &lt;&lt; std::endl;
</code></pre>
<p>However, the output I get is the following:</p>
<blockquote>
<p>Gradient before: [ Tensor (undefined) ]<br>
Gradient after: [ Tensor (undefined) ]</p>
</blockquote>
<p>Does anyone know why this happens?<br>
What would be the correct way to print the data I am looking to visualize?</p>
<p>Thank you very much.</p>

username: Keunhoi_An
cooked: <p>Did you solve your problem? I met Tensor (undefined) problem also while I push_back TensorList to a vector and print the element of vector (which is input TensorList).</p>

username: cruzas
cooked: <p>Hi <a class="mention" href="/u/keunhoi_an">@Keunhoi_An</a>,</p>
<p>No, I unfortunately did not solve the problem <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"></p>

username: yyn19951228
cooked: <p>HI <a class="mention" href="/u/keunhoi_an">@Keunhoi_An</a>, are you using the <code>ParameterList</code></p>

username: Keunhoi_An
cooked: <p>No, I didn’t use <code>ParameterList</code>. I just store my <code>bbox_xywh</code> and <code>score</code> to <code>torch::Tensor</code> variables and combined them to a single <code>TensorList</code>. This is my case. May I ask you what is your case struggling with <code>TensorList</code> ?</p>
How to set the weight of background class in cross_entropy?

username: Michael-J98
cooked: <p>Hi,<br>
I noticed that the fasterrcnn loss uses cross_entropy loss without weight, in the meantime, the default value of box_positive_fraction in fasterrcnn is 0.25.<br>
Does it mean, after training, the model would like to predict a background rather than a foreground target?<br>
I get an unbalanced issue with my own dataset, so I want to set weights for each class.<br>
How can I get a reasonable weight value for the background class?<br>
Thanks for any advice!</p>
How to load latent dimension of auto-encoder data in data loader

username: _joker
cooked: <p>I created a function that takes input as a “list” and converts it into a “dataloader” input.</p>
<pre><code class="lang-auto">def load_data(list_):
    if len(list_[0])==2:
        img, lab = [], []
        for i in list_:
            img.append(i[0]), lab.append(i[1])
        xs = torch.stack(img)
        xs = xs.squeeze(1)
        ys = torch.Tensor(lab)
        dataset = TensorDataset(xs, ys)
        del img, lab
    elif len(list_[0])==3:
        img, lab, flag = [], [], []
        for i in list_:
            img.append(i[0]), lab.append(i[1]), flag.append(i[2])
        xs = torch.stack(img)
        xs = xs.squeeze(1)
        ys = torch.Tensor(lab)
        flag = torch.Tensor(flag)
        dataset = TensorDataset(xs, ys, flag)
        del img, lab, flag
    return dataset

dataset = load_data(list_) # Coverting the list into dataset
train_data, val_data = torch.utils.data.random_split(dataset, [int(0.90*len(dataset)), int(0.10*len(dataset)+1)] )
train_loader = DataLoader(train_data, batch_size=BATCH_SIZE, shuffle=True)
val_loader = DataLoader(val_data, batch_size=BATCH_SIZE, shuffle=True)
</code></pre>
<p>The error I am getting is because of different sizes which I don’t understand correctly.</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "newpipeline.py", line 242, in &lt;module&gt;
    dataset = load_data(list_) # Coverting the list into dataset
  File "newpipeline.py", line 26, in load_data
    xs = torch.stack(img)
RuntimeError: invalid argument 0: Sizes of tensors must match except in dimension 0. Got 64 and 28 in dimension 1 at ../aten/src/TH/generic/THTensor.cpp:689
</code></pre>
<p>The data I dump as a list from the latent dimension of an autoencoder is of the shape - [Batch-size, 256, 2, 2]</p>

username: Nikronic
cooked: <p>Hi,<br>
Error is happening here, which means <code>img</code>s that you are stacking do not have same shape. Based on error, for some images, you have <code>64</code> and another <code>28</code> for height.</p>
<aside class="quote no-group" data-username="_joker" data-post="1" data-topic="87774">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/_joker/48/37415_2.png" class="avatar"> _joker:</div>
<blockquote>
<p><code>xs = torch.stack(img)</code></p>
</blockquote>
</aside>
<p>Bests</p>
Merge classes before or after training?

username: FaultyBagnose
cooked: <p>Sorry if this more of a general methodology question than PyTorch.<br>
I’m training a mushroom classifier, and some classes might be merged for various reasons (classes are small, labels are often switched, the end user will consider them similar, etc). Is it best to create the merged classes before training or train with the original classes and “merge” afterwards?</p>
<p>If I merge afterwards , should I add up the probability estimates after soft-maxing or combine (add) the neural net output before applying softmax?</p>
<p>If merging afterwards usually works equally well, it has the advantage that one can experiment with different mergers without retraining.</p>

username: Nikronic
cooked: <p>Hi,</p>
<p>Actually, using more classes usually needs more training and tricks to help network to learn. So, first criterion is that if you can make sure you can still achieve desirable results for not merged case.<br>
Now let’s assume you have a pretrained model with all mushroom classes but you are interested in common ones and as you have mentioned you want to merge classes with small contribution. In this case, I prefer merging probs after softmax as it is more intuitive. In both cases it should work just fine as sotfmax works more like a normalization term on logits.</p>
<p>In my case, I used a pretrained scene recognition network, but I was not interested in different type of chairs or something like that. So, I just got final probabilities and added up probs.</p>
<p>Bests</p>

username: Nikronic
cooked: <p><a class="mention" href="/u/faultybagnose">@FaultyBagnose</a><br>
I have asked this question at the time of using idea similar to yours.</p>
<aside class="quote quote-modified" data-post="1" data-topic="47888">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nikronic/48/71873_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/merging-classes-of-the-output-of-the-semantic-segmentation-module/47888">Merging classes of the output of the semantic segmentation module</a> <a class="badge-category__wrapper " href="/c/vision/5"><span data-category-id="5" style="--category-badge-color: #AB9364; --category-badge-text-color: #FFFFFF;" data-drop-close="true" class="badge-category " title="Topics related to either pytorch/vision or vision research related topics"><span class="badge-category__name">vision</span></span></a>
  </div>
  <blockquote>
    Hi all, 
My question is about the image segmentation task. 
I have a tensor with the size of (batch_size, 150, height, width) and the second number(150) corresponds to the number of classes. 
Now I want to merge those classes into 25 classes by summing their probabilities. 
First of all, is this procedure correct? It seems it should be, for example I want to merge river and lake into one class. 
Note: I have the source and target indexes to be merged. For example: 
target=[1, 2]
source= [[3,4],
…
  </blockquote>
</aside>
<p>
I hope it helps.</p>
Data Loss when converting images from np.array to PIL Image

username: spadel
cooked: <p>I want to use a CNN for classification of galaxies. For this, I intended to use the torchvision.transforms module to perform some data augmentation. However, for this purpose there are PIL Images required. So I have to convert my <code>float type</code> images to <code>uint8</code> and hence I have lose some data. Can I do something to prevent this or is this not from relevance anyways?</p>

username: Nikronic
cooked: <p>Hi,</p>
<p>There is a <code>mode='F'</code> in PIL which enables 32bit floating point images. Have you tried it?</p>
<p>Also, if you really need to use original images (numpy array possibly), you can use <a href="https://kornia.readthedocs.io/" rel="nofollow noopener">Kornia</a> after converting images to tensors. It works directly on tensors and supports many different transforms.</p>
<p>Bests</p>

username: spadel
cooked: <p>Thank you!! I figured out I could also just change the original image from <code>float64</code> to <code>float32</code> and then I didn’t need to specify a mode at all.</p>

username: Nikronic
cooked: <p>Ow, I did not know that images are in <code>float64</code>. Yes, if you convert to <code>.float32</code>, PIL will automatically use <code>mode='F'</code>.</p>
<p>Good luck</p>
Run torch::jit::load(model_path) crash

username: k3833653
cooked: <p>I have pack libtorch libraries(1.2) to a computational module in a so file with cmake, if I only test this module , everythin is ok. However, if I link this so file to another module and generate a exe file, when I run the exe file, function torch::jit::load(model_path) crashed:<br>
terminate called after throwing an instance of ‘std::regex_error’, what():  regex_error.<br>
When I re-link libtorch.so to the excutable file in the CmakeLists.txt, and recompile the exe file, the problem solved. I’m puzzled that I have link the libtorch.so in the previous computational module, and excutable file link it with noproblem, why the torch::jit::load crashed if I don’t llink libtorch.so a second time?<br>
This is exe comiple infomation, with no link libtorch.so<br>
Building depend module “extern/private/scenarios/webpage/qa/src/query_modify:webpage”<br>
[100%] Built target webpage<br>
– The C compiler identification is GNU 4.9.2<br>
– The CXX compiler identification is GNU 4.9.2<br>
– Check for working C compiler: /usr/local/bin/gcc<br>
– Check for working C compiler: /usr/local/bin/gcc – works<br>
– Detecting C compiler ABI info<br>
– Detecting C compiler ABI info - done<br>
– Detecting C compile features<br>
– Detecting C compile features - done<br>
– Check for working CXX compiler: /usr/local/bin/g++<br>
– Check for working CXX compiler: /usr/local/bin/g++ – works<br>
– Detecting CXX compiler ABI info<br>
– Detecting CXX compiler ABI info - done<br>
– Detecting CXX compile features<br>
– Detecting CXX compile features - done<br>
– Get dep module c/qa:Aoeiuv_c_qa<br>
– Get dep module c/proto:Aoeiuv_c_proto<br>
– Get dep module c/common/base:Aoeiuv_c_common_base<br>
– Get dep module c/extern/misc_proto:Aoeiuv_c_extern_misc_proto<br>
– Get dep module extern/private/scenarios/webpage/common/server_common:qc_server_common<br>
– Get dep module extern/private/scenarios/webpage/qa/src/query_modify:webpage<br>
– Configuring done<br>
– Generating done<br>
– Build files have been written to: /data/qc_rebuild_webpage/Aoeiuv/extern/private/scenarios/webpa<br>
Scanning dependencies of target test_webpage<br>
[ 50%] Building CXX object CMakeFiles/test_webpage.dir/src/test_webpage.cpp.o<br>
[100%] Linking CXX executable test_webpage<br>
/usr/bin/ld: warning: libgomp-4f651535.so.1, needed by /data/qc_rebuild_webpage/Aoeiuv/extern/private/scenarios/webpage/qa/src/query_check_inference/…/…/…/…/…/…/…/extern/private/scenarios/webpage/common/libtorch/libtorch.so, not found (try using -rpath or -rpath-link)<br>
[100%] Built target test_webpage</p>
Does GPU speed up the optimization process

username: Vibhu_Bhatia
cooked: <p>I’m trying to optimize very simple functions like quadratic, absolute value etc using pytorch, and i was wondering if GPU will speedup that process. I am using no neural net or anything just a 1D tensor and using the above functions and loss functions and trying to find the minima.</p>

username: tom
cooked: <p>At a very basic level, GPUs work best when doing “the same thing to a lot of datapoints” (à la SIMD - Single Instruction Multiple Data), even if the details are a bit more intricate than for typical CPU SIMD units. This is applicable for convolutions, matrix multiplications (of reasonably large matrices), and other expressions.</p>
<p>Thus, if you look at your computation, if you only have a few values and very little computation with them, you won’t see much speedup. If, on the other hand, you have so many datapoints that you are dealing with large matrices, it might be worth a try.</p>
<p>Best regards</p>
<p>Thomas</p>
NCC version and Pytorch NCCL version mismatch

username: nash
cooked: <p>I have NCCL 2.5 installed on the system, but torch.cuda.nccl.version() shows 2.4.8.<br>
On a single machine with 2 gpus, it works fine.</p>

username: mrshenli
cooked: <p>Hey <a class="mention" href="/u/nash">@nash</a>, NCCL is packaged in PyTorch as a submodule. The current version if 2.4.8. If you would like to use your own version, you can set <code>USE_SYSTEM_NCCL=1</code>.</p>
What is the meanings of `*` in the method of `add_`?

username: coincheung
cooked: <p>Hi,<br>
I noticed that in definition of the method of tor <code>add_(other, *, alpha=1) → Tensor</code>, there is a <code>*</code>. What does that mean ?</p>

username: Nikronic
cooked: <p>Hi,</p>
<p>Actually, this is a python convention for forcing to use positional arguments.<br>
</p><aside class="onebox stackexchange">
  <header class="source">
      <a href="https://stackoverflow.com/questions/53797057/star-as-an-argument-in-python-function/53797072#53797072" target="_blank" rel="nofollow noopener">stackoverflow.com</a>
  </header>
  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/3058609/adam-smith" target="_blank" rel="nofollow noopener">
    <img alt="Adam Smith" src="https://www.gravatar.com/avatar/2c0357556707713572abbf637a907ef9?s=128&amp;d=identicon&amp;r=PG&amp;f=1" class="thumbnail onebox-avatar" width="60" height="60">
  </a>
<h4>
  <a href="https://stackoverflow.com/questions/53797057/star-as-an-argument-in-python-function/53797072#53797072" target="_blank" rel="nofollow noopener">Star (*) as an argument in python function</a>
</h4>

<div class="tags">
  <strong>python</strong>
</div>

<div class="date">
  
  answered by
  <a href="https://stackoverflow.com/users/3058609/adam-smith" target="_blank" rel="nofollow noopener">
    Adam Smith
  </a>
  on <a href="https://stackoverflow.com/questions/53797057/star-as-an-argument-in-python-function/53797072#53797072" target="_blank" rel="nofollow noopener">08:40PM - 15 Dec 18 UTC</a>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>Bests</p>

username: Dipam_Vasani
cooked: <p>Yes, look at the official docs:<br>
<a href="https://docs.python.org/3/whatsnew/3.8.html#positional-only-parameters" class="onebox" target="_blank" rel="nofollow noopener">https://docs.python.org/3/whatsnew/3.8.html#positional-only-parameters</a></p>