akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Layer performing weighted sum over 1 single dimension

NicolasCvzl: <p>Hi there,</p>
<p>I’have build a LSTM network to process time sequences.<br>
Each of my sequences are one-hot encoded on a 13 embedding dim space (pitch chroma + ‘pad’)<br>
Each of my sequences are of same length (padded).<br>
After putting them in batches of size : bsz, I end with 3D Tensors of size: (seq_len, bsz, embedding_size).<br>
Once a batch is fed into the LSTM, i end with outputs of size (seq_len, bsz, nb_tokens)</p>
<p>What I want: feed this output into a layer that performs a weighted average over a single dimension, that is the time dimension in order to learn weights for each time and end with a final output that should be of dimension (bsz, nb_tokens)</p>
<p>I haven’t been able to find such layers, I believe it can be done concatenating along some dimension and applying some average pooling layer with some parameters, but for the moment I haven’t been able to formulate it in a correct way.</p>
<p>Do you guys have any idea or knowledge of such layers ?</p>
<p>Thanks in advance!</p>

zyc: <p>Have you tried to use a <code>nn.Parameter()</code> to store the weights and perforn <code>torch.bmm()</code>?</p>
Custom Loss Function for a Batch

Abhranta: <p>Hi, I have to imlpement a custom loss function in pytorch. The network outputs a single tensor and my batch size if 1000. I have to take each tensor from the output and do some operations with it and return a value. My question is, do I need a for loop to iterate over all the tensors in the output batch, find the loss for each tensor and then do the mean or how can I do it more efficiently ?</p>

tom: <p>It depends on the loss formula, but you would want to vectorize the loss instead of using for loops if possible l. Most PyTorch functions lend themselves to it quite naturally through broadcasting.</p>
<p>Best regards</p>
<p>Thomas</p>

Abhranta: <p>Thank for the reply.<br>
The nature of my loss is such that i have to iterate over a list of numbers for each ouput of my network and then calculate the total loss. So, since my network kas 1 output and a batch size of 1000, is there any way to do it without a for loop ?</p>

tom: <p>At this level of generality, the answer necessarily is “maybe”, but so a typical thing would be to do the iteration in a loop but batched on all batch items at once.<br>
In my courses I always give <strong>Thomas’s rule of thumb</strong>: A for loop is OK as long as the things you operate on have 100s of items, but if you are down to 10s or single numbers, you are in trouble for efficiency.</p>
<p>While there are cases where it is hard or even not doable without custom kernels, many cases lend themselves to a batch strategy.</p>
<p>Best regards</p>
<p>Thomas</p>
Transfer part of the weights from one network to another

Andrew_Legend: <p>I have a pre-trained VAE. I want to transfer the weights of the decoder to the Generator.</p>
<p>I’ve searched all over the internet for a method to copy parts a Neural Network. None of them really helped what I am trying to do.</p>
<p>I have made sure both the generator and the decoder of the network to have <strong>the exact same architecture</strong> to make sure everything is implemented correctly.</p>
<p>However the way it is programmed in PyTorch is still unclear to me</p>
<p>Any kind of help is appreciated. Thanks in advanced</p>

nitaifingerhut: <p>You can iterate over the layers (module.children) - for the decoder from start to end and for the encoder from end to start, and assign the .weight parameter from decoder to encoder</p>

Andrew_Legend: <p>Do you mind to elaborate on this please?</p>

nitaifingerhut: <p>You can create two different nn.modules<br>
One for encoder and second for decoder<br>
After that you can iterate over the layers (for later in module.children) and for each layer you can override its weights (an attribute of the layer)</p>
Fixed Noise parameters

ArshadIram: <p>Hi everyone,</p>
<p>I know this question is fundamental for many of you, but I want to know the meaning of this 1, 1, in this torch. randn function while creating Gaussian noise distribution for generators.<br>
nz=100<br>
input = 64<br>
fixed_noise = torch.randn(64, nz, <strong>1, 1,</strong> device=device)</p>
<p><strong>Second</strong>: What is the meaning of (b_size**,** ) this comma after b_size parameter?</p>
<p>torch.full((b_size, ), real_label, dtype=torch.float, device=device)</p>

nitaifingerhut: <ol>
<li>This is simply the size of the tensor you are creating. In your case you’ll get a Gaussian distributed (64, 100, 1, 1) tensor.</li>
<li>This is again the size you are creating. In this case (as torch convention) the batch size is the first dim, and in torch.full you are creating a tensor of size batch_size filled with the required value.<br>
Since both this methods can receive any size tensor to create (python *args), you can pass the size of each dim separated by commas.</li>
</ol>

ArshadIram: <p>What is the meaning of 1, 1 is still unclear to me? Can you elaborate it more?</p>

nitaifingerhut: <p>You can think of it as dummy dimensions. For example take a 2 matrices of shape h,w. Each is of shape (h,w). You can think of it as a 3D matrix where one of the dimensions is of size 1, let’s say (1,h,w). Now you can for example to concatenate the two 2d matrices on the first dim to get a (2,h,w) 3D matrix</p>
Deploy deeplabv3_resnet101 on android (boiler plate) for binary segmentation?

adnankarim: <p>Hi there,<br>
I am using the code of</p>
<p>(msminhas93/DeepLabv3FineTuning) from github for binary segmentation of crackforest dataset.</p>
<p>It uses the threshold post processing of the model’s output to get segmentation output to be displayed else we are getting blank single color output (same output is displaying when this model’s .ptl file deployed in example android deeplabv3 segmentation code)</p>
<p>“You can see the inference for binary segmentation notebook below in python”</p>
<p>Url: <a href="https://github.com/msminhas93/DeepLabv3FineTuning/blob/master/Analysis.ipynb" rel="noopener nofollow ugc">DeepLabv3FineTuning/Analysis.ipynb at master · msminhas93/DeepLabv3FineTuning (github.com)</a></p>
<p>I have limited knowledge of java, thus what changes I should make to the given android java code to display the image segmented output image as done in python in above code.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/0/1093343b317c3bc6a9ce6fbb9d2f652ebcfe177b.png" data-download-href="https://discuss.pytorch.org/uploads/default/1093343b317c3bc6a9ce6fbb9d2f652ebcfe177b" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/0/1093343b317c3bc6a9ce6fbb9d2f652ebcfe177b.png" alt="image" data-base62-sha1="2mD1pgMMXn9WgAVxpB2shUs1qBB" width="484" height="500" data-dominant-color="13161C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">855×882 14.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><a href="https://github.com/pytorch/android-demo-app/blob/master/ImageSegmentation/app/src/main/java/org/pytorch/imagesegmentation/MainActivity.java" rel="noopener nofollow ugc">android-demo-app/MainActivity.java at master · pytorch/android-demo-app (github.com)</a></p>
<p>as you can see I am getting predicted pytorch tensor in java, how to apply same post processing steps to this tensor as done in  analysis notebook shared above.</p>
Multi-dimensional slicing

NicoAdamo: <p>I have a tensor of indices &gt; 0 and &lt; 256, of shape <code>[1,5,8]</code>, and a tensor of images of shape <code>[1,5,256,3,84,84]</code>. All I want to do is create a new tensor with the images at those indices - i.e. a tensor of size <code>[1,5,8,3,84,84]</code>. But nothing I do seems to work. torch.index_select seems to do this across one axis, but not multiple, and torch.gather would work, except it requires the index tensor and input tensor to be the same size. Do I really have to resort to a for loop here, or is there a nice way to do this?</p>

nitaifingerhut: <p><a href="https://pytorch.org/docs/stable/generated/torch.gather.html" rel="noopener nofollow ugc">torch.gather</a> should do just this<br>
You don’t need them to have the same shape, or you can just I squeeze it to the proper dimension</p>

NicoAdamo: <p>Except torch.gather does require the same <em>number</em> of dimensions. I assume that if I take the index tensor to size <code>[1,5,8,1,1,1]</code>, it’ll just give me the first pixel of each image, so how do I leave the images intact when taking the parts?</p>
Equivalence of "slice(None)" in tensor indices

degi: <p>I am trying to convert <a href="https://github.com/jefkine/zeta-learn/blob/04388f90093b52f5df2f334c898f3a1224f5a13f/ztlearn/utils/im2col_utils.py#L93" rel="noopener nofollow ugc"><code>np.add.at(x_padded, (slice(None), k, i, j), cols_reshaped)</code></a> this numpy code to pytorch.</p>
<p>Tried <code>x_padded.index_put_((slice(None), k, i, j), cols_reshaped, accumulate=True)</code>, but got <code>TypeError: expected Tensor as element 0 in argument 0, but got slice</code> error.</p>
<p>How can I convert</p>
<pre><code class="lang-python">np.add.at(x_padded, (slice(None), k, i, j), cols_reshaped)
</code></pre>
<p>to pytorch?</p>
<p>Thanks a lot <img src="https://discuss.pytorch.org/images/emoji/apple/pray.png?v=10" title=":pray:" class="emoji" alt=":pray:">!</p>
How to concatenate two tensor [B, H, V] and [H]

PatricYan: <p>two tensors in different dimensions, for example, A1=[B, H, V] A2=[H], how to contenate the two tensors to [B, H, V+1]?</p>

ptrblck: <p>This should work:</p>
<pre><code class="lang-python">B, H, V = 2, 3, 4

A1 = torch.randn([B, H, V])
A2 = torch.randn([H])
A2 = A2[None, :, None].expand(B, -1, -1)
print(A2.shape)
out = torch.cat((A1, A2), dim=2)
print(out.shape)
torch.Size([2, 3, 5])
</code></pre>

PatricYan: <p>Thanks. I have initialized<br>
A2=torch.zeros(1,H,1)<br>
A2 = A2.expand(B, -1, -1)<br>
torch.cat((A2,  A1), dim=2)</p>
Problems in Transformer

PaTrickWwW: <p>Problem: When i train the Transformer, i found the attention values in encoder, decoder are nearly same.<br>
Encoder:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/4/04f7f1f1a70f82cabd1de2cb4bd1ed4d9122b49a.png" data-download-href="https://discuss.pytorch.org/uploads/default/04f7f1f1a70f82cabd1de2cb4bd1ed4d9122b49a" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/4/04f7f1f1a70f82cabd1de2cb4bd1ed4d9122b49a.png" alt="image" data-base62-sha1="HX82VJU8TvPCcgu6sBpQHUtvTk" width="690" height="108" data-dominant-color="30363E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">703×111 5.54 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

nitaifingerhut: <p>You need to specify more details and put some of your code here to get more help</p>
Batchnorm2d in a CNN regressor

Hemanth_Pidaparthy: <p>I am learning a CNN regressor. I am experimenting with a very basic network (seen below).</p>
<pre><code>        self.features = nn.Sequential(
            nn.Conv2d(3, 8, kernel_size=5),
            nn.MaxPool2d(kernel_size=2, stride=2),
            nn.ReLU(inplace=True),
            
            nn.Conv2d(16, 32, kernel_size=5),
            nn.MaxPool2d(kernel_size=2, stride=2),
            nn.ReLU(inplace=True),
        )
</code></pre>
<p>With this network, the model does not learn anything - both training and test loss are very high and decreasing very little with training.</p>
<p>When I add BatchNorm2d() layer after the conv layers, the model is overfitting heavily.</p>
<pre><code>        self.features = nn.Sequential(
            nn.Conv2d(3, 8, kernel_size=5),
            nn.BatchNorm2d(8),
            nn.MaxPool2d(kernel_size=2, stride=2),
            nn.ReLU(inplace=True),
            
            nn.Conv2d(8, 16, kernel_size=5),
            nn.BatchNorm2d(16),
            nn.MaxPool2d(kernel_size=2, stride=2),
            nn.ReLU(inplace=True),
        )
</code></pre>
<p>Is there a reason why BatchNorm2d is causing the model to overfit heavily? Am I applying BatchNorm correctly? Irrespective of whether I double the data or decrease the size of the network further, batchNorm2d() causes heavy overfitting. So i am curious how to apply regularization with batchnorm.</p>

nitaifingerhut: <ol>
<li>The order seems right:<br>
CONV/FC → BatchNorm → activation → Dropout → CONV/FC → …</li>
</ol>
<p>There are two few details here, but if your train data and test data are from different distributions, the BN may seem to cause overfitting due to its moving average that fits the train data and not the test data.</p>
Help with understanding models architectures

Samuel_Bachorik: <p>Hi, I want to ask you if you can help me with understanding models architectures. I´m still quite new in AI stuff and I was using only created models from internet. Now when I want to create model for my problem I don’t know how to build model for it. Do you have any Tips or good book or article or anything that can help ? How do you guys know what layer to put in model. Same with activation functions.</p>
<p>Thank you</p>

nitaifingerhut: <p>Personally, I love <em>Deep Learning</em> by Goodfellow. There are many others.<br>
Most of the basics are learned over time with experience.</p>
How to track from where im getting UserWarning: floor_divide is deprecated?

hadaev8: <pre><code class="lang-auto">/opt/conda/lib/python3.7/site-packages/torch/_tensor.py:575: UserWarning: floor_divide is deprecated, and will be removed in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values.
To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor'). (Triggered internally at  /pytorch/aten/src/ATen/native/BinaryOps.cpp:467.)
  return torch.floor_divide(self, other)
</code></pre>
<p>Getting a lot of such warnings, but without trace.</p>

ptrblck: <p>The warning is triggered by a division with could have been executed via <code>torch.div</code> or the div operator (<code>a/b</code>), so you could check the code for the usage of those.<br>
You could also try to convert warnings to errors via:</p>
<pre><code class="lang-python">import warnings
warnings.filterwarnings("error")
</code></pre>
<p>to hopefully get a better stacktrace.</p>

abhisek: <p>I am also having this warning. <a class="mention" href="/u/ptrblck">@ptrblck</a> Shouldn’t this be fixed in <code>Pytorch</code> as the warning suggests instead of manually filtering warnings?</p>

ptrblck: <p>I don’t think the warning should be changed in PyTorch itself, but the user should decide, if she wants to fix or ignore it. My suggestion was to raise the warning to an error in order to figure out which line of code raises it so that you could fix it in your code.</p>

hadaev8: <p>Well, I’m just using tensor // 2.<br>
I don’t see how it should be fixed.<br>
This warning is kind of useless.</p>

ptrblck: <p>The warning provides two explicit ways to fix this depending on which behavior you expect.<br>
Sure, if you don’t find the warning useful, feel free to ignore it. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=10" title=":wink:" class="emoji" alt=":wink:"></p>

hadaev8: <p>But I want to use simple python // and I have no negative values in such tensors.<br>
So in my case, this warning is just spam.<br>
It also triggers for each core of dataloader multiprocessing, so it spams a lot.</p>

ptrblck: <p>In that case, since you’ve made sure you won’t be running into issue of this deprecation, filter out the warning.<br>
Other users should still be notified, that the <code>div</code> op changed its behavior and should be careful about their code, as it could suddenly break in later releases and could create a debugging hell (e.g. model is not converging anymore).</p>

hadaev8: <p>I think filtering out all warnings might be a bad idea.<br>
Added context manager or using pytorch function would make code bit less readable.<br>
So for now i decided warning spam is a lesser evil.</p>
How to clean edges of Mask

Neelam_Jain: <p>Hello,</p>
<p>I have a image and i need to create mask for every person</p>
<ul>
<li>
<p>List item<br>
have used resnet50 to extract mask for every person in the original image</p>
</li>
<li>
<p>List item</p>
</li>
</ul>
<p>For every mask i create a image</p>
<p>However, the edges of the mask needs to be filtered with proper algorithm. Refer the attached mask image<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/1/8/18aec092b5b06ab15998110e320ee2d59b421d6b.jpeg" alt="mask" data-base62-sha1="3wlRA7sHxlnDNIEhfJWsD8XEvIn" width="350" height="458"></p>
How to cast all floating point parameters and buffers to 8-bit datatype

One4ever: <p>Hi,<br>
I want to cast all floating-point parameters and buffers to  8-bit datatype (half of the half datatype). How can I do that? (I mean 8-bit floating point datatype)<br>
Thanks <img src="https://discuss.pytorch.org/images/emoji/apple/sunflower.png?v=10" title=":sunflower:" class="emoji" alt=":sunflower:"></p>
Gather_out_cpu(): Expected dtype int64 for index

Iulian_Petru: <p>Hii i get this error and i don’ t really know how to solve it:<code>gather_out_cpu(): Expected dtype int64 for index</code></p>
<p>and this is my code <code>    def learn(self, batch_state, batch_next_state, batch_reward, batch_action):         outputs = self.model(batch_state).gather(1, batch_action.unsqueeze(1)).squeeze(1)         next_outputs = self.model(batch_next_state).detach().max(1)[0]         target = self.gamma * next_outputs + batch_reward         td_loss = F.smooth_l1_loss(outputs, target)         self.optimizer.zero_grad()         td_loss.backward(retain_variables = True)         self.optimizer.step()</code></p>

eqy: <p>What is the data type of your index tensor here? You might want to call <code>.long()</code> on it to change the datatype before passing it to <code>gather</code>. (e.g., if it is <code>batch_action</code>, you may want to use <code>batch_action.long()</code>.</p>

Iulian_Petru: <pre><code>    batch_action = batch_action.type(torch.int64)
</code></pre>
<p>i had to add this line to make it work</p>
When using libtorch 1.9.0, there are some ld messages that I don’t understand

Fitanium: <p>Yesterday, I found that libtorch 1.9.0 had been released, so I downloaded and installed it on my computer. When I compiled my code, there were some <code>ld</code> messages that I was confused at the moment, does anyone can tell me why or how to  solve it?</p>
<p>System info:<br>
Xubuntu 20.04, x86_64, gcc version 9.3.0</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/f/0f747331fc99c1b6eb02cb7be2c254bc2ccba5bf.png" data-download-href="https://discuss.pytorch.org/uploads/default/0f747331fc99c1b6eb02cb7be2c254bc2ccba5bf" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/f/0f747331fc99c1b6eb02cb7be2c254bc2ccba5bf_2_690x299.png" alt="image" data-base62-sha1="2cIEBMNkCkf61AjMCzB57bKg8GP" width="690" height="299" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/f/0f747331fc99c1b6eb02cb7be2c254bc2ccba5bf_2_690x299.png, https://discuss.pytorch.org/uploads/default/optimized/3X/0/f/0f747331fc99c1b6eb02cb7be2c254bc2ccba5bf_2_1035x448.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/0/f/0f747331fc99c1b6eb02cb7be2c254bc2ccba5bf.png 2x" data-dominant-color="272D38"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1107×481 121 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

tom: <p>As far as I understand, it is an inconsistency in the shared libraries information, but  <a href="https://sourceware.org/bugzilla/show_bug.cgi?id=25135" rel="noopener nofollow ugc">it should be mostly harmless</a> (but yeah, it’s annoying…). One thing to know here is that (as far as I know, I didn’t check recently), the PyTorch distributables are built with a rather old toolchain (gcc, libc etc.) because the condition for them working are “the run-time environment must be at least as new as build-time environment”. Maybe that old toolchain produces these inconsistent bits of information.<br>
If you are very determined, you could try building PyTorch yourself and see if the warning disappears.</p>
<p>Best regards</p>
<p>Thomas</p>

Fitanium: <p>Hi Thomas,</p>
<p>Thank you very much for your experience and advice. Since that time, I have been using libtorch 1.9.0. And after searching some information on internet, I have found that if I add <code>add_link_options(-fuse-ld=gold)</code> in <code>CMakeLists.txt</code>, the warning message disappeared. So in my opinion , I guess the libtorch 1.9.0 was build with <code>ld.gold</code> linker for large project or application. Maybe I am wrong about that, but compiling code successfully without warning message makes me happy.</p>
<p>Best regards,</p>
<p>Fitanium</p>
Resuming a model for training after wrapping in Data Parallel

Ammara: <p>Hi, I create a model and then wrap it in Data Parallel.<br>
model = DataParallel(model)</p>
<p>Now, can I resume the model from a checkpoint using load_state_dict()?<br>
model.load_state_dict(checkpoint[‘model_state_dict’])</p>
<p>will the new model weights be broadcasted across all gpus?</p>

ptrblck: <p>You could directly load the <code>state_dict</code>, in case it was also stored from an <code>nn.DataParallel</code> model.<br>
Otherwise you would see key errors pointing to the missing <code>.module</code> keys.<br>
I would thus generally recommend to store and load the <code>state_dict</code> from the plain model before wrapping it into <code>nn.DataParallel</code>.</p>

Ammara: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>, thankyou for your reply.</p>
<p>The stored model was trained in DistributedDataParallel, and I am resuming training in DataParallel. If I load the state_dict from the plain model first, I have to  remove the prefix ‘module.’ from the state_dict keys, and then I wrap the model in DataParallel.</p>
<p>If I wrap the model in DataParallel first, and then load the state_dict, I don’t get any key mismatch issues. But I want to know if this approach is correct? Loading the state_dict after wrapping the model in DataParallel essentially means changing the weights of the model. Will the model after loading the state_dict be replicated across all gpus?</p>

ptrblck: <p>Yes, <code>nn.DataParallel</code> will scatter the parameters in each forward pass, so it should work.<br>
This is also the reason for the lower performance compared to DDP, which has a reduced communication overhead.</p>

Ammara: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> thankyou for replying.</p>
<p>So in DDP the parameters are not scattered in each forward pass? In case of DDP, will wrapping the model first and then loading the state_dict after work as well?</p>

ptrblck: <p>DDP will share the parameters as explained in the <a href="https://pytorch.org/docs/stable/notes/ddp.html">docs</a>.</p>
<blockquote>
<p><strong>Construction</strong>: The DDP constructor takes a reference to the local module, and broadcasts <code>state_dict()</code> from the process with rank 0 to all other processes in the group to make sure that all model replicas start from the exact same state. Then, each DDP process creates a local <code>Reducer</code>, which later will take care of the gradients synchronization during the backward pass.</p>
</blockquote>
<p>I haven’t checked different workflows of loading the <code>state_dict</code> before and after the model wrapping in util. functions such as DDP, as I’m sticking to initialize the model first and then wrap it into these classes.<br>
However, I guess it should work and you could verify it with a small script by checking the parameters on each rank.</p>

Ammara: <p>Thankyou <a class="mention" href="/u/ptrblck">@ptrblck</a>. This helps a lot.</p>
Function "'UnsafeViewBackward'" returned nan values in its 0th output

fengqh: <p>[W python_anomaly_mode.cpp:104] Warning: Error detected in UnsafeViewBackward. Traceback of forward call that caused the error:<br>
File “/anaconda3/lib/python3.6/threading.py”, line 884, in _bootstrap<br>
self._bootstrap_inner()<br>
File “/anaconda3/lib/python3.6/threading.py”, line 916, in _bootstrap_inner<br>
self.run()<br>
File “/anaconda3/lib/python3.6/threading.py”, line 864, in run<br>
self._target(*self._args, **self._kwargs)<br>
File “/anaconda3/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py”, line 61, in _worker<br>
output = module(*input, **kwargs)<br>
File “/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 727, in _call_impl<br>
result = self.forward(*input, **kwargs)<br>
File “”, line 166, in forward<br>
pixels = pixel_attn(pixels) + pixels<br>
File “/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 727, in _call_impl<br>
result = self.forward(*input, **kwargs)<br>
File “”, line 31, in forward<br>
return self.fn(self.norm(x), **kwargs)<br>
File “/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 727, in _call_impl<br>
result = self.forward(*input, **kwargs)<br>
File “”, line 69, in forward<br>
q, k, v = self.to_qkv(x).chunk(3, dim = -1)<br>
File “/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 727, in _call_impl<br>
result = self.forward(*input, **kwargs)<br>
File “/anaconda3/lib/python3.6/site-packages/torch/nn/modules/linear.py”, line 93, in forward<br>
return F.linear(input, self.weight, self.bias)<br>
File “/anaconda3/lib/python3.6/site-packages/torch/nn/functional.py”, line 1692, in linear<br>
output = input.matmul(weight.t())<br>
(function _print_stack)</p>
<p>who can help me, i feel very terrible</p>

fengqh: <p>i meet many backward errors, i check the inputs do not contain nan</p>

ptrblck: <p>I assume <code>self.to_qkv</code> uses a linear layer internally? If so, could you check the activation inputs as well as the parameters for invalid values right before the error is raised?</p>

fengqh: <p>when i use single GPU, no error, but 3 GPU, many erros like “***Backward nan”</p>
Training loss changes after resuming

NosremeC: <p>I have been stuck with this wired phenomenon for a really long time and read through many discussions but not yet found a solution.</p>
<p>So I have trained several models on colab pro, with a Tesla V100 GPU. The loss generally decreased in the first few epochs, but after that it started to go up sometimes. Natually, I decided to restarted the training by resuming from the checkpoint every time I saw the loss increasing(and I’m confident all parameters from models and optimizers were properly saved and loaded), and then magically the loss went down after I did it. This is almost always the case.</p>
<p>I am working on mutiple object tracking, so it is a muti-task model, and I used uncertainty loss, both s_det and s_id are trained, saved and loaded. I also used Adam as my optimizer.</p>
<pre><code class="lang-auto">        if opt.multi_loss == 'uncertainty':
            loss = torch.exp(-self.s_det) * det_loss + torch.exp(-self.s_id) * id_loss + (self.s_det + self.s_id)
            loss *= 0.5
</code></pre>
<pre><code class="lang-auto">        self.s_det = nn.Parameter(-1.85 * torch.ones(1))
        self.s_id = nn.Parameter(-1.05 * torch.ones(1))
</code></pre>
<p>I am just wondering if anyone know what happened, and why resuming the training can somehow magically ‘force’ the loss to decrease even it was intially increasing? Many thanks!</p>

ptrblck: <p>I don’t think resuming the training from a specific epoch forces the loss to decrease, but it sounds more as if your model diverges non-deterministically. I.e. I assume you don’t expect to see bitwise-accurate outputs after resuming the training compared to the initial training run.</p>

NosremeC: <p>I do agree with you that simply resuming from the checkpoint should not be the reason the loss was decreasing. It didn’t make sense to me either <img src="https://discuss.pytorch.org/images/emoji/apple/rofl.png?v=10" title=":rofl:" class="emoji" alt=":rofl:">. However, the fact the loss went down after resuming told me the model somehow started to perform better, at least on the training set, right? Are you suggesting the model would only perform better on training set, but not on the test set if I kept paused and resuming the training?</p>
Convert float32 to float16 with reduced GPU memory cost

origin_of_symmetry: <p>Hi there,</p>
<p>I have a huge tensor (Gb level) on GPU and I want to convert it to float16 to save some GPU memory. How could I achieve this?</p>
<p>I tried</p>
<pre><code class="lang-auto">a_fp16 = a.to(torch.float16)
</code></pre>
<p>But it actually reserves another memory block to save the fp16 tensor and the fp32 tensor is still there.<br>
I also tried</p>
<pre><code class="lang-auto">del a
</code></pre>
<p>after casting. But the memory is not released.</p>
<p>Thanks</p>

ptrblck: <aside class="quote no-group" data-username="origin_of_symmetry" data-post="1" data-topic="128117">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/origin_of_symmetry/48/29831_2.png" class="avatar"> origin_of_symmetry:</div>
<blockquote>
<p>But it actually reserves another memory block to save the fp16 tensor and the fp32 tensor is still there.</p>
</blockquote>
</aside>
<p>That’s expected, as you are still holding references to both tensors.</p>
<aside class="quote no-group" data-username="origin_of_symmetry" data-post="1" data-topic="128117">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/origin_of_symmetry/48/29831_2.png" class="avatar"> origin_of_symmetry:</div>
<blockquote>
<p>I also tried</p>
<pre><code class="lang-auto">del a
</code></pre>
<p>after casting. But the memory is not released.</p>
</blockquote>
</aside>
<p>This shouldn’t be the case, as <code>a</code> would be released and the memory would be added to the cache as seen here:</p>
<pre data-code-wrap="python"><code class="lang-python"># create a tensor of 4MB
a = torch.randn(1024, 1024, device='cuda')
print(torch.cuda.memory_allocated() / 1024**2)
&gt; 4.0
print(torch.cuda.memory_reserved()/ 1024**2)
&gt; 20.0

# create a FP16 copy of 2MB
a_fp16 = a.to(torch.float16)
print(torch.cuda.memory_allocated() / 1024**2)
&gt; 6.0
print(torch.cuda.memory_reserved()/ 1024**2)
&gt; 20.0

# delete 4MB tensor
del a
print(torch.cuda.memory_allocated() / 1024**2) 
&gt; 2.0
print(torch.cuda.memory_reserved()/ 1024**2)
&gt; 20.0
</code></pre>

origin_of_symmetry: <p>For my case, is it because tensor a is in the computing graph? do I need to detach it to reduce memory?</p>

ptrblck: <p>I believe the issue is that the backward pass uses additional memory by computing the gradients.<br>
Detaching the tensor would lower the memory usage, but would also break the training, so no you should not detach the output/loss, if you want to train the model.</p>

origin_of_symmetry: <p>I believe so. Unfortunately, del a still doesn’t work for me…</p>

ptrblck: <p>Could you post an executable code snippet, which would show this non-working behavior?</p>
How to write a simple dataloader

nikiguo93: <p>I want to write a simple dataloader. Can I use something like the following:<br>
for epoch in range(MaxEpoch):<br>
for i in range(numple of sample/batch_size):<br>
x_input = X[i*batch_size:(i+1）*batch_size,:,:,:]<br>
y_input = Y[i*batch_size:(i+1)*batch_size,:,:,:]<br>
y_pred = model(x_input)<br>
…<br>
here, X.size = 16540 * 3 * 500*500 (there are totally 16540 images, each of which is of size 3 * 500 * 500) and Y.size = 16540 * 1 * 17 * 100. The numer of sample is 16540 and the batch size is 64. Then the numer of batches is 258.</p>

ayalaa2: <p>Sure there’s no reason why something like this can’t work. I always prefer to just use PyTorch’s dataset/dataloading classes though.</p>
<p>You can use <code>torch.utils.data.TensorDataset</code> to wrap your X and Y and then wrap the dataloader around them.</p>
<pre><code class="lang-auto">dataset = torch.utils.data.TensorDataset(X, Y)
dataloader = torch.utils.data.DataLoader(dataset, batch_size=batch_size, num_workers=4)

for x_input, y_input in dataloader:
    y_pred = model(x)
    
</code></pre>
<p>The dataloader class will give you future flexibility</p>
What is fast-pathed or fast-pass functions in Pytorch?

DJ_Kim: <p>I am reading pytorch code base and issue to get a better understanding of the design choices. I keep seeing fast-pathed or fast-passed function. I was wondering what these are? For instance, the issue here (<a href="https://github.com/pytorch/pytorch/pull/46469" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/pull/46469</a>).</p>
<p>Thank you in advance!</p>

DJ_Kim: <p>I was wondering if anyone is willing to shed some light on this question? Thank you!</p>

googlebot: <p>it is a generic designation of execution shortcuts, i.e. skipping code conditionally or by altering function call dispatching</p>
Conversion from keras to pytorch (

Mathieu_Grosso: <p>I’m trying to convert CNN model code from Keras to Pytorch.</p>
<p>here is the original keras model: it comes from this article: " Squeeze-and-Excitation Networks"</p>
<pre><code class="lang-auto">def se_block(in_block, ch, ratio=16):
    x = GlobalAveragePooling2D()(in_block)
    x = Dense(ch//ratio, activation='relu')(x)
    x = Dense(ch, activation='sigmoid')(x)
    return multiply()([in_block, x])
</code></pre>
<p>here is what i have done but i am unsure since i have mixed results when adding this module to my network:</p>
<pre><code class="lang-auto">def GlobalAveragePooling(x):
    x=torch.mean(x.view(x.size(0),x.size(1),-1),dim=2)
    return x   

class se_block(nn.Module):
    def __init__(self,in_block,ch,ratio=16):
        super(se_block,self).__init__()
        self.GlobalAveragePooling=GlobalAveragePooling()
        self.Dense1=nn.Linear(in_features=in_block,out_features=ch//ratio,)
        self.Dense2=nn.Linear(in_features=ch//ratio,out_features=ch)

    def forward(x):
        x=GlobalAveragePooling(x)
        x=Dense1(x)
        x=F.relu(x)
        x=Dense2(x)
        x=F.sigmoid(x)
        return x 



</code></pre>
<p>Thanks <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

albanD: <p>Hi,</p>
<p>That looks good.<br>
Aren’t you missing the last multiply though? If you you might want to modify your forward to keep a reference to the input and return input * x.</p>

Mathieu_Grosso: <p>Hello,<br>
Thanks for the help, yes indeed I had forgotten to multiply the input by the weight !</p>
Process doesn't exit after enumerate dataloader

Ardeal: <p>Hi</p>
<p>I init 2 processes on 2 GPU with the followin code</p>
<p>command is:<br>
<code>CUDA_VISIBLE_DEVICES=4,5 WORLD_SIZE=2 python -m torch.distributed.launch --nproc_per_node=2 --master_port 44744 train.py </code></p>
<p>after executing the above command, 2 processes will not exit.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/a/f/af14380769c6720ed7305d79c94f56ea44ddebf9.png" alt="图片" data-base62-sha1="oYOSjvA7fTuTzIWuMVpSp1y5d6V" width="643" height="145"></p>
<p><strong>if I comment out the for loop to enumerate loader_eval in the following code, 2 processes could successfully exit.</strong><br>
<strong>if I un-comment  out the for loop to enumerate loader_eval in the following code, 2 processes could not exit.</strong></p>
<p><strong>is there any way to check where the process is pending? or how to check where the process is pending?</strong></p>
<p>is there anything wrong with my code?<br>
how to make those processes exit after the code finalized?</p>
<pre data-code-wrap="python"><code class="lang-python">
if args.distributed:
	torch.cuda.set_device(args.local_rank)
	torch.distributed.init_process_group(backend='nccl', init_method='env://')
	args.world_size = torch.distributed.get_world_size()
	args.rank = torch.distributed.get_rank()
	logger_trainer_pim.info('----------&gt; world_size == {}, rank=={}, local_rank == {}, gpu index =={}'.format(args.world_size, args.rank, args.local_rank, args.device.index))
	logger_trainer_pim.info('Training in distributed mode with multiple processes, 1 GPU per process. Process %d, total %d.' % (args.rank, args.world_size))
else:
	logger_trainer_pim.info('Training with a single process on 1 GPU.')
assert args.rank &gt;= 0

dataset_eval = create_dataset(
            args.dataset, root=args.data_validate_dir, split=args.val_split, is_training=False, batch_size=args.batch_size)

loader_eval = create_loader(
	dataset_eval,
	input_size=data_config['input_size'],
	batch_size=args.validation_batch_size_multiplier * args.batch_size,
	is_training=False,
	use_prefetcher=args.prefetcher,
	interpolation=data_config['interpolation'],
	mean=data_config['mean'],
	std=data_config['std'],
	num_workers=args.num_workers,
	distributed=args.distributed,
	crop_pct=data_config['crop_pct'],
	pin_memory=args.pin_mem,
)

print('------------------------------------------&gt;')
print(args)
for batch_idx, (input, target) in enumerate(loader_eval):
	if batch_idx&gt;=1: break
print('-------------------------&gt; enumerate loader_eval done.')

return {"init": 0.888, "final": 1}




</code></pre>

H-Huang: <p>You are setting world_size=2 and have nproc_per_node=2 so you actually have 4 processes total. For each of the nodes, it looks like you are setting the device to local_rank then spawning 2 subprocesses to load the data. I don’t see anything off about what you are doing but just wanted to make sure that was what was intended.</p>
<p>I am not able to run the example you provided due to missing arg parsing, imports, etc. Do you have a sample script I can run locally to reproduce the error?</p>
Convert existing model/weights fromData Parallel to generic?

DrEGPU: <p>Sorry if this isn’t the correct topic. I did try to look for my problem, but I don’t I think I’m using the right search terms, which is why I’m here.</p>
<p>So, I trained a model using <code>nn.DataParallel</code> and it worked fine. I was able to inference fine on my machine, but only my machine. I finally figured out that I mistakenly did NOT save my model as generic as described here: <a href="https://pytorch.org/tutorials/beginner/saving_loading_models.html#saving-torch-nn-dataparallel-models" rel="noopener nofollow ugc">https://pytorch.org/tutorials/beginner/saving_loading_models.html#saving-torch-nn-dataparallel-models</a></p>
<p>Is there a way to convert my existing model to a generic form so it can be used by anyone? Hopefully in a automated way and not me sludging through the weights manually?</p>

ayalaa2: <p>Are you still able to load the checkpoint? DataParallel just hides the actual model away under the attribute <code>module</code>. So you can do something like this:</p>
<pre><code class="lang-auto">my_model = ModelClass(...)
my_model = DataParallel(my_model, ...)

state_dict = torch.load(...)
my_model.load_state_dict(state_dict)

actual_model = my_model.module
new_state_dict = actual_model
</code></pre>
<p>Haven’t tested this, but I bet something like this would work as well:</p>
<pre><code class="lang-auto">state_dict = torch.load(...)

new_state_dict = {}
for k, v in state_dict.items():
    k = k.replace('module.', '')  # Each weight k will be prefix with the word "module"
    new_state_dict[k] = v

model = SomeModel(...)
model.load_state_dict(new_state_dict)
</code></pre>

DrEGPU: <p>Thanks, we tried something similar to the second method and it “works”, but unsure if the model is accurately predicting or not. Still investigating… Thanks again!</p>
Delete a model from memory

omarfoq: <p>Hello,</p>
<p><strong>What is the correct to delete a model from memory?</strong> I tried to use</p>
<pre><code class="lang-auto">del model
</code></pre>
<p>But it doesn’t seem to work, because the optimizer still contains the model weight.</p>

Mazhar_Shaikh: <p><a href="https://stackoverflow.com/questions/21514631/how-to-delete-an-instantiated-object-python" rel="noopener nofollow ugc">Relevant stack overflow thread on object deletion in python</a><br>
I think in general in python, del only removes that particular reference to the object. It does not remove any other reference to that object’s child members. So when you assign the model parameters to the optimizer, the model parameters now have a reference in the optimizer. The only way to remove those would be to manually remove either the optimizer itself or all references to the relevant param groups in the optimizer, along with any other references to any of the child members of the model.</p>
<pre><code class="lang-auto">some_module = model.layer
del model
del optimizer
del some_module
</code></pre>

omarfoq: <p>Thanks <a class="mention" href="/u/mazhar_shaikh">@Mazhar_Shaikh</a>. Btw your answer suggests that  to go through all the modules of the model and delete them. Is this what you ment or it is enough to delete the model and the optimizer?</p>
Grid_sampling padding mode

gabrieldernbach: <p>I am a big fan of the grid_sampling method in</p>
<pre><code class="lang-auto">https://pytorch.org/docs/master/generated/torch.nn.functional.grid_sample.html#torch.nn.functional.grid_sample
</code></pre>
<p>I now face a case, where all my images have white background, but the padding mode only allows for zero padding (black).  Can someone guide me through the steps for extending the function with constant padding (e.g. all white)?</p>
Weight Initialization Strategy in Pretrained Models

mattshomepage: <p>Given say, pretrained VGG16 models, how can I know what init distributions of weights were used? Do I assume the relevant paper was followed exactly? For example, in VGG16 models you have (<a href="https://arxiv.org/pdf/1409.1556.pdf" rel="noopener nofollow ugc">https://arxiv.org/pdf/1409.1556.pdf</a>):</p>
<p><code>For random initialization (where applicable), we sampled the weights from a normal distribution with the zero mean and 10^−2 variance</code></p>
<p>Aside from assuming this, is there a way to check this was in fact the case for these pre-trained models?</p>
Getting very high validation loss for skewed dataset

Arohan_Ajit: <p>I have a very skewed dataset in which number of class samples is [74,859]. My training method is correct so I feel I am making an error is using the weighted random sampler and weights in cross entropy. Is my implementation correct? I used WeightedRandomSampler from train dataloader and SubsetRandomSampler for test dataloader.</p>
<pre><code class="lang-auto">splits = 10

kfold = KFold(n_splits=splits,shuffle=False)

for fold,(train_idx,test_idx) in enumerate(kfold.split(data)):

    print(f'FOLD {fold}')

    print('--------------------------------')

    class_sample_count = [74,859].
    weights = 1 / torch.Tensor(class_sample_count)
    weights = weights.double()
    train_sampler = torch.utils.data.sampler.WeightedRandomSampler(weights, 10)
    test_sampler = SubsetRandomSampler(test_idx)

    loaders = {
    'train': torch.utils.data.DataLoader(data, batch_size=16, sampler=train_sampler),
    'test': torch.utils.data.DataLoader(data, batch_size=16, sampler=test_sampler)
    }

    vgg16 = torchvision.models.vgg16(pretrained=True)
    for param in vgg16.features.parameters():
        param.requires_grad = False

    num_ftrs = vgg16.classifier[6].in_features
    vgg16.classifier[6] = nn.Linear(num_ftrs, 2)
    model = vgg16.to(device)
    criterion = nn.CrossEntropyLoss()
    optimizer = optim.Adam(model.parameters(), lr=0.001)
    train(3, loaders,   model, optimizer, criterion, train_on_gpu, 'model.pt')
    model.load_state_dict(torch.load('model.pt'))
</code></pre>
<p>Output:</p>
<pre><code class="lang-auto">Batch: 0	Epoch: 1 	Training Loss: 0.666256
Epoch: 1 	Training Loss: 0.666256 	Validation Loss: 11.949794 	Accuracy: 0.237942
Validation loss decreased (inf --&gt; 11.949794).  Saving model ...
Batch: 0	Epoch: 2 	Training Loss: 0.000000
Epoch: 2 	Training Loss: 0.000000 	Validation Loss: 32.636715 	Accuracy: 0.237942
Batch: 0	Epoch: 3 	Training Loss: 0.000000
Epoch: 3 	Training Loss: 0.000000 	Validation Loss: 58.557861 	Accuracy: 0.237942

FOLD ACCURACY : 0.2379421221864952

Batch: 0	Epoch: 1 	Training Loss: 0.561094
Epoch: 1 	Training Loss: 0.561094 	Validation Loss: 22.119741 	Accuracy: 0.000000
Validation loss decreased (inf --&gt; 22.119741).  Saving model ...
Batch: 0	Epoch: 2 	Training Loss: 0.000000
Epoch: 2 	Training Loss: 0.000000 	Validation Loss: 53.926662 	Accuracy: 0.000000
Batch: 0	Epoch: 3 	Training Loss: 0.000000
Epoch: 3 	Training Loss: 0.000000 	Validation Loss: 91.861961 	Accuracy: 0.000000

FOLD ACCURACY : 0.0
</code></pre>
<p>Is using WeightedRandomSampler and class weights in cross entropy toghether alright? Also should I use WeightedRandomSampler for testloader as well?</p>
Compute cartesian product for batched tensor

Samue1: <p>I have the following tensor</p>
<pre><code class="lang-auto">x = torch.tensor([[[[ 4,  6],
                    [12, 10]],
                   [[20, 22],
                    [28, 30]]],
                  [[[ 1,  6],
                    [13, 15]],
                   [[16, 18],
                    [29, 26]]]])
</code></pre>
<p>with dimensions <code>(batch_size, channels, height, width)</code>. Now I want to apply <code>torch.cartesian_prod()</code> to each element of the batch. I can use <code>.flatten(start_dim=0)</code> to get a one-dimensional tensor for each batch element with shape <code>(batch_size, channels*height*width)</code>. However, <code>torch.cartesian_prod()</code> is only defined for one-dimensional tensors. Is there a workaround to compute the cartesian product for each batch dimension?</p>
<p>Currently I use</p>
<pre><code class="lang-auto">batch_size = 2
indices = list()
for batch in range(batch_size):
        indices.append(torch.cartesian_prod(x[batch], x[batch]))
</code></pre>
<p>which is not really elegant.</p>