akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Get values between item from LongTensor

alabijesujoba: <p>Is there an efficient way of getting tensor of items between two values in a LongTensor<br>
For example</p>
<pre><code class="lang-auto">#input
x = torch.LongTensor([0,2,3,5,1,4,2,7,1,8])

#output
x = [torch.LongTensor([3,5]), 
torch.LongTensor([7])
]
</code></pre>
<p>In the example above I want to get the values between 2 and 1 into a new tensor. Is there any method in pytorch for this? or any efficient way?</p>
Differentiable indexing with float tensors

Alon_Zolfi: <p>Hey,</p>
<p>Is there any way to index a tensor with float tensors and still keeping these tensors differentiable?<br>
For example, I have a tensor of shape (3, 400, 400) which stands for an RGB image.<br>
I want to “draw” a square in the image within some boundaries.<br>
So there’s the code:<br>
img = torch.full((3, img_size, img_size), dtype=torch.float32, fill_value=1)<br>
And I have 2 float tensors representing the center of the square and a fixed width.<br>
x_center = torch.rand(1, requires_grad=True)<br>
y_center = torch.rand(1, requires_grad=True)<br>
And I have a representation of the square left upper corner and right lower corner:<br>
x1 = (x_center - (width/2))*img_size<br>
x2 = (x_center + (width/2))*img_size<br>
y1 = (y_center - (width/2))*img_size<br>
y2 = (y_center + (width/2))*img_size</p>
<p>So finally I want to do this:<br>
img[: x1:x2, y1:y2] = some_value</p>
<p>which of course cannot be done since these indices are floats and not longs. for example:<br>
x1 = ((x_center - (width/2))*img_size).long()</p>
<p>converting these parameters to long does not keep the computation graph of x_center and y_center.</p>
<p>Is there any way to do this so when I call the backward function x_center and y_center grads will be updated?</p>
<p>Thanks!</p>

albanD: <p>Hi,</p>
<p>The only way I can think of is to use <code>affine_grid</code> and <code>grid_sample</code> (doc <a href="https://pytorch.org/docs/stable/nn.functional.html?highlight=grid_sample#torch.nn.functional.grid_sample">here</a>) that are used by the Transformer layers.</p>

Alon_Zolfi: <p>Can you please show an example? I’ve read so much about these functions and I don’t have any idea how to do it.</p>

albanD: <p>Hi,</p>
<p>You can find an example here: <a href="https://discuss.pytorch.org/t/gradient-zero-for-one-parameter-in-custom-model-resulting-in-no-update/81091/6" class="inline-onebox">Gradient zero for one parameter in custom model resulting in no update</a></p>
What methods do I override for ToTensor() and Normalize()

nlam: <p>If I wanted to write my own custom implementations of these classes, which methods would I need to implement?</p>

ptrblck: <p>A lot of <code>torchvision.transforms</code> are a simple class with a <code>__call__</code> method as seen <a href="https://github.com/pytorch/vision/blob/8e611cfd9fbd2cccd95b2f312f778ccf0d3218c8/torchvision/transforms/transforms.py#L73-L95">here</a>. You could thus implement a custom class using this template or you could alternatively also use <a href="https://pytorch.org/docs/stable/torchvision/transforms.html#torchvision.transforms.Lambda"><code>transforms.Lambda</code></a>, which would accept your custom function.</p>

nlam: <p>alright thank you, will try these out</p>
Take the mean of the innermost tensor

Shantanu_Ghosh: <p>I have a tensor like the following:<br>
x = torch.tensor([[[[2, 4, 7],<br>
[6, 5, 4],<br>
[6, 5, 5]],</p>
<pre><code>               [[12, 14, 17],
                [26, 25, 42],
                [62, 52, 52]]]])
</code></pre>
<p>Now I want to take the mean of the inner most 3X3 array<br>
So my final array looks like this after taking mean:<br>
[[[4.88],<br>
[33.5]]]<br>
the final size is 1x2x1</p>

albanD: <p>Hi,</p>
<p>You can use the <a href="https://pytorch.org/docs/stable/torch.html#torch.mean"><code>torch.mean(t, dim=)</code></a> function to do that by specifying which dimension you want to compute the mean over.</p>

Shantanu_Ghosh: <p>I do it using<br>
y = x.mean(dim=(2,3), dtype=float)<br>
but it makes the size as (1,2)<br>
but i want (1, 2, 1)</p>

albanD: <p>You can use <code>.unsqueeze(-1)</code> to add a new dimension of size 1 on your Tensor.</p>
Focal loss normalization (RetinaNet)

JohannesB: <p>In RetinaNet (e.g., in the Detectron2 <a href="https://github.com/facebookresearch/detectron2/blob/master/detectron2/modeling/meta_arch/retinanet.py#L180-L187" rel="nofollow noopener">implementation</a>), the (focal) loss is normalized by the number of <em>foreground</em> elements <code>num_foreground</code>. However, the number of elements being considered in the loss function are the <em>valid</em> elements <code>valid_idxs</code>, i.e., foreground <strong>and</strong> background elements. So I would expect the last code line to be something like <code>max(1, valid_idxs.sum())</code>. This is most probably also the behavior when using <code>torch.nn.functional.binary_cross_entropy_with_logits</code> with <code>reduction='mean'</code>.</p>
<p>The paper <em>Focal Loss for Dense Object Detection</em> from Lin et. al does not mention the normalization as far as I can see. The results are far better when normalizing with the number of foreground elements like in the Detectron2 code shown below.</p>
<p>Can someone please explain why the number of foreground elements is chosen for normalization instead the number of valid elements (foreground + background)?</p>
<pre><code class="lang-auto">	valid_idxs = gt_classes &gt;= 0
	foreground_idxs = (gt_classes &gt;= 0) &amp; (gt_classes != self.num_classes)
	num_foreground = foreground_idxs.sum()
	...
	# logits loss
	loss_cls = sigmoid_focal_loss_jit(
		pred_class_logits[valid_idxs],
		gt_classes_target[valid_idxs],
		alpha=self.focal_loss_alpha,
		gamma=self.focal_loss_gamma,
		reduction="sum",
	) / max(1, num_foreground)
</code></pre>

RobinBeilvert: <p>My guess is that focal loss <em>focuses</em> on outputting high values only for the foreground boxes and the difficult background ones, aiming at balancing the importance given to the foreground and background boxes and trying to ignore the large amount of easily classified background boxes.<br>
Hence in the case of an overwhelming quantity of backgrounds boxes, normalizing against number of foreground boxes (which is kind of close to foreground boxes + smoothed number of difficult boxes) seems to be a better choice than normalizing against number of (foreground boxes + humongous amount of background boxes).<br>
Maybe something slightly more intuitive would be normalizing again 2*(number of foreground boxes) to take into account that there are not only foreground boxes processed by the focal loss but also a kind of equal “equivalent smoothed quantity” of background boxes.<br>
Hope it makes sense, that is just my “intuitive math” interpretation.</p>
Questions about one_batch() and padding

pattiJane: <p>Hello all,</p>
<p>SortSampler (validation) and <a href="https://github.com/fastai/fastai/blob/705840a6525b65fb34290dca3b5b45706c5dd040/fastai/text/data.py#L107" rel="noopener nofollow ugc">SortishSampler</a> (training) both act as follows:</p>
<blockquote>
<p>collate the  <code>samples</code>  in batches while adding padding with  <code>pad_idx</code> . If  <code>pad_first=True</code> , padding is applied at the beginning (before the sentence starts) otherwise it’s applied at the end.</p>
</blockquote>
<p>So does this mean the sequence length will be maximum sequence in batch? or each partition (e.g. train, valid) or whole dataset?</p>
<p>So, I have set  <code>pad_first=True</code>  and I wanted to check this by looking at several batches,</p>
<p>I have <a href="https://github.com/fastai/fastai/blob/705840a6525b65fb34290dca3b5b45706c5dd040/fastai/text/learner.py#L45" rel="noopener nofollow ugc">RNNLearner</a> as my model, and <code>x = model.data.one_batch()</code> shows a batch from  <code>training_dl</code>. Here is my output:</p>
<blockquote>
<p>tensor([[1, 2, 1,  …, 1, 1, 1],<br>
[0, 0, 0,  …, 4, 2, 1],<br>
[0, 0, 0,  …, 1, 1, 3],<br>
…,<br>
[0, 0, 0,  …, 2, 4, 3],<br>
[0, 0, 0,  …, 3, 1, 4],<br>
[0, 0, 0,  …, 3, 3, 1]])</p>
</blockquote>
<p>And the shape of this  tensor is :</p>
<blockquote>
<p>x[0].shape<br>
torch.Size([32, 11556])</p>
</blockquote>
<p>Where 32 is batch size and 11556 is the longest sequence in my dataset. And when I want to look at a batch from validation set, <code>y=model.data.one_batch(ds_type=2)</code></p>
<blockquote>
<p>(tensor([[1, 2, 1,  …, 1, 1, 1],<br>
[4, 1, 2,  …, 3, 2, 1],<br>
[0, 0, 0,  …, 3, 2, 4],<br>
…,<br>
[0, 0, 0,  …, 2, 1, 2],<br>
[0, 0, 0,  …, 3, 1, 3],<br>
[0, 0, 0,  …, 3, 3, 2]]),<br>
tensor([1, 0, 1, 0, 1, 1, 0, 1, 1, 0, 0, 1, 0, 1, 0, 1, 0, 0, 1, 1, 0, 0, 1, 1,<br>
0, 1, 0, 0, 1, 1, 0, 0]))</p>
</blockquote>
<blockquote>
<p>y[0].shape<br>
torch.Size([32, 11556])</p>
</blockquote>
<p>However, the longest sequence in the validation is around 6000 characters, so this tensor size does not make sense.</p>
<p>Can you please help me about these? I also want to look at other batches and not sure what it is the correct way to do that.</p>
<p>Thank you!</p>
How to share CPU memory in distributed training?

KaiHoo: <p>Hello, I am trying to train ImageNet on a 8GPU machine with DDP mode. However, my machine is not good at reading large scale small files. I have to make a tar file of the whole dataset (130Gb), read the tar file into memory and extract them in memory. I have a CPU memory of 360Gb. So it would be OK to use DataParallel mode. But it seems I cannot use DistributedDataParallel since I need to load the dataset 8 times. Is there any method that I can train the model with DDP mode?</p>
<p>Thanks!</p>

mrshenli: <p>One option is to use <a href="https://pytorch.org/docs/stable/notes/multiprocessing.html#reuse-buffers-passed-through-a-queue" rel="nofollow noopener"><code>torch.multiprocessing.Queue</code></a> as the shared memory. The main process can prepare multiple queues and then pass one queue to each DDP processes. The main process reads from the file and dispatch data items to the queue, while DDP processes wait on their own queue for that data item.</p>
<p>Another option is to split the tar file into multiple smaller pieces and let each DDP process read a different one.</p>

KaiHoo: <p>Hello! Thanks for your answer! I have one more question about the second option. Does that mean something like this?</p>
<p>I split the tar file into data_1.tar, data_2.tar, …, data_8.tar. For the k^{th} GPU, i.e., local_rank = k, the process read data_k.tar and build data loader with data_k.tar. Then I get 8 different data loaders (their data are different). In this case, I guess I should set shuffle=True and do not need a train sampler?</p>

mrshenli: <blockquote>
<p>I split the tar file into data_1.tar, data_2.tar, …, data_8.tar. For the k^{th} GPU, i.e., local_rank = k, the process read data_k.tar and build data loader with data_k.tar. Then I get 8 different data loaders (their data are different). In this case, I guess I should set shuffle=True and do not need a train sampler?</p>
</blockquote>
<p>Yes, but one caveat is that those input data splits need to generate the same number of input batches for DDP. If, say rank 0 processes 3 batches and rank 1 process 4 batches, rank 1 would hang on the last batch.</p>
How to create a self endecoder and decoder for semantic segmentation using pytorch

Zohaib: <p>Hi all!<br>
I am new in the field. I am trying to implement semantic segmentation. I used the pretrained models (as given below). It works fine. However, I am interested in creating self encoder and decoder. So that, I can modify only decoder for different applications. Can any one please suggest me any useful tutorial or pytorch library? Thank you.</p>
<blockquote>
<pre><code># Pretrained Model : **Fully Convolutional Network (FCN)**
fcn = models.segmentation.fcn_resnet101(pretrained=True).eval()
dlab = models.segmentation.deeplabv3_resnet101(pretrained=1).eval()
</code></pre>
</blockquote>
Captum tutorials for parameters attribution

TinfoilHat0: <p>Hi,</p>
<p>I’ve been looking into <a href="https://captum.ai/tutorials/" rel="nofollow noopener">Captum library</a> to do some parameter/neuron attribution. I.e., given a model and an input, quantify neurons’ impact on model’s prediction. Captum seems to have some tutorials for feature attribution but not any for parameter attribution. Is that the case or am I overlooking it ?</p>

Narine: <p>Hi <a class="mention" href="/u/tinfoilhat0">@TinfoilHat0</a>! Thank you for the question! What do you exactly mean by parameter ? Do you mean the weights of the neurons ?  Current attribution for neurons is meant for neuron / layer activation.</p>

TinfoilHat0: <p>yes thats what i meant. there are no tutorials for layer/neuron attribution as far as i could see.</p>

vivekm: <p>Hi <a class="mention" href="/u/tinfoilhat0">@TinfoilHat0</a>, you can take a look at this getting started <a href="https://captum.ai/tutorials/Titanic_Basic_Interpret" rel="nofollow noopener">tutorial</a> on the Titanic dataset, it covers example usage of Captum for layer and neuron attribution.</p>

TinfoilHat0: <p>Thanks Vivek. This had evaded my attention.</p>
Training a model with a pre trained network

Zacharie_SOUIOUED: <p>So I made model to do Super Resolution over MNIST dataset, I’ve already trained a classifier to whether or not it can recognize the number from the output of my SR model.<br>
I use the results of my classifier to comput the loss of my SR model and the question I have is this one :<br>
During the training loop</p>
<p>SRmodel.train()<br>
classifier.eval()<br>
ouptut = SRmodel(data)<br>
predictedLabel = classifier(output)<br>
loss = MyLoss(data, output, predictedLabel, label)<br>
loss.backward()</p>
<p>How do I know whether the backward propagation is occurring only over my SR model, or over my classifier, or both? (So far the output of my SR model are correct, but I might be missing something)</p>

ptrblck: <p>You could check the gradients in both models separately before and after the <code>backward</code> call via:</p>
<pre><code class="lang-python">for param in SRmodel.parameters():
    print(param.grad)
for param in classifier.parameters():
    print(param.grad)
</code></pre>
<p>These should be <code>None</code> before the first <code>backward</code> call, zeros after zeroing out the gradients via <code>optimizer.zero_grad()</code>, and contain valid values after the <code>backward</code> call.</p>

Zacharie_SOUIOUED: <p>thank you for your awnser, yes the gradients contain valid values after the backward call</p>
Pytorch conv_transpose2d and TF conv2d_trasnpose

m75: <p>I am converting a TF weight to Pytorch.<br>
I copied the weight for ConvTranspose but it produces the different results.</p>
<ol>
<li>
<p>To simulate padding=‘same’ in TF, Pytorch needs to zero-pad by 1x1 in some cases. Should it be before F.conv_transpose2d or after F.conv_transpose2d?</p>
</li>
<li>
<p>Why is the snippet below produces the reversed result?</p>
</li>
<li>
<p>Is there any extra step for Pytorch to match the behavior of TF?</p>
</li>
</ol>
<pre><code class="lang-auto">import tensorflow as tf
import torch
import torch.nn as nn
import torch.nn.functional as F
import numpy as np

dim = 4
inp = np.ones((1, dim, dim, 3))
filter = np.ones((4, 4, 3, 3))

# Pytorch [in, out, kernel_size[0] (h), kernel_size[1] (w)] 3,3,4,4
# TF      [filter_height, filter_width, out_channels, in_channels] 4,4,3,3

x = tf.placeholder(shape=(1, dim, dim, 3), dtype=tf.float64)
# x = tf.layers.conv2d_transpose(x, 3, kernel_size=4, strides=1, padding='SAME')
y = tf.nn.conv2d_transpose(x, filter, output_shape=(1, dim, dim, 3), strides=(1, 1, 1, 1))

with tf.Session() as sess:
    y = sess.run([y], feed_dict={x: inp})[0]

    y = np.transpose(y, (0, 3, 1, 2))
    print(y.shape)
    print('sum', y.sum())
    print(y)


###################

inp = torch.tensor(np.ones((1, 3, dim, dim)))
weight = torch.tensor(np.ones((3, 3, 4, 4)))

inp = F.pad(inp, [0, 1, 0, 1], mode='constant', value=0)
y = F.conv_transpose2d(inp, weight, bias=None, padding=2, output_padding=0)
y = y.numpy()
print(y.shape)
print('sum', y.sum())
print(y)

</code></pre>
<pre><code class="lang-auto">sum 1296.0
[[[[12. 18. 24. 18.]
   [18. 27. 36. 27.]
   [24. 36. 48. 36.]
   [18. 27. 36. 27.]]
  [[12. 18. 24. 18.]
   [18. 27. 36. 27.]
   [24. 36. 48. 36.]
   [18. 27. 36. 27.]]
  [[12. 18. 24. 18.]
   [18. 27. 36. 27.]
   [24. 36. 48. 36.]
   [18. 27. 36. 27.]]]]
(1, 3, 4, 4)
sum 1296.0
[[[[27. 36. 27. 18.]
   [36. 48. 36. 24.]
   [27. 36. 27. 18.]
   [18. 24. 18. 12.]]
  [[27. 36. 27. 18.]
   [36. 48. 36. 24.]
   [27. 36. 27. 18.]
   [18. 24. 18. 12.]]
  [[27. 36. 27. 18.]
   [36. 48. 36. 24.]
   [27. 36. 27. 18.]
   [18. 24. 18. 12.]]]]

</code></pre>

eastchun: <p>Didn’t this one work?</p>
<pre><code>  inp = F.pad(inp, [1, 0, 1, 0], mode='constant', value=0)</code></pre>
(Cross entropy loss) accuracy on siamese net always seems 50%

John_J_Watson: <p>I have a siamese network which I am training on positive and negative samples and minimizing the cross entropy loss between the outputs(pos/neg) with their corresponding targets. However, the test accuracy ALWAYS seems 50% (i.e 50% of the test sets classified as correct, presumably the positive samples) and I am not sure how to trouble shoot this issue <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"></p>
<p>I was wondering of the pytorch community had any tips for me to trouble shoot this as I seem lost!</p>

Isaac_Kargar: <p>Can you please put your code here? Then we can help more.</p>
Torch.stack yielding wrong dimensions

John_J_Watson: <p>I am quite new to pytorch, and I am trying to create data within a dataloader and my code looks like so:</p>
<pre><code class="lang-auto">a=[]
.. within a for loop
self.a.append(torch.stack([b[ith_idx][j], \
b[ith_idx][rnd_dist], \
b[rnd_cls_idx][rnd_dist_rnd_cls]]\
))
self.c.append([1,0])
</code></pre>
<p>where, b is a python list of tensors. For example, the first element of  <code>b</code>  has shape  <code>torch.Size([46, 3, 512, 512])</code> .</p>
<pre><code class="lang-auto">        self.a = torch.stack(self.a)
        self.c = torch.tensor(self.c)
</code></pre>
<p>and I notice I have shapes of  <code>[500,3,3,512,512]</code>  and [500,2] for  <code>a</code>  and  <code>b</code> , while I was expecting  <code>500,3,3,512,512</code>  and  <code>500</code>  as tensor shapes.</p>
<p>Any pointers as to why this is happening would be helpful.</p>

ptrblck: <p>Since you are passing two values to <code>self.c</code>, your expected shape would be <code>[N, 2]</code>.<br>
Could you explain a bit what your use case is and how <code>self.c</code> should be stacked to create a <code>[500]</code> tensor?</p>

John_J_Watson: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>: Actually, your explanation makes sense, it does indeed seem correct. Thank you for your time!</p>
Set elements of a matrix prior to a specific index within each row

nima_rafiee: <p>Hi</p>
<p>I have an index vector  like:<br>
idx_v = [1,2,4,0,3]<br>
A = [1, 1, 1, 1, 1<br>
1, 1, 1, 1, 1<br>
1, 1, 1, 1, 1<br>
1, 1, 1, 1, 1<br>
1, 1, 1, 1, 1]</p>
<p>I want to set all elements of matrix A located before elements of idx_v vector to zero within each row<br>
like:<br>
At =  [0, 1, 1, 1, 1<br>
0, 0, 1, 1, 1<br>
0, 0, 0, 0, 1<br>
1, 1, 1, 1, 1<br>
0, 0, 0, 1, 1]</p>
<p>Is there any way to do it without using For loops?</p>

ptrblck: <p>This code should work:</p>
<pre><code class="lang-python">idx_v = torch.tensor([1,2,4,0,3])
A = torch.tensor([[1, 1, 1, 1, 1],
                  [1, 1, 1, 1, 1],
                  [1, 1, 1, 1, 1],
                  [1, 1, 1, 1, 1],
                  [1, 1, 1, 1, 1]])

torch.arange(idx_v)
res = torch.zeros_like(A).scatter_(1, idx_v.unsqueeze(1), 1)
res = torch.cumsum(res, 1)
print(res)
&gt; tensor([[0, 1, 1, 1, 1],
          [0, 0, 1, 1, 1],
          [0, 0, 0, 0, 1],
          [1, 1, 1, 1, 1],
          [0, 0, 0, 1, 1]])
</code></pre>

nima_rafiee: <p>Thanks it is working</p>
Every time I input data, I want to call the method for each input

Hyeongu_Oh: <pre><code class="lang-auto">    for data in loader:
        train = data['input'][:,:-2700]
        train = train.to(DEVICE)

        target = data['input'][:,-2700:]
        target = target.to(DEVICE)

        model.set_model(data['input_num'])

        output = model(data)
</code></pre>
<p>Every time I input data, I want to call the set_model method for each input.<br>
However, Error occurs because input_num, which is a single value, changes to batch_size.<br>
Is there a good way?</p>
<p>For example, I set the batch_size to 8, and I want to call each of the eight methods in one batch.</p>

ptrblck: <aside class="quote no-group" data-username="Hyeongu_Oh" data-post="1" data-topic="81930">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hyeongu_oh/48/14475_2.png" class="avatar"> Hyeongu_Oh:</div>
<blockquote>
<p>For example, I set the batch_size to 8, and I want to call each of the eight methods in one batch.</p>
</blockquote>
</aside>
<p>You could use a loop, if you want to call each value separately.<br>
I don’t know, what <code>set_model</code> means, but if it cannot accept batched data, you would have to fallback to the loop (or use a batch size of 1).</p>
Model starts to overfit right away

Fatemeh_Darbehani: <p>Hi, I am using a pretrianed 3d resnet on brain mri images to do binary classification. My problem is that validation error and auc never decreases while training error decreases very fast and auc of training goes up to 1.</p>
<p>Im using a batch size of 2 and SGD optimizer with lr 0.001. I also do minMax normalization on the input.<br>
Here is what the train/val loss graph looks like:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/b/1b286d8439f3b9684d0cda4fce21c644ad4caf70.png" data-download-href="https://discuss.pytorch.org/uploads/default/1b286d8439f3b9684d0cda4fce21c644ad4caf70" title="Screen Shot 2020-05-17 at 11.42.02 AM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/b/1b286d8439f3b9684d0cda4fce21c644ad4caf70_2_690x379.png" alt="Screen Shot 2020-05-17 at 11.42.02 AM" data-base62-sha1="3SfvudFrxE3Wi6mSJ7XIYWrKSvm" width="690" height="379" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/b/1b286d8439f3b9684d0cda4fce21c644ad4caf70_2_690x379.png, https://discuss.pytorch.org/uploads/default/optimized/3X/1/b/1b286d8439f3b9684d0cda4fce21c644ad4caf70_2_1035x568.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/1/b/1b286d8439f3b9684d0cda4fce21c644ad4caf70_2_1380x758.png 2x" data-dominant-color="FAF8F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2020-05-17 at 11.42.02 AM</span><span class="informations">1698×934 79.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

Fatemeh_Darbehani: <p>This is a sample of my data:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/8/7/870011a8570613cb6f453926b6dddbacabafdf01.png" alt="patient('OAS30001',)_visit1" data-base62-sha1="jggBAzzrgSvnWIQduiY4F2GdCdb" width="640" height="480"></p>

ptrblck: <p>You could try to add more regularization via (more) dropout, weight decay etc.<br>
Data augmentation might also help.</p>
Row-Wise Dot Product

Jack_Rolph: <p>Suppose I have two tensors:</p>
<pre><code class="lang-auto">a = torch.randn(10, 1000, 1, 4)
b = torch.randn(10, 1000, 6, 4)
</code></pre>
<p>Where the third index is the index of a vector.</p>
<p>I want to take the dot product between each vector in <code>b</code> with respect to the vector in <code>a</code>.</p>
<p>To illustrate, this is what I mean:</p>
<pre><code class="lang-auto">dots = torch.Tensor(10, 1000, 6, 1)
for b in range(10):
     for c in range(1000):
           for v in range(6):
            dots[b,c,v] = torch.dot(b[b,c,v], a[b,c,0]) 
</code></pre>
<p>How would I achieve this using torch functions?</p>

ptrblck: <p>This code should work:</p>
<pre><code class="lang-python">a = torch.randn(10, 1000, 1, 4)
b = torch.randn(10, 1000, 6, 4)

dots = torch.Tensor(10, 1000, 6, 1)
for x in range(10):
     for y in range(1000):
         for z in range(6):
             dots[x,y,z] = torch.dot(b[x,y,z], a[x,y,0]) 


ret = torch.matmul(b, a.permute(0, 1, 3, 2))
print((ret - dots).abs().max())
&gt; tensor(9.5367e-07)
</code></pre>
Improving Model Test Accuracy further when it is flat

amitkayal: <p>Hello,</p>
<p>I have a classification model and have used a mini version of ResNext50. It has ReduceLROnPlateau also included.</p>
<p>Model  accuracy in test data is stuck with 79% for last 15 epoch although LR getting reduced. My dataset is not skewed and my dataset has variable shape. So the dataloader has resizing to 128. Model has bit over fitting also. Any guidance please how to get out of this plateau?</p>
<p>class1 =&gt; height min :  50 width min :  59<br>
class1 =&gt; height max :  500 width max :  500<br>
class2 =&gt; height min :  87 width min :  78<br>
class2 =&gt; height max :  1023 width max :  1024<br>
class3 =&gt; height min :  32 width min :  55<br>
class3 =&gt; height max :  500 width max :  500</p>
<p>Epoch: 69<br>
Train Loss: 0.320908 Acc: 0.8598<br>
Elapsed 1005.24s, 14.36 s/epoch, 0.38 s/batch, ets 0.00s</p>
Keep num_layer*num_directions dimension for LSTM when concatenating final forward an final backward hidden state in BiLSTM

Talita: <p>Suppose that I have the following LSTM:</p>
<pre><code class="lang-auto">output, (hidden_state, cell_state) = nn.LSTM(5,100, num_layers = 1, bidirectional = True, batch_first = False)
</code></pre>
<p>Where hidden_state is of shape</p>
<pre><code class="lang-auto">torch.Size([2, 10, 100])
</code></pre>
<p>And that I want to concatenate the final forward and final backward layer of <code>hidden_state</code>:</p>
<pre><code class="lang-auto">torch.cat((hidden_state[-2,:,:], hidden_state[-1,:,:]), dim = 1))
</code></pre>
<p>Which results to the shape</p>
<pre><code class="lang-auto">torch.Size([10, 100])
</code></pre>
<p>How can I do this concatenation without losing the first dimension of hidden_state, which is here 2 (1 x 2)?</p>

ptrblck: <p>You could use range indices to keep the dimension or simply <code>unsqueeze</code> the result:</p>
<pre><code class="lang-python">x = torch.randn([2, 10, 100])
res = torch.cat((x[0:1,:,:], x[1:2,:,:]), dim = 1)
</code></pre>

Talita: <p>Thanks!  But that would still result to</p>
<pre><code class="lang-auto">torch.Size([1, 20, 100])
</code></pre>
<p>and not to <code>torch.Size([2, 20, 100])</code> , right?</p>

ptrblck: <p>Yes, because you are concatenating in dim1.<br>
I’m not sure what your use case if, but your desired output shape contains two times the number of sliced elements. Would you like to repeat some values or how would you like to achieve this shape?</p>
If two different way to modify grad manually is equivalent?

hhxx: <p>I think there are two different way to modify gradients manually:</p>
<ol>
<li>use hook</li>
<li>set weight.grad.data directly</li>
</ol>
<p>demo scripts like this:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import pdb
batch_size = 8

class ModifyGrad(nn.Module):
    def __init__(self):
        super(ModifyGrad, self).__init__()
        self.fc = nn.Linear(100, 10)

    def forward(self, x):
        return self.fc(x)


model = ModifyGrad()
optimizer = torch.optim.SGD(model.parameters(), 0.1, \
    momentum=0.09, weight_decay=0.0001)

x = torch.randn(batch_size, 100)
out = model(x).mean()


# modify grad by hook
h = model.fc.weight.register_hook(lambda grad: grad+0.1 )
model.zero_grad()
out.backward()
optimizer.step()

# modify grad by set weight.grad.data directly
model.zero_grad()
out.backward()
model.fc.weight.grad.data = model.fc.weight.grad.data + 0.1
optimizer.step()
</code></pre>
<p>I wonder if above two methods is equivalent?</p>
<p>In addition, the hook-method seems is officially recommended. if above two methods is equivalent, what is the advantage of hook-method?</p>

ptrblck: <p>Use the hook approach, as the manipulation of the <code>.data</code> attribute is not recommended and might yield unwanted side effects.</p>

hhxx: <p>Can you give me some detail explanation about the side effects?</p>
<p>At least in my example above, in two different ways, optimzer.step will yield same update on model.fc.weight</p>

ptrblck: <p><a class="mention" href="/u/alband">@albanD</a> explains it <a href="https://discuss.pytorch.org/t/using-tensor-data/79640/4">here</a> better than I could. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Convolutional MultiheadAttention?

Luke_Taylor: <p>Hi,</p>
<p>I am fairly new to PyTorch and Attention. As I understand MultiheadAttention maps a fixed sequence of vectors of length T to an embedded sequence of length T. I would be interested in constructing this operation as a convolutional operation e.g. convolving the MultiheadAttention operation over a sequence of length N  &gt; T. I understand that this would produce a matrix (amount_of_slides x embedded_sequence_of_length_T).</p>
<p>The reason that I am interested in doing so, is that I have a sequence of length N, where contiguous subsequences of length T &lt; N have some sort of relation. I want to map the original sequence of length N to a new sequence. It however (1) seems computational expensive to use MultiheadAttention on the entire sequence, considering that there are only relations in the subsequences and (2) the sequence can be of variable length, hence my curiosity in using the MultiheadAttention as a convolutional operation.</p>
<p>Does anyone know of an implementation of this?</p>
<p>Thanks!</p>
Using Two Data Iterators Under One Loop

Hasan_Sait_ARSLAN1: <p>Dear Pytorch enthusiasts,</p>
<p>I have a question for you.</p>
<p>I have two datasets, and would like to use a different data iterator for each of them to train one model.</p>
<p>To do so, I have used DataLoader for each of them to get data iterators, and <strong>shuffle</strong> argument was set to <strong>True</strong>.</p>
<p>Then, I have used them as following under one loop, for each epoch:</p>
<p><code>for idx, datum in enumerate(zip(data_iterator1, data_iterator2)):</code></p>
<p>Here, the length of <strong>data_iterator1</strong> is <strong>~28</strong> times longer than the length of <strong>data_iterator2</strong>.</p>
<p>Here I have the following questions in my mind:</p>
<ol>
<li>When the loop ends? Does it end when the loop sees all the batches belonging to <strong>data_iterator2</strong>, or it ends until seeing all the batches belonging to <strong>data_iterator1</strong>?</li>
<li>If the loop ends after seeing all the batches belonging to <strong>data_iterator2</strong>, at the next epoch, are the data belonging to both data iterators shuffled again?</li>
</ol>
<p>I would be thankful if someone here would clarify these 2 questions for me.</p>
<p>Thank for your time.</p>
<p>Best regards,</p>

JuanFMontesinos: <p>zip by default get exhausted when the 1st generator get exhausted (it’s python’s).</p>
<p>If I’m not wrong (I don’t remember) zip will keep exhausted untill you rebuild the object. if you do so both iterators will restart.</p>

Hasan_Sait_ARSLAN1: <p>Much obliged, for answering Juan!</p>
<p>Could you explain in detail, what do you mean by “both iterators will restart”?</p>

JuanFMontesinos: <p>When you call zip you are invoking the iterator of the object.<br>
Depending on how it has been code it will work one way or another.<br>
Here you have a explanation.<br>
<a href="https://anandology.com/python-practice-book/iterators.html" class="onebox" target="_blank" rel="nofollow noopener">https://anandology.com/python-practice-book/iterators.html</a></p>
<p>Objects which are supposed to be iterated several times are coded in such a way that, once they get exhausted, they will reset the iterable so that you can iterate over them without reinstanciating.</p>

Hasan_Sait_ARSLAN1: <p>So it means, when they are restarted, they are not shuffled?</p>
How to efficiently index tensor with a batch

Abe: <p>Having a tensor with dimension (B,C,w,h). I want to sort the channels ( C ) based on the average pooled values of the tensor.<br>
I have this code for that, but the indexing part is not working</p>
<pre><code class="lang-auto">import torch
torch.manual_seed(0)
tensor = torch.rand(2,4,3,3)
# average pooling
polling = torch.nn.AdaptiveAvgPool2d(1)
tensor_pooled = polling(tensor)
#sorting the averaged pooled values
sorted_tensor_pooled, sorted_indx = torch.sort(tensor_pooled, 1, descending=True)

# trying to sort the original tensor based on the index from the sorted avg_pool. But this is not working 
sorted_tensor = tensor[sorted_indx]
</code></pre>
<p>How could I do this without using loop ?<br>
Thanks</p>
Is there a backend difference when running a scripted model in python vs c++?

vladium: <p>Greetings. A couple of questions regarding x86 inference backend(s):</p>
<ol>
<li>Assume that I produced a model via torch.jit.script(). Is it correct to think that essentially the same JIT runtime + BLAS implementation are used regardless of whether this model is subsequently evaluated via the python or the c++ interface? (assuming that the c++ app is compiled and linked against lib{torch, torch_cpu, c10}.so that are part of the pytorch distribution)</li>
</ol>
<p>Put differently, other than being able to run in a python-less environment, are there benefits to doing model <em>inference</em> via the c++ API?</p>
<ol start="2">
<li>I read everything I could find about fbgemm but remain confused as to when it actually kicks in. Does it get used only if the model contains fused or quantized ops?</li>
</ol>
<p>Thank you.</p>
Class imbalance with WeightedRandomSampler

dachosen1: <p>I have a dataset with two classes and a severe class imbalance.</p>
<p>I tried to use the weighed sampler to equalize the classes for training but only class showed during training. What did i do wrong?</p>
<p>My dataset contains individual CSV , not images.</p>
<pre><code class="lang-auto">def csv_loader(path: str) -&gt; torch.Tensor:
    data = np.array(pd.read_csv(path, header=None))
    sample = torch.from_numpy(data)
    return sample

train_dataset = DatasetFolder(root=train_dir, loader=csv_loader, extensions=".csv")

batch_size = config.MODEL_PARAM['BATCH_SIZE']
weights =  [0.5,0.5]
sampler = torch.utils.data.sampler.WeightedRandomSampler(weights, batch_size)
    train_data_loader = DataLoader(
        train_dataset, batch_size=config.MODEL_PARAM['BATCH_SIZE'], 
        sampler = sampler, num_workers=4, drop_last=True
    )
</code></pre>

ptrblck: <p>The <code>weigths</code> tensor should contain the weight for each sample in your dataset, nor the class weights only.<br>
Have a look at <a href="https://discuss.pytorch.org/t/how-to-handle-imbalanced-classes/11264/2">this example</a> which shows a dummy use case.</p>

dachosen1: <p>That’s very helpful here’s the solution with a datafolder:</p>
<pre><code class="lang-auto">def sample_weight(data_folder):
    
    class_sample_count = np.array([len([i for i in data_folder.targets if i == t]) for t in range(0, len(data_folder.classes))])
    weight = 1 / class_sample_count
    samples_weight = np.array([weight[t] for t in data_folder.targets])
    samples_weight = torch.from_numpy(samples_weight)
    samples_weight = samples_weight.double()
    sampler = WeightedRandomSampler(samples_weight, len(samples_weight))
    return sampler

train_dataset = DatasetFolder(root=train_dir, loader=csv_loader, extensions=".csv")
train_sample_weight = sample_weight(train_dataset)
</code></pre>
Normalizing non-image tensors

nlam: <p>I have a numpy ndarray containing 4 floats in 1-dim, looks like [a, b, c, d] for example. I am trying to use the transforms functions in order to normalize my data. It currently looks like this</p>
<pre><code class="lang-auto">transform = transforms.Compose([
        transforms.ToTensor(),
        transforms.Normalize(get_mean(), get_std())
])
</code></pre>
<p>and in my custom dataset class, I call the transformation as such in the <strong>getitem()</strong></p>
<pre><code class="lang-auto"> return self.transform(result)
</code></pre>
<p>However, I get the following error message</p>
<pre><code class="lang-auto">raise ValueError('pic should be 2/3 dimensional. Got {} dimensions.'.format(pic.ndim))
ValueError: pic should be 2/3 dimensional. Got 1 dimensions.
</code></pre>
<p>I understand that torchvision is mainly meant for image tensors, but it seems like there should not be a difference since both are just tensors of eventual numerical data. Is there anyway that I can make this work, or do I have to normalize manually. Thank you for your time and assistance.</p>

sid-ls: <p>There isn’t a difference if you’re using an “image tensor”. The error is because the function just expects a 2 or 3 dimensional tensor</p>

nlam: <p>so is there no way to feed in a 1-dim tensor?</p>
Code the loss function

Anuj_Daga: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/9/a932fadd98b896e3f2d76c7b4fffc176238793ac.png" data-download-href="https://discuss.pytorch.org/uploads/default/a932fadd98b896e3f2d76c7b4fffc176238793ac" title="loss func"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/9/a932fadd98b896e3f2d76c7b4fffc176238793ac_2_690x117.png" alt="loss func" data-base62-sha1="o8NUP9aSs7Q5P1vu3bcMnZ5LQss" width="690" height="117" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/9/a932fadd98b896e3f2d76c7b4fffc176238793ac_2_690x117.png, https://discuss.pytorch.org/uploads/default/optimized/3X/a/9/a932fadd98b896e3f2d76c7b4fffc176238793ac_2_1035x175.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/a/9/a932fadd98b896e3f2d76c7b4fffc176238793ac.png 2x" data-dominant-color="EFEFEF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">loss func</span><span class="informations">1311×223 55.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
where<br>
Let P_X denote the data distribution, and P_G denotes the encoded training distribution, where Q is the encoders and G is the decoders, X ∼ P_X and Z ∼ Q(Z |X).<br>
Considering the sparsity of the transition matrix P, we focus on non-zero elements in P to speed up our model. we use the transition matrix P as the input feature X. The reconstruction process will make the nodes with similar neighbourhoods have similar latent representations.</p>
After 1.4->1.5 how to do backward() twice

Landu: <p>hello i found that 1.5 upgrade has change of autograd<br>
and also there is a tutorial for me who doesn’t know what’s wrong(<a href="https://github.com/pytorch/pytorch/releases" rel="nofollow noopener">https://github.com/pytorch/pytorch/releases</a>)  issue:[torch.optim optimizers changed to fix in-place checks for the changes made by the optimizer]</p>
<pre><code class="lang-auto">def model(input, target, param):
    return `(input * param ** 2 - target).norm()`

param = torch.randn(2, requires_grad=True)
input = torch.randn(2)
target = torch.randn(2)
sgd = optim.SGD([param], lr=0.001)
loss = model(input, target, param)
loss.backward(retain_graph=True)
sgd.step()
loss.backward()
param.grad
</code></pre>
<p>before 1.5, ↑ codes works. but after 1.5</p>
<pre><code class="lang-auto">def model(input, target, param):
    return (input * param ** 2 - target).norm()

param = torch.randn(2, requires_grad=True)
input = torch.randn(2)
target = torch.randn(2)
sgd = optim.SGD([param], lr=0.001)
loss = model(input, target, param.clone())
loss.backward(retain_graph=True)
sgd.step()
loss.backward()
param.grad
</code></pre>
<p>i have to put param.clone() into model</p>
<p>but in reality, i don’t use a model like above, i just put only inputs into my model.</p>
<pre><code class="lang-auto">class test1(nn.Module):
    def __init__(self):
        super(test1, self).__init__()
        self.layer1 = nn.Linear(10, 1)

    def forward(self, x):
        x = self.layer1(x)
        return x

t = test1().to(device)

optimizer1 = torch.optim.Adam(t.parameters(), lr=0.001)

for i, (images, labels) in enumerate(data_loader):
    images = images.view(batch_size, -1)[:, :10].to(device)
    labels = labels.float().to(device)

    a = t(images)
    loss = criterion(a, labels)

    optimizer1.zero_grad()
    loss.backward(retain_graph=True)
    optimizer1.step()
    loss.backward()
</code></pre>
<p>and ↑ this is my test code.</p>
<p>how can i change ↑ this code?</p>

albanD: <p>Hi,</p>
<p>The problem is that the original code here was computing wrong gradients.</p>
<p>You can modify this quite easily by overriding the linear forward function for this case:</p>
<pre><code class="lang-auto">class MyLinear(nn.Linear):
    def forward(self, input):
        return F.linear(input, self.weight.clone(), self.bias.clone())

# And use this one later:
self.layer1 = nn.MyLinear(10, 1)
</code></pre>
Torch.cuda.is_available() is false after installing PyTorch via conda

Nihang_Fu: <p>Hi,</p>
<p>Does CUDA install automatically by installing PyTorch? I run the command to install PyTorch following <a href="https://pytorch.org/:" rel="nofollow noopener">https://pytorch.org/:</a></p>
<p>conda install pytorch torchvision cudatoolkit=10.0 -c pytorch</p>
<p><code>torch.cuda.is_available</code> is false after this command.<br>
However, I see a similar discussion in <a href="https://discuss.pytorch.org/t/will-conda-install-pytorch-torchvision-c-pytorch-also-install-cuda-and-cudnn/19717" class="inline-onebox">Will "conda install pytorch torchvision -c pytorch" also install CUDA and cuDNN?</a>, and they said  yes…</p>
<p>What’s wrong?</p>

Isaac_Kargar: <p>You have to check if your nvidia-driver and cuda versions are compatible with the pytorch version you want to install. I have pytorch=1.2 with cuda 10 and nvidia-driver 410 on my system. I think you can use this command if your nvidia driver and cuda versions are as I mentioned:<br>
<code>conda install pytorch==1.2.0 torchvision==0.4.0 cudatoolkit=10.0 -c pytorch</code></p>

Nihang_Fu: <p>You are right. Thank you!</p>
Keypoint localization

sab3rclaw: <p>I am trying to create a Keypoint detection program(with 12 points). I would really appreciate if someone can guide me in which model will be the best suit for this. I tried custom CNN models but accuracy is a problem in them. So is there any state of the art model that I can utilize? Also if there is one can someone please provide a way to utilize it as well.</p>
<p>Thanks</p>
While using torch_xla.it shows the following error

Sunil_Sharma1: <pre><code>518   torch_xla._XLAC._xla_step_marker(
519       torch_xla._XLAC._xla_get_default_device(), [],
</code></pre>
<p>– 520       wait=xu.getenv_as(‘XLA_SYNC_WAIT’, bool, False))<br>
521   # Only emit metrics from the first local device index, to avoid emitting the<br>
522   # same values from different threads.</p>
<p>RuntimeError: Failed precondition: From /job:tpu_worker/replica:0/task:0:<br>
The TPU system has not been initialized.<br>
[[{{node XRTCompile}}]]</p>

ailzhang: <p>Hi, for all pytorch/xla (or pytorch on tpu) questions, please open an issue in <a href="https://github.com/pytorch/xla" rel="nofollow noopener">https://github.com/pytorch/xla</a>.<br>
Thanks!</p>
std::vector<torch::Tensor> turns into nans

AlconDivino: <p>Hello there. I really need some help on the topic.</p>
<p>first i am using libtorch-1.4.0 on macos and Linux.</p>
<p>Now i get a weird error when creating tensors with from_blob via “loading function” looking somewhat like this:</p>
<pre><code class="lang-auto">void compute(std::vector&lt;torch::Tensor&gt; &amp;output)
{
    for(int i = 0; i &lt; batch_size; i++)
    {
        std::vector&lt;float&gt; vec(1 * 4 * 60 * 60);
        for(auto&amp; v : vec)
        {
            // assigning some values after computation etc
        }
        
        // creating tensor
        output.emplace_back(torch::from_blob(vec.data(), {1, 4, 60, 60}) );

    }

    // Check for nan values
    for(size_t i = 0; i &lt; out.size(); i++)
    {
        if(at::isnan(out[i]).any().item&lt;bool&gt;() )
        {
             throw std::string("Error in compute");
        }
}

</code></pre>
<p>No error is thrown but when i check after the function returns, I get an error message on linux for some reason. sometimes macos fails afterwards but most of the time it runs without errors (maybe silent ones tho)</p>
<p>The constructor calling the compute() function looks like this</p>
<pre><code class="lang-auto">Environment()
{
    compute(obs_storage);
   
    for(size_t i = 0; i &lt; storage.size(); i++)
    {    
         if(at::isnan(out[i]).any().item&lt;bool&gt;() )
        {
             throw std::string("Error in compute"); // ERROR for some reason
        }
    }
}
</code></pre>
<p>I dont really understand the problem there. because i call the exact same function twice and i get completely different results</p>

SimonW: <aside class="quote no-group" data-username="AlconDivino" data-post="1" data-topic="81853">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alcondivino/48/19902_2.png" class="avatar"> AlconDivino:</div>
<blockquote>
<p><code>vec.data()</code></p>
</blockquote>
</aside>
<p>from_blob does not copy, yet the pointer from vec.data() can’t be relied on after vec is freed.</p>
CUDA out of memory. Tried to allocate 538.00 MiB (GPU 0; 11.00 GiB total capacity; 230.80 MiB already allocated; 8.53 GiB free; 242.00 MiB reserved in total by PyTorch)

Fewen: <p>Hello everyone!<br>
I tried to run the camvid project of the FastAI course (<a href="https://course.fast.ai/videos/?lesson=3" rel="nofollow noopener">https://course.fast.ai/videos/?lesson=3</a>, <a href="https://nbviewer.jupyter.org/github/fastai/course-v3/blob/master/nbs/dl1/lesson3-camvid.ipynb" rel="nofollow noopener">https://nbviewer.jupyter.org/github/fastai/course-v3/blob/master/nbs/dl1/lesson3-camvid.ipynb</a>) on my pc.<br>
(specs: Windows 10 Education 64 bit, processor Intel® Core™ i7-7700K CPU @ 4.20GHz, 4200 MHz,<br>
GPU: Nvidia GeForce GTX 1080 Ti 11GB )</p>
<p>Its my first time running any kind of cnn/unet and for me the error message doesnt really make sense since it tries to allocate 538 MiB and it wont work altough 8.53 GiB is free.<br>
Is it because PyTorch has only reserved 242 MiB ?<br>
I checked the other related forum topics but all of the have “Tried to allocate X GiB and &lt;X GiB is free”.<br>
And batchsize reduction doesnt solve the problem for me like mentioned in the other forum topics.</p>
<p>Anyone has had the same problem?</p>
<p>Best regards!</p>

ptrblck: <p>What was your initial batch size and what is the current one?<br>
Also, did you make sure that your GPU is empty and no other processes use device memory?</p>

Fewen: <p>My inital batch size was 8 but I tried it with 4 and 2 and had the same issue for both of them.<br>
Yes before im running the code I clear the GPU memory with torch.cuda.empty_cache()<br>
Is it maybe because Im running it on windows?</p>

ptrblck: <p><code>torch.cuda.empty_cache()</code> will only clear the PyTorch memory cache on the device.<br>
I meant you should check via <code>nvidia-smi</code>, if other processes are using the GPU.<br>
Also, if a batch size of 1 doesn’t fit on the GPU, you might need to use <code>torch.utils.checkpoint</code> to trade compute for memory.<br>
Although it would be surprising to see a FastAI lecture code would need a bigger GPU.</p>

Fewen: <p>Thanks for the help!<br>
I guess there was a problem with my Anaconda version and the fastAi library together with windows.<br>
I solved it by buying a new ssd where I installed the new ubuntu 20.04 and it worked first try with batch size of 4.</p>