akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
BCEWithLogitsLoss calculating pos_weight

ni_tempe: <p>I have a neural network as below for binary prediction. My classes are heavily imbalanced and class 1 occurs only 2% of times. Showing last few layers only</p>
<pre><code class="lang-auto">self.batch_norm2 = nn.BatchNorm1d(num_filters)

self.fc2 = nn.Linear(np.sum(num_filters), fc2_neurons)

self.batch_norm3 = nn.BatchNorm1d(fc2_neurons)

self.fc3 = nn.Linear(fc2_neurons, 1)
</code></pre>
<p>My loss is as below. Is this a correct way to calculate <code>pos_weight</code> parameter? I looked into official documentation at [this] and it shows that <code>pos_weight</code> needs to have one value for each class for multiclass classification. Not sure if for the binary class it is a difference scenario. I tried to input 2 values and I was getting error<br>
(<a href="https://pytorch.org/docs/stable/generated/torch.nn.BCEWithLogitsLoss.html#torch.nn.BCEWithLogitsLoss" class="inline-onebox" rel="noopener nofollow ugc">BCEWithLogitsLoss — PyTorch 1.11.0 documentation</a>)</p>
<pre><code class="lang-auto">BCE_With_LogitsLoss=nn.BCEWithLogitsLoss(pos_weight=class_wts[1]/class_wts[0])
</code></pre>

mMagmer: <p>BCE is designed for binary classification, but it can be used in multi-label not multi-class classification.<br>
what is the error?<br>
only thing left is to have target with same shape of the output in one-hot format.</p>

ni_tempe: <p>i am not using <code>bce</code>. I am using <code>BCE_With_LogitsLoss</code> for binary classification. Please confirm that my <code>pos_weight</code> calculation is correct</p>

KFrank: <p>Hi Ni!</p>
<aside class="quote no-group" data-username="ni_tempe" data-post="1" data-topic="146336" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/n/e9c0ed/48.png" class="avatar"> ni_tempe:</div>
<blockquote>
<pre><code class="lang-auto">BCE_With_LogitsLoss=nn.BCEWithLogitsLoss(pos_weight=class_wts[1]/class_wts[0])
</code></pre>
</blockquote>
</aside>
<p>The question is what are <code>class_wts[1]</code> and <code>class_wts[0]</code>?</p>
<p>A common choice for <code>pos_weight</code> would be<br>
<code>num_negative_samples / num_positive_samples</code>.  (This is<br>
because you want to reweight the positive samples so that they<br>
have about the same influence as the negative samples.)</p>
<p>If <code>class_wts[1]</code> were <code>num_positive_samples</code> (or proportional<br>
to it), your <code>pos_weight</code> would be upside down.</p>
<p>Anyway, if “class-1” occurs about 2% of the time, you would want<br>
<code>pos_weight = (1.0 - 0.02) / 0.02</code> (which is <code>49.0</code> and is almost<br>
equal to <code>50.0</code>).</p>
<p>Best.</p>
<p>K. Frank</p>

ni_tempe: <p>that makes sense. I need to switch <code>pos_weight</code>. My confusion was whether I need one value per class as shown at the official documentation. But that doesnt seem to be the case at least for binary prediction…</p>
Overwriting PyTorch DataLoader shuffle?

Alison_Ma: <p>Is it possible to write a custom dataloader / are there any examples? I’d like to customize the shuffle function.</p>

mMagmer: <p>hi,<br>
you can use custom sampler in <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader" rel="noopener nofollow ugc"><code>torch.utils.data.DataLoader</code></a></p>
<blockquote>
<ul>
<li><strong>sampler</strong> (<a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.Sampler" rel="noopener nofollow ugc"><em>Sampler</em></a> <em>or</em> <em>Iterable</em> <em>,</em> <em>optional</em>) – defines the strategy to draw samples from the dataset. Can be any <code>Iterable</code> with <code>__len__</code> implemented. If specified, <code>shuffle</code> must not be specified.</li>
</ul>
</blockquote>

Alison_Ma: <p>Thanks! Would you recommend overwriting both the sampler and batch_sampler? Or is one sufficient? I plan to use a batch size higher than 1.</p>

mMagmer: <p>i think setting up <code>sampler</code> is enough.</p>

nivek: <p>In <code>DataLoader</code>, <code>sampler</code> and <code>batch_sampler</code> are mutually exclusive.<br>
The option is to either use <code>sampler</code> with <code>batch_size &gt; 1</code> or to use <code>batch_sampler</code>.</p>
Model Inference when Training using BCEWithLogitloss

nicozhou: <p>Hi, when Training using BCEWithLogitLoss, we know that it internally apply sigmoid for us, and we don’t need any sigmoid again in our model.</p>
<p>But for inference, do we need to add any sigmoid in the last layer of our model? To make the logits become (0,1)?</p>

JuanFMontesinos: <p>Indeed.<br>
It is applied together with the loss for numerical stability, thus, you need to compute it manually at inference.</p>
Using Torchserve to deploy Tacotron2 and WaveGlow

kdot: <p>Hello together,</p>
<p>I wanted to ask if anyone has experience in the deployment of Tacotron2 and WaveGlow (<a href="https://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/SpeechSynthesis/Tacotron2" rel="nofollow noopener">https://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/SpeechSynthesis/Tacotron2</a>) via Torchserve and could provide some insights. I am slightly confused as both models are required for inference and in the documentation of Torchserve I did not notice anything similar. Moreover, I would like to run the inference step of the model on a CPU, is this possible?</p>

androiddrew: <p>It seems possible <a href="https://github.com/pytorch/serve/blob/master/examples/text_to_speech_synthesizer/waveglow_handler.py" class="inline-onebox" rel="noopener nofollow ugc">serve/waveglow_handler.py at master · pytorch/serve · GitHub</a></p>
Stagnant accuracy while using Transfer learning

AIBOI: <p>Hey guys, I am trying to apply transfer learning for the Cifar-10 dataset, I have been trying multiple learning rates, scheduler, image augmentations. However nothing worked, my accuracy is at a stagnant 81%, I can’t seem to get it above 90%</p>
<p>I am currently using the pre-trained vgg16 model.</p>
<p>Image augmentations:</p>
<pre><code class="lang-auto">transform = Compose([
    tt.Resize(size=(224, 224)),
    tt.ToTensor(),
    tt.Normalize( 
       (0.4914, 0.4822, 0.4465), (0.2023, 0.1994, 0.2010) 
    ),
    tt.RandomPerspective(0.3,0.3)
])
</code></pre>
<p>fine-tuning last layer</p>
<pre><code class="lang-auto">models.classifier[6] = nn.Linear(4096,10)
models.classifier[6].requires_grad = True
models.classifier[6].to(device)
</code></pre>
<p>functions for training:</p>
<pre><code class="lang-auto">def accuracy (output,labels):
  _,preds = torch.max(output, dim = 1)
  return torch.tensor(torch.sum(preds == labels).item()/len(preds))


def training_step(train_dl,model):
    model.train()
    images,labels =train_dl
    images,labels = images.to(device),labels.to(device)
    output = model(images)
    loss = F.cross_entropy(output,labels)
    return loss

def validation_step(val_dl,model):
   model.eval()
   images,labels = val_dl
   images,labels = images.to(device),labels.to(device)
   output = model(images)
   loss = F.cross_entropy(output,labels)
   acc = accuracy(output,labels)
   return {"val_loss": loss, "val_acc": acc }

def val_epoch_loss (outputs, model):
  epoch_validation_step = [validation_step(batch,model) for batch in outputs]
  batch_loss = [x["val_loss"] for x in epoch_validation_step]
  epoch_loss = torch.stack(batch_loss).mean()

  batch_acc = [x["val_acc"] for x in epoch_validation_step]
  epoch_acc = torch.stack(batch_acc).mean() ####
  return {"Loss": epoch_loss.item(), "Accuracy": epoch_acc.item()}


def epoch_end(result,epoch):
  print("epoch: {},  Epoch_loss:{}, Epoch_accuracy {}, train_loss {}" .format(epoch, result["Loss"], result["Accuracy"], result["train loss"] )) 
</code></pre>
<p>optimizer and lr scheduler:</p>
<pre><code class="lang-auto">optimizer = optim.Adam(models.parameters(), lr = 0.1)
lr_scheduler = optim.lr_scheduler.StepLR(optimizer, 5, 0.8)
</code></pre>
<p>training of model:</p>
<pre><code class="lang-auto">def fit(epochs,model,train_dl,val_dl):
  history = []
  for epoch in range(epochs):
    train_loss = []
    for batch in train_dl:
      optimizer.zero_grad()
      loss = training_step(batch, model)
      train_loss.append(loss)
      loss.backward()
      optimizer.step()
    lr_scheduler.step()
    #validation portion
    results = val_epoch_loss(val_dl, model)
    results["train loss"] = torch.stack(train_loss).mean()
    history.append(results)
    epoch_end(results,epoch)
  return history

</code></pre>
<p>training augmentations</p>
<pre><code class="lang-auto">train = fit(15,models,train_dl,val_dl)
</code></pre>
<p>I tried to keep the code as short as possible, but can anybody give me tips on getting out of this plateau?</p>
Parallelizing loop over task-specific batches through task-specific weights

rcmse: <p>Hi,</p>
<p>I was hoping for guidance on how to parallelize forward passes of task-specific batches through task-specific weights. My multitask model looks something like this:</p>
<pre><code class="lang-auto">class my_model(nn.Module):
     def __init__(self, task_names):
          self.task_names = task_names  # list of strings
          self.shared_layer = nn.Linear(64, 64)
          self.heads = nn.ModuleDict({task_name: nn.Linear(64, 1) for 
                                      task_name in self.task_names})
     def forward(self, in_data, task_name):
          feature = self.shared_layer(in_data)
          return self.heads[task_name](feature)
</code></pre>
<p>Now, when I go to train this thing, I loop over task-specific batches, compute the task losses, and then perform a backward pass over the total loss. The code looks like:</p>
<pre><code class="lang-auto">task_losses = []
for task_name in task_names:
     in_data, target = next(task_dataloaders[task_name])  # task_dataloaders is a dict
     prediction = my_model(in_data, task_name)
     task_loss = loss(prediction, target)
     task_losses.append(task_loss)
total_loss = torch.sum(torch.stack(task_losses))
total_loss.backward()
</code></pre>
<p>My training seems to be pretty slow for ~20 tasks and I suspect it can be drastically sped up by parallelizing the <code>for</code> loop, possibly on multiple GPUs. Is there a recommended way to do this? I’m not sure how to use the traditional <code>nn.DataParallel</code> method with task-specific weights. Thanks!</p>

suraj.pt: <p>Hi <a class="mention" href="/u/rcmse">@rcmse</a>,</p>
<p>Instead of defining a separate linear layer for each task and looping over it, you can define a single linear layer <code>nn.Linear(64, len(task_names))</code> that will give you an output of size <code>[batch_sz, len(task_names)]</code>.</p>
<p>Example:</p>
<pre><code class="lang-python">x = torch.rand(1,5)

# Linear layer for each individual tasks...
l1 = torch.nn.Linear(5,1)
l2 = torch.nn.Linear(5,1)

o1 = l1(x)    # tensor([[0.4597]])
o2 = l2(x)    # tensor([[0.0697]])

# ... is equivalent to one (faster) linear layer

l = torch.nn.Linear(5,2)
l.weight.data[0] = l1.weight.data
l.bias.data[0] = l1.bias.data
l.weight.data[1] = l2.weight.data
l.bias.data[1] = l2.bias.data

o = l(x)   # tensor([[0.4597, 0.0697]])
</code></pre>
<p>Since in multitask learning the inputs are the same for all tasks, you can skip looping over it by modifying your target to be a tensor of shape <code>[batch_sz, len(task_names)]</code> instead of <code>[batch_sz, 1]</code> for each task.</p>

rcmse: <p>Hi <a class="mention" href="/u/suraj.pt">@suraj.pt</a> ,</p>
<p>Thanks for your reply! Unfortunately, my inputs are not the same for all tasks - my set of task datasets are all of different size, sometimes by an order of magnitude.</p>
<p>For context, it’s a scientific dataset and I cannot really change it.</p>
Accuracy doesn't change regardless of different models

Sevde_Aydogdu: <p>Hi all! I’m trying to do binary segmentation with the slice of CT data. Whenever I computed accuracy I got the same value for the average of each epoch. I used the sigmoid function for the output of the model. any helps would be appreciated!<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/1/9122cf03eea5d4b86b182b9c743082451bf23548.png" data-download-href="https://discuss.pytorch.org/uploads/default/9122cf03eea5d4b86b182b9c743082451bf23548" title="iterator"><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/1/9122cf03eea5d4b86b182b9c743082451bf23548.png" alt="iterator" data-base62-sha1="kHVOfdPsHh0YUKvYzqlQaYb21DO" width="690" height="485" data-dominant-color="212223"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">iterator</span><span class="informations">917×645 22.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/a/ea571f60154f3f4ebb55aa35d278c422a68a34e5.png" data-download-href="https://discuss.pytorch.org/uploads/default/ea571f60154f3f4ebb55aa35d278c422a68a34e5" title="loss_acc_func"><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/a/ea571f60154f3f4ebb55aa35d278c422a68a34e5.png" alt="loss_acc_func" data-base62-sha1="xr4o0OvPSh7U3vlk9eoWL4fowoB" width="555" height="500" data-dominant-color="222424"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">loss_acc_func</span><span class="informations">658×592 22.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

mMagmer: <p>check and see if your model can learn any thing with small subset of your dataset, try to over train on them.<br>
10 sample should be enough for test.<br>
see if your loss decease.</p>

mMagmer: <p><strong>¯\_(ツ)_/¯</strong><br>
sorry, i don’t know</p>
[Resolved]Convert 5D numpy array with dtype('O) to a 5D Tensor

Yilin_Liu: <p>Hi all,</p>
<p>So I’ve tried this but the dtype of img_patches is still ‘O’ which can not do the conversion.</p>
<p>for i in xrange(len(img_patches)):<br>
for j in xrange(len(img_patches[i])):<br>
img_patches[i][j].astype(float)</p>
<p>Thanks so much in advance!</p>

alexis-jacq: <p>Simply doing <code>img_patches.astype(float)</code> does not work?</p>

Yilin_Liu: <p>Thank you for the help!</p>
<p>Yeah I 've tried that but then I got this error:</p>
<p>Traceback (most recent call last):<br>
File “&lt;pyshell#4&gt;”, line 1, in <br>
img_patches.astype(float)<br>
ValueError: setting an array element with a sequence.</p>
<p>So it seems that I have to loop over the items in the “img_patches” to do the conversion as it somehow supports 3D array conversion but not 4D or 5D.</p>
<p>But I want the whole 5D array to be a tensor so that they can be a batch of inputs for the network.</p>
<p>PS: DataLoader does not support my data format.</p>

alexis-jacq: <p>I can reproduce your error with this:</p>
<pre><code class="lang-python">u = np.array([[1,2], [2, 3, 4]])
u.astype(float)
</code></pre>
<p>it looks you are creating an array of vectors with different sizes. So your objects (‘o’) you are trying to convert into float are probably python sequences.</p>

Yilin_Liu: <p>thank you so much for pointing that out! I’m a newbie for python and pytorch and my dataset is too large that I didn’t detect such errors.<br>
But I’m still struggling with this because I couldn’t exclude those data with different size otherwise the results will probably be biased…</p>

Yilin_Liu: <p>Never mind. Solved it! Thank you so much!!</p>

aaaa1: <p>Hi ! I have the same problem… how did you solve it ?</p>
How to deal with unequal length input in CNN or MLP?

beann: <p>I have a variable length input for a neural network (e.g., CNN or MLP). The shape of this tensor is like [b, n, d], where b is the batch size, [n, d] is the feature for a sample, but only part of the data is valid. e.g., for n=3, d=1, a tensor may like [[1], [0], [0]] or [[1], [1], [0]] or [[1], [1], [1]], 0 means the padding.</p>
<p>How can I process this with CNN or MLP (I can flatten the tensor to 1D). I think I need some operation like masking?</p>

Sahil_Goyal: <p>You may use the Adaptive Pooling layer, which outputs the tensor of the same size regardless of the input size.<br>
<a href="https://pytorch.org/docs/stable/generated/torch.nn.AdaptiveAvgPool2d.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.nn.AdaptiveAvgPool2d.html</a></p>

beann: <p>Yeah, but it seems the adaptive pooling does not support batch processing if the input size not equal?</p>

Sahil_Goyal: <p>Then, you can make a custom dataset using PyTorch and add adaptive pooling in the <strong>getitem</strong> function. <a href="https://pytorch.org/tutorials/beginner/basics/data_tutorial.html#creating-a-custom-dataset-for-your-files" class="inline-onebox" rel="noopener nofollow ugc">Datasets &amp; DataLoaders — PyTorch Tutorials 1.11.0+cu102 documentation</a>. Then you can choose whatever batch size you want using the PyTorch data loader class.</p>

beann: <p>Thank you for your suggestions. Another question is what are the advantages of this method compared to direct filling with 0?</p>

Sahil_Goyal: <p>So, one difference between these is, pooling will always reduce dimensions, and filling with 0s to make all input sizes equal will increase the dimensions.<br>
So if we are adding a pooling layer in the <strong>getitem</strong> function, it will have the same general advantages that we get after adding a pooling layer inside the model (eg: reducing no. of training parameters, making the model more robust, etc), whereas adding 0s will not have any such effect.</p>

beann: <p>Thank you for your suggestion. I’ll try this.</p>

Sahil_Goyal: <p><a class="mention" href="/u/beann">@beann</a> You can also make your custom collate function for making a batch of variable sizes.<br>
see this <a href="https://discuss.pytorch.org/t/how-to-create-a-dataloader-with-variable-size-input/8278" class="inline-onebox">How to create a dataloader with variable-size input</a><br>
And then you can use adaptive pooling inside the model.</p>
Indices graphs in original dataset

FloorEijkelboom: <p>Hi!</p>
<p>When generating a batch from a <code>torch_geometric</code> <code>loader</code> object while training, is it possible to refer back to the indices in the original dataset? i.e., let’s say my <code>loader</code> is based on the graph dataset [g_0, g_1, g_2, g_3, g_4, g_5] and my batch is based on graphs [g_3, g_1], is it possible to retrieve indices (3, 1)?</p>
<p>Thanks!</p>

ptrblck: <p>I’m not sure how <code>torch_geometric</code> implements their datasets, but assuming they are similar to <code>torch.utils.data.Dataset</code> (or the same), you could return the <code>index</code> in the <code>__getitem__</code> and retrieve it in the <code>DataLoader</code>.</p>

FloorEijkelboom: <p>Thanks! This worked <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"> To be more precise (for someone else perhaps encountering this), I altered the <code>get</code> function (see <a href="https://pytorch-geometric.readthedocs.io/en/latest/_modules/torch_geometric/data/in_memory_dataset.html" class="inline-onebox" rel="noopener nofollow ugc">torch_geometric.data.in_memory_dataset — pytorch_geometric 2.0.5 documentation</a>) to return the index. Thanks again!</p>
How to specify the interpolate layers output shape when export to onnx?

seungtaek94: <p>Hi.</p>
<p>When I exporting a model that final layer is an “interpolate layer”.</p>
<p>That model doesn’t have specific output shape.</p>
<p>I tested flowing simple model that has only interpolate layer.</p>
<p>When I print output shape of ort_session its show <code>['batch_size', 'Resizeoutput_dim_1', 'Resizeoutput_dim_2', 'Resizeoutput_dim_3']</code>.</p>
<pre><code class="lang-python">import onnxruntime
import onnx
import numpy as np

import torch
import torch.nn as nn
import torch.nn.functional as F

class SimpleTest(nn.Module):
    def __init__(self):
        super(SimpleTest, self).__init__()

    def forward(self, x):        
        y = F.interpolate(x, size=(x.shape[2] * 2, x.shape[2] * 2))
        return y

if __name__ == "__main__":    
    model = SimpleTest()
    model.cuda()
    model.eval()

    dummy_input = torch.rand(1, 1, 4, 4).cuda()

    out = model(dummy_input)

    torch.onnx.export(model, dummy_input,"./results/simple_test.onnx",export_params=True,
                            opset_version=11, do_constant_folding=True, 
                            input_names=['input'], output_names=['output'],
                            dynamic_axes={'input':{0: 'batch_size'},
                            'output':{0: 'batch_size'}})


    onnx_model = onnx.load("./results/simple_test.onnx")
    onnx.checker.check_model(onnx_model)

    dummy_input = np.array([[[[1.0, 1.0, 1.0, 1.0], [2.0, 2.0, 2.0, 2.0], [3.0, 3.0, 3.0, 3.0], [4.0, 4.0, 4.0, 4.0]]]], dtype=np.float32)
        
    ort_session = onnxruntime.InferenceSession("./results/simple_test.onnx")
    ort_inputs = {ort_session.get_inputs()[0].name: dummy_input}

    print("Input_Shape: ", ort_session.get_inputs()[0].shape)
    print("Output_Shape: ", ort_session.get_outputs()[0].shape)
    ort_outs = ort_session.run(None, ort_inputs)

    print(ort_outs[0])

"""
&gt;&gt;&gt; Input_Shape:  ['batch_size', 1, 4, 4]
&gt;&gt;&gt; Output_Shape:  ['batch_size', 'Resizeoutput_dim_1', 'Resizeoutput_dim_2', 'Resizeoutput_dim_3']
&gt;&gt;&gt;[[[[1. 1. 1. 1. 1. 1. 1. 1.]
   [1. 1. 1. 1. 1. 1. 1. 1.]
   [2. 2. 2. 2. 2. 2. 2. 2.]
   [2. 2. 2. 2. 2. 2. 2. 2.]
   [3. 3. 3. 3. 3. 3. 3. 3.]
   [3. 3. 3. 3. 3. 3. 3. 3.]
   [4. 4. 4. 4. 4. 4. 4. 4.]
   [4. 4. 4. 4. 4. 4. 4. 4.]]]]
"""
</code></pre>
<p>How to specify the interpolate layers output shape when export to onnx?</p>

YUAN_LIANG: <p>Met the same problem. Did you get a solution?</p>

seungtaek94: <p>Hi… Actually, I couldn’t found any solution about this issue…</p>
<p>But. In my case it doesn’t matter to inference.</p>
<p>I just do inference even if last layer doesn’t have specific shape,then reshape the output of model.</p>
<p>I hope you solve the problem <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
Different MAE values from predict() versus actual calculated form data

asifk: <p>Hi all,</p>
<p>This is my first post here. Hopefully will get some quick feedback. I’m working on crowd counting and density estimation on images using (ShanghaiTech dataset) and using MSE as loss and MAE as metric function. I’m calculating both MAE in train(), validate() and predict() functions as follow (showing predict() function for reference):</p>
<pre><code class="lang-auto">def predict(model, dataloader, device):
    '''
    Make prediction from a pytorch model 
    '''
    # set model to evaluate model
    model.eval()
    test_images = torch.tensor([], device=device)
    test_gt     = torch.tensor([], device=device)
    test_pred   = torch.tensor([], device=device)
    epoch_mae   = 0.0
    
    # deactivate autograd engine and reduce memory usage and speed up computations
    with torch.no_grad():
        for i, data in tqdm(enumerate(dataloader), total=int(len(dataloader)/dataloader.batch_size)):
            img, gt = data[0].to(device), data[1].to(device)  # Read a single batch
            outputs        = model(img)  # predict the output
            
            # Metric
            actual_count = gt.sum()
            pred_count   = outputs.sum()
            epoch_mae     += abs(actual_count - pred_count) # sum over all batches (try .data.sum())
            
            # Output to return
            test_images  = torch.cat((test_images, img), 0)  # concatenate images in this batch to 'images'
            test_gt      = torch.cat((test_gt, gt), 0)
            test_pred    = torch.cat((test_pred, outputs), 0)
            
        # Calc MAE
        mae  = epoch_mae/len(dataloader.dataset)
        # print(f'mae: {mae}')
    return test_images, test_gt, test_pred, mae
</code></pre>
<p>When run predict() function over test set, I got MAE = 10.6043. And that’s correct as this MAE was calculated as lowest value at which I saved the model during training and then reloaded the saved model before running predict().</p>
<p>But then when I plot prediction performance as follow, I get a higher MAE.</p>
<ol>
<li>Find actual and predicted counts for test set.</li>
</ol>
<pre><code class="lang-auto">test_actual_count  = []
test_pred_count    = []
## Test predictions
for i in range(0, len(test_dm)): # test_ds_unbatch.as_numpy_iterator():
    test_actual_count.append(torch.sum(test_dm[i,:]))
    test_pred_count.append(torch.sum(test_dm_h[i,:]))

# Lists to Tensors
test_actual_count  = torch.tensor(test_actual_count) #.detach().cpu()
test_pred_count    = torch.tensor(test_pred_count) #.detach().cpu()

# Calculate MAE
test_mae = abs(test_actual_count - test_pred_count).sum()/len(test_dl.dataset)
</code></pre>
<p>The value of ‘test_mae’ calculated in the last line above is 61. Is this strange or am I doing something wrong? (By the way looking into the scatter plot of actual_count vs pred_count, it seems that the value of “test_mae” calculated in the last line is correct, but something is wrong with the value returned by predict().</p>
Weird image truncated error when num_workers larger than 0

jdhao: <p>I have met this weird error when <code>num_workers</code> is set to 1 or larger. Here is my MWE to reproduce:</p>
<pre data-code-wrap="python"><code class="lang-python">from PIL import Image


import torch
from torchvision import transforms


class FrameDataset(torch.utils.data.Dataset):
    def __init__(self, pil_imgs):
        super(FrameDataset, self,).__init__()

        self.pil_imgs = pil_imgs
        self.transform = make_transform()

    def __len__(self):
        return len(self.pil_imgs)

    def __getitem__(self, idx):
        img = self.pil_imgs[idx]

        return self.transform(img)


def make_transform():
    trans = transforms.Compose([
        transforms.CenterCrop(224),
        transforms.ToTensor(),
    ])

    return trans


def main():
    im = Image.open("../data/debug_img/S04000000003v5Rw0Nr9y7R___0000.jpg")
    pil_imgs = [im]

    num_round = 2
    for _ in range(num_round):
        val_dataset = FrameDataset(pil_imgs)
        val_loader = torch.utils.data.DataLoader(val_dataset,
                                                 batch_size=1,
                                                 num_workers=1)

        for im_batch in val_loader:
            continue


if __name__ == "__main__":
    main()
</code></pre>
<p>Okay, the actual image does not matter, as long as I set <code>num_workers=1</code>, for any image, the above script will error out something like this:</p>
<blockquote>
<p>OSError: image file is truncated ( 11 bytes not processed)</p>
</blockquote>
<p>Yes, I have read <a href="https://discuss.pytorch.org/t/oserror-image-file-is-truncated-150-bytes-not-processed/64445/2">this post</a> and <a href="https://discuss.pytorch.org/t/oserror-image-file-is-truncated-28-bytes-not-processed-during-learning/36815/17">this post</a>. But I do not think setting <code>ImageFile.LOAD_TRUNCATED_IMAGES = True</code> is the right way to solve the issue, because the image is not broken or corrupt at all.</p>
<p>I also noticed that if I set <code>num_round</code> to 1, the error will disappear.</p>
<p>So in summary, I have the following table</p>
<div class="md-table">
<table>
<thead>
<tr>
<th>num_round</th>
<th>num_workers</th>
<th>error or not?</th>
</tr>
</thead>
<tbody>
<tr>
<td>1</td>
<td>1</td>
<td>no error</td>
</tr>
<tr>
<td>1</td>
<td>0</td>
<td>no error</td>
</tr>
<tr>
<td>2</td>
<td>1</td>
<td>error</td>
</tr>
<tr>
<td>2</td>
<td>0</td>
<td>no error</td>
</tr>
</tbody>
</table>
</div>

ejguan: <p>This may related to how <code>Pillow.Image</code> being handled by multiprocessing. Do you mind converting it into Tensor?</p>

jdhao: <p>That may not be viable since I need to preprocess the pil images and then convert the images to Tensor. If I convert them to Tensor, I can not apply the pre-processing anymore.</p>

jdhao: <p>Okay, I tried to convert PIL imgs to tensor beforehand and remove the ToTensor() transform in <code>make_transform()</code>, the error stops.</p>
Different batch sizes in DistributedDataParallel

zizhao.mo: <p>Hi,</p>
<p>I want to know if there is any method to use different batch sizes across processes in distributedDataParallel? If yes, to my knowledge, gradients from different processes should be weighted by their batch sizes. So can PyTorch intelligently handle this issue?</p>
<p>Thanks for any comments in advance.</p>

ptrblck: <p><a href="https://github.com/pytorch/pytorch/issues/67253#issuecomment-952229018">This comment</a> suggest different approaches using hooks.</p>

zizhao.mo: <p>thank you so much, this is what I am finding exactly.</p>
CuDNN error for loss.backwards

bhaktatejas922: <pre><code class="lang-auto">    allow_unreachable=True, accumulate_grad=True)  # allow_unreachable flag
RuntimeError: cuDNN error: CUDNN_STATUS_NOT_INITIALIZED
</code></pre>
<p>training code:</p>
<pre><code class="lang-auto">device = torch.device("cuda:2" if torch.cuda.is_available() else "cpu")
dataset = LoveDA(root="loveda", split="train", scene=['urban'], download=True, transforms=None) # Download the dataset
# sampler = RandomGeoSampler(dataset, size=512, length=10000)
dataloader = DataLoader(dataset, batch_size=1, shuffle=True, num_workers=4) # Create a dataloader


# Using segmentation models, train a Unet model with an efficientnet backbone. Iterate over each batch in the dataloader and train the model.

model = smp.Unet('tu-efficientnetv2_rw_s', encoder_weights=None, classes=1)
# criterion = nn.BCELoss()
criterion = nn.BCELoss()
optimizer = optim.Adam(model.parameters(), lr=0.001)
sched = torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer, mode='min', factor=0.5, patience=5, verbose=True)
model = model.to(device)

for epoch in range(NUM_EPOCHS):
    model.train()
    with tqdm(total=len(dataloader)) as pbar:
        for batch_idx, batch in enumerate(dataloader):
            data = batch['image'].to(device).type(torch.float32)
            target = batch['mask'].to(device).type(torch.float32)
            output = model(data)
            output=torch.squeeze(output, dim=1)
            print(output.shape)
            print(target.shape)
            loss = criterion(output, target)
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()
            pbar.update(1)```</code></pre>

ptrblck: <p>Could you post the output of <code>python -m torch.utils.collect_env</code> here, please?<br>
<code>CUDNN_STATUS_NOT_INITIALIZED</code> is often raised due to a setup issue, which shouldn’t happen if you are using the binaries.</p>
How to use Hook or other way to modify a layer's output

Handa_gou: <p>Hello,<br>
I am trying to modify a layer’s output and pass it to the next, I try to use the forward hook function but I can only read the data, not modify them, could you give me some advice? I really appreciate it!</p>
<pre><code class="lang-auto">activation = {}
def get_activation(name):
    def hook(model, input, output):
        activation[name] = output.detach()
    return hook

model2.fc3.register_forward_hook(get_activation('fc3'))

outputs = model2(images)
print(outputs)
kc = activation['fc3']
zc=kc[:, :16]
out_feature=find_close(feature,zc,kc)
outputs = model2(images)
</code></pre>

ptrblck: <p>You can modify the activation by returning it in the hook:</p>
<pre><code class="lang-python">activation = {}
def get_activation(name):
    def hook(model, input, output):
        activation[name] = output.detach()
        return torch.ones_like(output)
    return hook

model = nn.Linear(1, 1)
model.register_forward_hook(get_activation('fc'))

outputs = model(torch.randn(1, 1))
print(outputs)
# &gt; tensor([[1.]])
</code></pre>
Where can I find document for "accessors"?

Ziyu_Huang: <p>Hi! I am learning the detail of this tutorial: <a href="https://pytorch.org/tutorials/advanced/cpp_extension.html" class="inline-onebox" rel="noopener nofollow ugc">Custom C++ and CUDA Extensions — PyTorch Tutorials 1.11.0+cu102 documentation</a><br>
Which describes a concept of “accessors”, using this keyword, nothing shown on pytorch docs, but I find this document of pytorch using google: <a href="https://pytorch.org/cppdocs/notes/tensor_basics.html" class="inline-onebox" rel="noopener nofollow ugc">Tensor Basics — PyTorch master documentation</a><br>
Much more detailed! But again a sentence shows:</p>
<p>“Note that the template can hold other parameters such as the pointer restriction and the integer type for indexing. See documentation for a thorough template description of <em>accessors</em> and <em>packed accessors</em> .”</p>
<p>So I am interested in finding the documents of “accessors”, but this time, I really can not find anything… I guess there should be somethin? Not sure how to find this documents…</p>
<p>(Haha, maybe just no documents of this… Just wonder maybe I am using wrong method to find docs)</p>

ptrblck: <p>Unsure how helpful it is, but maybe the comments in the source code might help <a href="https://github.com/pytorch/pytorch/blob/dc169d53aa266560750ea25ee0cf31c7e614550d/aten/src/ATen/core/TensorAccessor.h#L28">here</a>.</p>