akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Documentation inconsistent for torch.vision.transform.functional.crop

username: cdyrhjohn
cooked: <p>Hi there,<br>
The documentation here: <a href="https://pytorch.org/docs/stable/_modules/torchvision/transforms/functional.html#crop" rel="nofollow noopener">https://pytorch.org/docs/stable/_modules/torchvision/transforms/functional.html#crop</a> is different from the actual source code in pytorch 1.6 (the documentation says the module support tensor as input but the source code only support PIL Image):</p>
<p>def crop(img, top, left, height, width):<br>
“”"Crop the given PIL Image.</p>
<pre><code> Args:
     img (PIL Image): Image to be cropped. (0,0) denotes the top left corner of the image.
     top (int): Vertical component of the top left corner of the crop box.
     left (int): Horizontal component of the top left corner of the crop box.
     height (int): Height of the crop box.
     width (int): Width of the crop box.

 Returns:
     PIL Image: Cropped image.
 """
 if not _is_pil_image(img):
     raise TypeError('img should be PIL Image. Got {}'.format(type(img)))
 return img.crop((left, top, left + width, top + height))</code></pre>

username: ptrblck
cooked: <p>Thanks for reporting this issue.<br>
It seems this PR didn’t get picked for the <a href="https://github.com/pytorch/vision/blob/78ed10cc51067f1a6bac9352831ef37a3f842784/torchvision/transforms/functional.py#L452-L455"><code>0.7</code> release</a>, but can be found in <a href="https://github.com/pytorch/vision/blob/df6a7960920534d53d31f6d06fa89dea90cc1b50/torchvision/transforms/functional.py#L411-L414">master</a>.</p>
<p>CC <a class="mention" href="/u/fmassa">@fmassa</a> in case this PR slipped.</p>
<p>As a workaround you could install the nightly binary or build <code>torchvision</code> from source.</p>
I have a problem in initialization. Please help me to get a PyTorch code for this. Input is like(256x256x1). Just write a dummy block for this 3x3 Conv. Feature maps in the starting is 16

username: Blizzard_boi2020
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/a/8a18641546cd10d24d3183a4cdf34c8538aecc03.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/8a18641546cd10d24d3183a4cdf34c8538aecc03" title="Screenshot_20200803-010027717 (1)"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/a/8a18641546cd10d24d3183a4cdf34c8538aecc03_2_690x302.jpeg" alt="Screenshot_20200803-010027717 (1)" data-base62-sha1="jHE9c1NFDJ9TKNsNcglEJ0WHxkv" width="690" height="302" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/a/8a18641546cd10d24d3183a4cdf34c8538aecc03_2_690x302.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/8/a/8a18641546cd10d24d3183a4cdf34c8538aecc03.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/8/a/8a18641546cd10d24d3183a4cdf34c8538aecc03.jpeg 2x" data-dominant-color="EBEBEB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot_20200803-010027717 (1)</span><span class="informations">941×413 120 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>The <a href="https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html#define-a-convolutional-neural-network">CIFAR10 tutorial</a> might be a good starter, as it’s creating a new model.</p>
<p>The <a href="https://pytorch.org/docs/stable/generated/torch.nn.Conv2d.html#torch.nn.Conv2d">Conv2d docs</a> give you some information about the expected input arguments, such as <code>in_channels</code>, <code>out_channels</code>, and <code>kernel_size</code>.</p>
<p>For your use case it seems that that first conv layer should be defined as:</p>
<pre><code class="lang-python">conv = nn.Conv2d(in_channels=?, out_channels=16, kernel_size=3)
</code></pre>
<p>The following layers would then use <code>in_channels=16</code>, since this is the number of channels in the output activation from the preceding layer.</p>

username: Blizzard_boi2020
cooked: <p>It’s fine. Thanks. What about the residual networks?</p>

username: ptrblck
cooked: <p>You could reuse the approach used in <code>torchvision</code>'s resnet implementation as seen <a href="https://github.com/pytorch/vision/blob/df6a7960920534d53d31f6d06fa89dea90cc1b50/torchvision/models/resnet.py#L56-L72">here</a>.</p>
Training while inferencing on GPU

username: ZeroMax
cooked: <p>Hello.</p>
<p>I have an RL agent which I would like to train on my GPU. I’d like to split training and acting with this RL agent, in a way which allows the RL agent to train with experience replay while moving in its environment. However, torch.multiprocessing will only allow me to use the CPU to do this. I have tried to use the ‘spawn’ method like the Pytorch documentations suggest, but this causes my script to try to pickle the lambda functions within the nn.Modules, which are unpicklable. Is there any way I can get around this?</p>
<p>Thanks.</p>
Get argmax as an int?

username: Jesper.Lindberg
cooked: <p>Hello!<br>
I have some troubles getting the predicted value as an int. I am doing a DQN algorithm with the following model:</p>
<blockquote>
<p>struct Model : torch::nn::Module {</p>
<pre><code>torch::nn::Linear input_layer{ nullptr }, first_hidden_layer{ nullptr }, output_layer{ nullptr };

//Constructor
Model() {
    //construct and register the layers we want
    input_layer = register_module("input_layer", torch::nn::Linear(k_input_size, 64));
    first_hidden_layer = register_module("first_hidden_layer", torch::nn::Linear(64, 64));
    output_layer = register_module("output_layer", torch::nn::Linear(64, k_output_size));

}

//Feed forward through the network
torch::Tensor forward(torch::Tensor tensor_x) {
    tensor_x = torch::relu(input_layer-&gt;forward(tensor_x));
    tensor_x = torch::relu(first_hidden_layer-&gt;forward(tensor_x));
    tensor_x = torch::relu(output_layer-&gt;forward(tensor_x));

    return torch::log_softmax(tensor_x, /*dim*/1);
}
</code></pre>
<p>};</p>
</blockquote>
<p>I then give it my input vector and I want it to return the index of wich of the output had the highest value. I try getting this with argmax but argmax returns a tensor and not a int?</p>
<blockquote>
<p>int getAction(std::vector &amp;input_vector)  {<br>
c10::IntArrayRef size({ k_input_size,1 });<br>
torch::Tensor input_tensor = torch::from_blob(input_vector.data(), size);</p>
<p>auto DNN_out = model.forward(input_tensor);<br>
int action = DNN_out.argmax(1); //Assuming dim 1 is the one I want here?</p>
<p>return action;<br>
}</p>
</blockquote>

username: LeviViana
cooked: <p>This only works if <code>DNN_out.argmax(1)</code> is a tensor having just one element (i.e., if <code>DNN_out</code> has a shape like {1, n}):</p>
<pre><code class="lang-auto">int action = DNN_out.argmax(1).item().toInt();
</code></pre>

username: jy_mo
cooked: <p>thank you very much!</p>
Loss on multiple outputs

username: kshitij98
cooked: <p>I have a model which takes an input image <em><strong>x</strong></em> and an integer factor <em><strong>f</strong></em> which is extrapolated with the constant value and added as a channel to the input image.</p>
<p>So if image x was a greyscale image [[1, 2][3, 4]], and the factor was 2,<br>
I’ll feed the following vector to the model:<br>
[<br>
[[1, 2], [3, 4]],<br>
[[2, 2], [2, 2]]<br>
]</p>
<p>Now, I want to implement a loss function on this model which computes on the combination of outputs from fixed multiple factors say [0, 1, 2, 3, 4] for the input image, and then the optimizer makes a step.</p>
<p>Say, the outputs are [O0, O1, O2, O3, O4], we want to compute the loss function as f([O0, O1, O2, O3, O4])</p>
<p>Can i do the same with pytorch somehow or will I need to train separate networks for each factor only?</p>
Concat Tensor of Multi-resolution Image

username: Noiran_Allen
cooked: <p>Hi,</p>
<p>I’m trying to train a CNN using multi-resolution images with mini-batch training. I plan to maintain its aspect ratio so resizing is not a preferable option.</p>
<p>Currently, I understand that to concatenate tensors of differing image resolution is impossible unless the tensors’ image resolutions are made equal, which is not what I’m trying to do. Unfortunately, this is the default collation in <code>DataLoader</code>.</p>
<p>A workaround I’ve tried is to feed the mini-batch tensors (images) in a list (with custom <code>DataLoader</code> collate function), iterating through each element and aggregating the loss of each tensor. But this seems to result in much slower training.</p>
<p>Any ideas on how to make multi-resolution training works? Thank you in advance</p>
Training Loss barely changes

username: natelang
cooked: <p>I am currently trying to use transfer learning with resnet model. I am running the below code for 10 epochs. The training and validation loss seem to barely, barely change if at all. Not sure why this is.</p>
<pre><code class="lang-auto">    for epoch in range(1, num_epochs+1):
        print('Epoch {}/{}'.format(epoch, num_epochs))
        print('-' * 10)

        # Each epoch has a training and validation phase
        for phase in ['train', 'valid']:
            if phase == 'train':
                m.train()  # Set model to training mode
                dataloader = train_data
            else:
                m.eval()   # Set model to evaluate mode
                dataloader = valid_data

            running_loss = 0.0
            running_corrects = 0

            # Iterate over data.
            for inputs, labels in dataloader:
                inputs, labels = inputs.to(device), labels.to(device)

                # zero the parameter gradients
                optimizer.zero_grad()

                # forward
                # track history if only in train
                with torch.set_grad_enabled(phase == 'train'):
                    outputs = m(inputs)
                    loss = criterion(outputs, labels)
                    _, preds = torch.max(outputs, 1)

                    # backward + optimize only if in training phase
                    if phase == 'train':
                        loss.backward()
                        optimizer.step()
                        scheduler.step()

                # statistics
                running_loss += loss.item() * inputs.size(0)
                running_corrects += torch.sum(preds == labels.data)
                
            if phase == 'train':
                epoch_loss = running_loss / train_data_size
                epoch_acc = running_corrects.double() / train_data_size
            else:
                epoch_loss = running_loss / valid_data_size
                epoch_acc = running_corrects.double() / valid_data_size

            

            print('{} Loss: {:.4f} Acc: {:.4f}'.format(
                phase, epoch_loss, epoch_acc))

            # deep copy the model
            if phase == 'valid' and epoch_acc &gt; best_acc:
                best_acc = epoch_acc
                best_model_wts = copy.deepcopy(m.state_dict())
</code></pre>
<p>This code produces the following output:</p>
<pre><code class="lang-auto">Epoch 1/5
----------
train Loss: 1.2359 Acc: 0.6404
valid Loss: 1.2904 Acc: 0.4593

Epoch 2/5
----------
train Loss: 1.2190 Acc: 0.6592
valid Loss: 1.2884 Acc: 0.4593

Epoch 3/5
----------
train Loss: 1.2176 Acc: 0.6565
valid Loss: 1.2878 Acc: 0.4593

Epoch 4/5
----------
train Loss: 1.2233 Acc: 0.6537
valid Loss: 1.2882 Acc: 0.4593

Epoch 5/5
----------
train Loss: 1.2163 Acc: 0.6594
valid Loss: 1.2883 Acc: 0.4593

</code></pre>
<p>Any help or suggestions are appreciated.</p>
BertForMaskedLM's loss and scores, how the loss is computed?

username: sanaz
cooked: <p>I have a simple MaskedLM model with one masked token at position 7. The model returns 20.2516 and 18.0698 as loss and score respectively. However, not sure how the loss is computed from the score. I assumed the loss should be<br>
loss = - log(softmax(score[prediction])<br>
but computing this loss returns 0.0002. I’m confused about how the loss is computed in the model.</p>
<pre><code class="lang-auto">import copy
from transformers import BertForMaskedLM, BertTokenizerFast
import torch
model = BertForMaskedLM.from_pretrained('bert-base-uncased')
tokenizer = BertTokenizerFast.from_pretrained('bert-base-uncased')


text = "Who was Jim Paterson ? Jim Paterson is a doctor".lower()
inputs  =  tokenizer.encode_plus(text,  return_tensors="pt", add_special_tokens = True, truncation=True, pad_to_max_length = True,
                                         return_attention_mask = True,  max_length=64)
input_ids = inputs['input_ids']
masked  = copy.deepcopy(inputs['input_ids'])
masked[0][7] = 103
for t in range(len(masked[0])):
  if masked[0][t] != 103:
    masked[0][t] = -100
loss, scores = model(input_ids = input_ids, attention_mask = inputs['attention_mask'] , token_type_ids=inputs['token_type_ids'] , labels=masked)
print('loss',loss)
print(scores.shape)
pred = torch.argmax( scores[0][7]).item()
print("predicted token:", pred, tokenizer.convert_ids_to_tokens([pred])  )
print("score:", scores[0][7][pred]) 


logSoftmax = torch.nn.LogSoftmax(dim=1)
NLLLos = torch.nn.NLLLoss()
output = NLLLos( logSoftmax(torch.unsqueeze(logit[0][7], 0)), torch.tensor([pred]))
print(output)
</code></pre>
Tips to creating a sequential GAN image generator

username: santi22
cooked: <p>I would like to train a GANs model that can generate 1 output image from a set of 3 images.</p>
<ul>
<li>
<p>Do you have any tips on this?</p>
</li>
<li>
<p>Will the GPU capacity needed to train this be really high? I just requested 1 P optimized instance on AWS to use a p2.xlarge instance. Will that be enough or do I need a p3.2xlarge ?</p>
</li>
</ul>
How to keep consistency for data preprocess between PyTorch model training and serving?

username: Bright
cooked: <p>Given the source tabular data, it’s common that we need preprocess the data and then feed the preprocessed result into the PyTorch model.The preprocessing methods contain OneHot encoding, Bucketize, Standardize, Normalize, Hash and so on.</p>
<p>At training stage, we can preprocess the data using sklearn.preprocessing package or Pandas API or other solutions.<br>
For serving stage, how can we deploy the consistent preprocess logic directly on production without rewrite the preprocessing code?</p>
What's the official high performance serving solution for PyTorch?

username: Bright
cooked: <p>Hi, all. Currently I’m looking for an official PyTorch model serving solution with high performance. From the <a href="https://pytorch.org/tutorials/intermediate/flask_rest_api_tutorial.html" rel="nofollow noopener">docs</a> about PyTorch model serving, there are several options for model deployment. I’m afraid that none of the solutions can meet all the requirement at the same time.</p>
<ul>
<li>Deploy model using Flask: Python have GIL, there may be performance issue if the traffic is heavy.</li>
<li>TorchServe: The key logic (preprocess, inference, postprocess) is in the <a href="https://github.com/pytorch/serve/tree/master/ts/torch_handler" rel="nofollow noopener">torch_handlers</a>. The handlers are also Python modules. There may be performance issue because of the same reason as the option above.</li>
<li>Load TorchScript using LibTorch: It provides the LibTorch library instead of a model server. And the solution doesn’t contain preprocess and postprocess.</li>
</ul>
<p>Look forward to more suggestions.</p>
Doing Data Augmnetation in Parallel

username: 887574002
cooked: <p>Hi, I have a model that I have to do a lot of Data Augmentations in 3D that is time-consuming. I want to do it in parallel. How can I do it?<br>
I read the other topics in the forum but I didn’t find the answer to my question.</p>
<p>Another question that I have I need to do some intensity augmentations like changing contrast, hue, and saturation for 3d images. I can not use the command that we have in PyTorch. So, if anyone knows any package that I can use for NumPy n-array that helps to change pixel intensity really helps me.</p>
<p>Thanks for your help.</p>

username: ptrblck
cooked: <p>If you are using a <code>DataLoader</code> with <code>num_workers&gt;0</code>, multiple workers will be used to create each batch in the background.</p>
<p>I don’t know which lib would provide all functionality you need for the 3D data augmentation, but maybe a medical lib could be helpful, such as <a href="https://github.com/Project-MONAI/MONAI">MONAI</a> or <a href="https://github.com/fepegar/torchio">torchio</a>.</p>

username: 887574002
cooked: <p>Thanks for your answer. So, you mean that when I set num_workers=4, then each batch is divided by 4 and augmentations are done on this 4 subsets in parallel?</p>
<p>Regarding MONAI, and torchio I checked them but I didn’t find any pixel intensity transformations.<br>
Thank you for your help.</p>

username: ptrblck
cooked: <p>No, each worker will create an individual batch.<br>
So 4 workers will create 4 batches simultaneously, add these batches to a queue and try to prefetch the next one.</p>
<p>There was some effort to use multiple workers for a single batch creation, but I don’t know the status of it.</p>

username: 887574002
cooked: <p>Ok, thank you for your answer.</p>

username: fepegar
cooked: <p>Hi, Mjavan. TorchIO does support NumPy arrays:</p>
<pre><code class="lang-python">import numpy as np
import torchio as tio

array = np.ones((1, 2, 2, 2))  # (C, D, H, W)
print(array)
transform = tio.RandomNoise()
transformed = transform(array)
print(transformed)
</code></pre>
<p>Output:</p>
<pre><code class="lang-python">[[[[1. 1.]
   [1. 1.]]

  [[1. 1.]
   [1. 1.]]]]
[[[[0.64996007 0.86066213]
   [1.01217787 0.97101844]]

  [[1.09215886 0.98876163]
   [0.94828662 0.77317567]]]]
</code></pre>
How does one compute the norm of a function in pytorch?

username: pinocchio
cooked: <p>My current way is to convert everything to numpy and do:</p>
<pre><code class="lang-auto">def functional_diff_norm(f1, f2, lb, ub, p=2):
    """
    Computes norm:

    ||f||_p = (int_S |f|^p dmu)^1/p

    https://en.wikipedia.org/wiki/Lp_space

    https://stackoverflow.com/questions/63237199/how-does-one-compute-the-norm-of-a-function-in-python
    """
    norm = integrate.quad(lambda x: (f1(x) - f2(x))**p, lb, ub)
    return norm

</code></pre>
<p>is this a good way to do it?</p>
<p>reference: <a href="https://stackoverflow.com/questions/63237199/how-does-one-compute-the-norm-of-a-function-in-python" rel="nofollow noopener">https://stackoverflow.com/questions/63237199/how-does-one-compute-the-norm-of-a-function-in-python</a></p>
Captum to explain Albert-based mutli-class classification

username: Kevin_Song
cooked: <p>Hello everyone, I have trained a multi-class classification model with pre-trained Albert LM. I am now experimenting with Captum to see the attribution score of each class/label. I expect to see positive/max attribution score for model prediction class/label, and negative and smaller scores for the rest of classes/labels. However, I don’t really see such a correlation there. Not sure if I don’t use Captum correctly or my expectation is simply not correct. Anyone has experience with this? Thanks.</p>

username: Narine
cooked: <p>Hi Kevin, thank you for the question! Do you have any example code snippets that you could post here ?<br>
Jupyter or google colab notebooks would be great too.</p>
PyTorch usability on AMD hardware?

username: trougnouf
cooked: <p>I would very much like to get an AMD GPU as my upcoming upgrade, but PyTorch support is crucial and I cannot find any report of successful application.</p>
<p>Can anyone provide feedback on PyTorch + AMD ROCm/HIP usability? Preferably on Linux.</p>
<p>My specific application is image denoising (UNet / cGAN) and I run Arch Linux but that’s mostly irrelevant.</p>

username: Felix_Lessange
cooked: <p>As of today, not easy to build, nor as effective as tensorflow benchmarks suggest. I would not recommend it. I’m keeping an eye open for next months upcoming ameliorations (maybe ?)</p>

username: Felix_Lessange
cooked: <p>New (I believe !) instructions here :</p>
<p><a href="https://rocm-documentation.readthedocs.io/en/latest/Deep_learning/Deep-learning.html" class="onebox" target="_blank" rel="nofollow noopener">https://rocm-documentation.readthedocs.io/en/latest/Deep_learning/Deep-learning.html</a></p>

username: tom
cooked: <p>As people link this thread: Building PyTorch for ROCm is quite easy: <a href="https://lernapparat.de/pytorch-rocm/" rel="nofollow noopener">https://lernapparat.de/pytorch-rocm/</a> also AMD <a href="https://rocmdocs.amd.com/en/latest/Deep_learning/Deep-learning.html#recommended-install-using-published-pytorch-rocm-docker-image" rel="nofollow noopener">publishes docker images</a>.</p>
<p>Best regards</p>
<p>Thomas</p>
How to optimize 'no. of hidden layers' and 'no. of neurons' using baysian optimization?

username: swap
cooked: <p>Hi,<br>
I am trying to find optimal hyperparameters for my deep learning model. I could able to optimize LR, BS and momentum. But I am strugging with optimization of number of layers and neurons within them. Could someone help me ?<br>
Here is what I have written, what I expect is to modify this class to find optimal input_neurons and output_neurons within each layer.</p>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self, n_lay=2, neur=32, dropout=0.1):
        super().__init__()
        self.name = "DNN"
        self.init_args = {"n_lay": n_lay, "neur": neur}      
        self.fc = nn.ModuleList([nn.Linear(1105, neur)])    # 1105  features/descriptors
        self.fc.extend([nn.Linear(neur, neur) for i in range(1, n_lay-1)])
        self.fc.append(nn.Linear(neur, 1))
        self.dropout = nn.Dropout(p=dropout)

    def forward(self, x):
        for i, l in enumerate(self.fc):
            x = self.dropout(self.fc[i](x))
        return x
</code></pre>
<p>parameters:</p>
<pre><code class="lang-auto">opt_BO = [{"name": "lr", "type": "continuous", "domain": (0.00001, 0.001)},
          {"name": "bs", "type": "discrete", "domain": range(16, 512, 16)},
          {"name": "n_lay", "type": "discrete", "domain": range(2, 6)},
          {"name": "neur", "type": "discrete", "domain": range(8, 128, 8)},
          {"name": "dropout", "type": "discrete", "domain": np.linspace(0, 0.5, 11)}]
</code></pre>
<p>The f_opt function is as below:</p>
<pre><code class="lang-auto">def f_opt(parameters):
    parameters = parameters[0]
    # Data loading:
    trainloader = torch.utils.data.DataLoader(dataset_train, batch_size=int(parameters[1]), shuffle=True)
    testloader = torch.utils.data.DataLoader(dataset_val, batch_size=int(parameters[1]), shuffle=False)
    train_dl, valid_dl = (WrapDL(trainloader, to_gpu), WrapDL(testloader, to_gpu))    

    # Model definition:
    model = Net(
        n_lay=int(parameters[2]),
        neur=int(parameters[3]),
        dropout=parameters[4]
    )

    model.to(device)
    
    # Optimizer:
    opt = optim.SGD(model.parameters(), lr=parameters[0])
    scheduler = optim.lr_scheduler.CyclicLR(opt, parameters[0], parameters[0] * 3,
                                            step_size_up = int(len(train_dl) * int(parameters[1]) * epochs * 0.25))
    # Fit:
    score = fit(epochs, model, loss_func, scheduler, train_dl, valid_dl, calc_accuracy)
    
    return np.array(score)
</code></pre>
How to freeze the FakeQuantize zero_point during train

username: Artak_Arakelyan
cooked: <p>I need to train a quantized model which has 0 offset due to limitations of my inference framework.<br>
I’m following the flow described in <a href="https://pytorch.org/tutorials/advanced/static_quantization_tutorial.html" rel="nofollow noopener">https://pytorch.org/tutorials/advanced/static_quantization_tutorial.html</a><br>
So the model is prepared with prepare_qat which adds FakeQuantize  layers<br>
The problem is that both scale and zero_point are being trained. I need the zero_point to be fixed at 0.</p>

username: supriyar
cooked: <p>The scale and zero_point aren’t trained - they are calculated by observers inserted in the network. You can implement an observer specific to your use-case which will fix the zero_point at 0. For reference the zero_point calculation happens in <a href="https://github.com/pytorch/pytorch/blob/master/torch/quantization/observer.py#L187" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/master/torch/quantization/observer.py#L187</a><br>
Observers are set when you initialize the qconfig (in this case you seem to be using the default. i.e. <a href="https://github.com/pytorch/pytorch/blob/master/torch/quantization/qconfig.py#L90" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/master/torch/quantization/qconfig.py#L90</a></p>

username: Artak_Arakelyan
cooked: <p>Thanks for your input.</p>
<p>Do you know a good example of applying custom observers ?<br>
qconfig = QConfig(activation=FakeQuantize.with_args(observer=,<br>
quant_min=0,<br>
quant_max=255,<br>
reduce_range=True),<br>
weight=default_per_channel_weight_fake_quant)<br>
Is this enough or setting a custom observer or there are some nuances ?</p>

username: supriyar
cooked: <p>You can follow any of the observers defined in [<a href="https://github.com/pytorch/pytorch/blob/master/torch/quantization/observer.py" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/master/torch/quantization/observer.py</a>] as a starting point.</p>
<p>To enable it in the qconfig you can do<br>
<code>FakeQuantize.with_args(observer=MyObserver, quant_min=0, quant_max=255, dtype=torch.qint8, qscheme=torch.per_tensor_affine)</code></p>
Inspecting a single tensor dimension

username: little_adventurer
cooked: <p>I have a weird output tensor with what seems to be an extra colour dimension.</p>
<p>The tensor is of shape T[4, 4, 300, 1200].</p>
<p>I would like to display purely the colour channels to understand what has happened and why there are 4 colour channels rather than 1.</p>
<p>So far I tried doing T[0, :, 0, 0] however I am uncertain whether it is correct.<br>
Thank you!</p>

username: Hmrishav_Bandyopadhy
cooked: <p>Hi,</p>
<p>The last layer of your model decides your output. I am guessing you are using a CNN as you mentioned color–so, your last layer should have an output channel size of 1 if you want the output to be single channeled tensor. Something like <code>nn.Conv2d(in_channels,1,kernel_size,stride,padding)</code></p>
<aside class="quote no-group" data-username="little_adventurer" data-post="1" data-topic="91479">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/l/d6d6ee/48.png" class="avatar"> little_adventurer:</div>
<blockquote>
<p>T[0, :, 0, 0]</p>
</blockquote>
</aside>
<p>This would allow you to train, however , you would have extra weights which are not being trained against any loss function.</p>

username: little_adventurer
cooked: <p>Thanks for the suggestion but it wasn’t for a Conv2d layer.<br>
I was trying to place some images on the tensorboard!</p>
<p>The solution in my case was to remove the dodgy 4th colour channel by</p>
<p>T = T[ :, :3, :, : ]</p>
Question about PyTorch checkpoints results in Testing

username: ahmediqbal
cooked: <p>Hello,</p>
<p>I have one theoretical question, In research papers, mostly we mention testing results, <em>accuracy, precision, recall etc</em>. But mostly In PyTorch training model generates multiple checkpoints, some peoples generate checkpoint after each 1 epoch, some people generate checkpoints after 5 epochs. Something like that.</p>
<p>My question is when I run model testing phase code each checkpoint pass through loop and deliver its own result. Right?</p>
<p>Now, what is criteria to mention testing result in research papers? Suppose i have 10 different checkpoints and each checkpoint has generated 10 different testing results. So I can select highest accuracy checkpoints as best model (with weights) ?</p>
<p>Please help in this regard !</p>

username: Hmrishav_Bandyopadhy
cooked: <p>You give best results obviously <img src="https://discuss.pytorch.org/images/emoji/apple/stuck_out_tongue.png?v=9" title=":stuck_out_tongue:" class="emoji" alt=":stuck_out_tongue:"></p>
<p>There is no predefined number of epochs you should run…so if your model gives best results at 2nd epoch then that’s it !</p>
<p>Well, technically you dont check testing results for each checkpoint…you create a validation set and check loss ONLY on the validation set. On basis of validation loss, you select the checkpoint with the lowest loss–and run it on the testing set–what you get is what you mention.</p>

username: ahmediqbal
cooked: <p>Thanks very much, and how we select results in 5 or 10-Fold cross validation?</p>

username: Hmrishav_Bandyopadhy
cooked: <p>You can summarize the results of a k-fold cross-validation run with the mean of the model scores. If you want to, you can also provide a measure of the variance of the scores for your experiments–maybe standard deviation?</p>
Random sampling from pixels of an image

username: pedram1
cooked: <p>Hi All</p>
<p>I have a dataset that has the shape [65536,1,94], which 65536 is number of my samples, 1 is num of the channel and 94 is length of the signal.(my dataset is 65536 signals).</p>
<p>Dose anyone know how I can split this dataset to test dataset and train dataset randomly? Randomly splitting is so important for me because  the number of datasets (65536) is actually pixels of an image (256*256).</p>
<p>tnx</p>

username: Nikronic
cooked: <p>Hi,</p>
<p>You can use <code>torch.utils.data.random_split</code>:</p>
<pre><code class="lang-auto">import torch
from torch.utils.data import random_split

data = torch.arange(0, 1000*94).reshape((1000, 1, 94))

train, test= random_split(data, [int(len(data)*0.9), int(len(data)*0.1)])

print(len(train))  # 900
print(len(test))  # 100
</code></pre>
<aside class="quote" data-post="1" data-topic="33987">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/7ea924/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-to-split-dataset-into-test-and-validation-sets/33987">How to split dataset into test and validation sets</a> 
  </div>
  <blockquote>
    I have a dataset in which the different images are classified into different folders. 
I want to split the data to test, train, valid sets. 
Please help
  </blockquote>
</aside>

<p>Bests</p>
Handling of Sparse Tensors

username: Byungheon_Jeong
cooked: <p>Does PyTorch take tensor sparsity into account when doing operations on these tensors?</p>
<p>I created a PyTorch script that created two torch.sparse.FloatTensor()s at various sparsity levels (0,100,10 interval) and multiplied them together using torch.sparse.mm().</p>
<blockquote>
<p>out1/out2 = torch.sparse.FloatTensor(indexTensor,valuesTensor).to_dense().to(device=cuda)</p>
</blockquote>
<blockquote>
<p>out2 = torch.transpose(out2, 0, 1).to(device=cuda)<br>
torch.sparse.mm(out1, out2)</p>
</blockquote>
<p>According to Nvidia’s profiler, every sparsity level used the same number of floating point operations, which suggests no performance difference between operations using sparse and operations using non sparse tensors.</p>
<p>Also tried this with a regular tensor and has the same results.</p>
Training model for another model

username: Frida
cooked: <p>Hi<br>
I want to use one model output as the parameters of another model, how can I do it?<br>
When doing something like:<br>
model.state_dict()[‘first_layer.conv.weight’].copy_(x1)<br>
It doesn’t really work.<br>
Cannot do the backward.<br>
Assume that x1 is the output of another model.<br>
Thanks a lot in advance</p>

username: little_adventurer
cooked: <p>Do you mean you would like to use one mode as a feature extractor?</p>

username: Frida
cooked: <p>not really. Specifically, I want to take one model’s output  say model A, and use it as the parameter of one layer in another model. But I want to update this layer following the model A. Any idea?</p>

username: little_adventurer
cooked: <p>Sorry, I want to be 100% certain.</p>
<p>You want to extract the output of layer N in model A.<br>
Take that output and feed it into model B, BUT continue training model A?</p>

username: Frida
cooked: <p>yes, I want to feed it such that this output is basically the parameters of some layer in model A</p>
Back-propagation through time and distributed training

username: TVN
cooked: <p>I have a Sampler to do backpropagation through time, similar to the one in the <a href="https://pytorchnlp.readthedocs.io/en/latest/source/torchnlp.samplers.html#torchnlp.samplers.BPTTBatchSampler" rel="nofollow noopener">torchnlp examples</a>. This requires giving the batch size of training as an argument so that batches are properly continuous (e.g. if the dataset is [abcdefghi] and batch size is 3, the batches are [adf] [beh] [cfi]).</p>
<p>My question is, how does this work in a distributed setting? From reading the code of the <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.distributed.DistributedSampler" rel="nofollow noopener">distributed sampler</a>, I got the impression that each process gets its copy of the sampler. In this case, I would need to know the local batch size in every process to properly separate batches. Is there a general rule to determine batch size per process (like an equal portion per GPU?), and if not, how could one determine the local batch size?</p>

username: mrshenli
cooked: <blockquote>
<p>Is there a general rule to determine batch size per process?</p>
</blockquote>
<p>Yes, if possible, it’s better that the data is evenly distributed on different processes, otherwise the process with lighter workload will have to frequently waiting for the stragglers, causing unnecessary slowdown.</p>
<p>Compare to that, if you are using <code>DistributedDataParallel</code> (DDP) a more important thing is that all processes must have the same number of forward/backward iterations, otherwise collective communications in DDP backward would hang.</p>
<blockquote>
<p>how could one determine the local batch size?</p>
</blockquote>
<p>See discussion here: <a href="https://discuss.pytorch.org/t/should-we-split-batch-size-according-to-ngpu-per-node-when-distributeddataparallel/72769" class="inline-onebox">Should we split batch_size according to ngpu_per_node when DistributedDataparallel</a></p>
Assigning values to a tensor after comparison between two other tensors

username: curious
cooked: <p>I have two 2d tensors like this:</p>
<pre><code class="lang-auto">x = tensor([[1311, 1297, 3872, 1299, 1445, 1299, 3876, 1298,    2],
        [1316, 1301, 3872, 1804, 1326, 3875, 1378, 1298,    2] ]])
y = tensor([[1311, 1297,    1, 1299, 1445, 1299,    2, 1298,    2],
        [1316, 1301,    1, 1804, 1326,    2, 1378, 1298,    2] ]])
</code></pre>
<p>The tensors differ from each other at some indices and I want another tensor to be updated based on the values in these tensors at differing positions. i.e.,</p>
<pre><code class="lang-auto">z= tensor([[   2,   16,   17,   10,   12,   13, 3876, 3872,   10,   14],
        [   2,   16,   17,   10,   12,   13, 3875, 3872,   10,   14]]])
</code></pre>
<p>z should become:</p>
<pre><code class="lang-auto">z= tensor([[   2,   16,   17,   10,   12,   13, 2, 1,   10,   14],
        [   2,   16,   17,   10,   12,   13, 2, 1,   10,   14]]])
</code></pre>
<p>How can this be done?</p>

username: ptrblck
cooked: <p>I don’t understand how <code>z</code> is created in your example.<br>
Based on <code>x</code> and <code>y</code>, the indices of the different values would be:</p>
<pre><code class="lang-python">(x != y).nonzero()
&gt; tensor([[0, 2],
          [0, 6],
          [1, 2],
          [1, 5]])
</code></pre>
<p>However, <code>z</code>  seems to be changed at columns 6 and 7 with the values 2 and 1?<br>
Could you explain how these values were set?</p>

username: curious
cooked: <p>z is a tensor which has some values. If z has some value &gt; 3870 in my case, we need to go the tensor x, and find its index. If the value is 3872, the index in tensor x is: [0][2] and its corresponding value in y tensor is 1 i.e., at index [0][2]. We need to substitute 3872’s value in z by corresponding value in y tensor. So, z[0][7] which had value 3872 becomes: 1.</p>
ModuleNotFound using C++/CUDA extension

username: aRI0U
cooked: <p>Hi,<br>
I want to include some C++/CUDA extension into some project. However, when I run my project, I have the following error:</p>
<pre><code class="lang-auto">ModuleNotFoundError: No module named 'HT.transformers.causal_product.causal_product_cpu'
</code></pre>
<p>My C++ files are located in folder <code>HT/transformers/causal_product</code> and are named <code>causal_product_cpu.cpp</code> and <code>causal_product_cuda.cu</code>. Those files come from another repository so I assume that my problem doesn’t come from here.<br>
I built the extensions running <code>python setup.py build_ext</code>. Here is the <code>setup.py</code> file:</p>
<pre><code class="lang-auto">from functools import lru_cache
from setuptools import find_packages, setup
from subprocess import DEVNULL, call

from torch.utils.cpp_extension import BuildExtension, CppExtension


@lru_cache(None)
def cuda_toolkit_available():
    try:
        call(["nvcc"], stdout=DEVNULL, stderr=DEVNULL)
        return True
    except FileNotFoundError:
        return False


def get_extensions():
    extensions = [
        CppExtension(
            "HT.transformers.causal_product.causal_product_cpu",
            sources=[
                "HT/transformers/causal_product/causal_product_cpu.cpp"
            ],
            extra_compile_args=["-fopenmp", "-ffast-math"]
        )
    ]
    if cuda_toolkit_available():
        from torch.utils.cpp_extension import CUDAExtension
        extensions += [
            CUDAExtension(
                "HT.transformers.causal_product.causal_product_cuda",
                sources=[
                    "HT/transformers/causal_product/causal_product_cuda.cu"
                ],
                extra_compile_args=["-arch=compute_50"]
            )
        ]
    print(extensions)
    return extensions


if __name__ == '__main__':
    setup(...,
          license='',
          packages=find_packages(exclude=['models']),
          zip_safe=False,
          ext_modules=get_extensions(),
          cmdclass={'build_ext': BuildExtension},
          install_requires=['torch'])
</code></pre>
<p>After I build extensions, a <code>build/</code> directory is created. In particular, subfolder <code>build/lib.linux-x86_64-3.6/HT/transformers/causal_product/</code> contains some <code>.so</code> files and subfolder <code>build/temp.linux-x86_64-3.6/HT/transformers/causal_product/</code> contains some <code>.o</code> files so I assume that they have been correctly been compiled.</p>
<p>It is the first time I try to include C++ code in a PyTorch project and I don’t know anything about c++ compilation, so I might have missed something obvious…<br>
Does anyone have an idea how to solve my problem?</p>
<p>Thanks in advance!<br>
Alain</p>
Combining Pytorch Geometric with Pytorch

username: 3michelin
cooked: <p>Is it possible to train a GAN that has a MLP generator and a GCN discriminator? The MLP generator is made using pytorch while the GCN discriminator is made using pytorch_geometric. I have included an example below where the generator generates a graph and the discriminator classifies if it is fake or not. The example I have provided is not the actual architecture I am trying to use, rather I have simplified it for the brevity and simplicity of the question.</p>
<pre><code class="lang-auto">import torch_geometric.data as data
from torch_geometric.nn import GCNConv
import torch.nn as nn
import torch.nn.functional as F
import torch

class Generator(nn.Module):
    def __init__(self):
        super().__init__()
        
        self.fc1 = nn.Linear(12, 25)
        
    def forward(self, z):

        return F.sigmoid(self.fc1(z))

class Discriminator(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = GCNConv(1, 64)
        self.fc1 = nn.Linear(64, 1)
    
    def forward(self, g):
        x, edge_index = g.x, g.edge_index
        x = self.conv1(x, edge_index)
        x = F.relu(x)
        x = self.fc1(x)
        return F.sigmoid(x).mean()

gen = Generator()
disc = Discriminator()
optimizer = torch.optim.Adam(gen.parameters(), lr=0.01)
loss_function = nn.BCELoss()

output = gen(torch.randn(12)).view((5,5))

# constructing COO connectivity matrix for Data object in pytorch geometric
edge_index = [[], []]
for i in range(5):
    for j in range(5):
        if output[i][j]&gt;=0.5:
            edge_index[0].append(i)
            edge_index[1].append(j)

# all node_features are set to 1 just for the example
d = data.Data(x=torch.Tensor([[1] for x in range(5)]), edge_index=torch.LongTensor(edge_index))

# classification of generated graph as real or fake by discriminator
v = disc(d)

# ground truth
real = torch.FloatTensor(1, 1).fill_(1.0)

# loss and optimization
loss = loss_function(v, real)
loss.backward()
optimizer.step()
</code></pre>
<p>The example just walks through one graph being generated and loss being calculated, again for brevity. The discriminator training is not included because it seems to be working fine. However, after trying this out, the weights in the generator are not being updated. I assume this is because of the <code>grad</code> not being transferred from the generator output to the <code>torch_geometric.data.Data</code> object. Does anyone know how to back propogate for a GAN in a situation like this? Is it even possible with the combination of pytorch and pytorch_geometric? Thanks!</p>
How to separate gradients from different losses on the same layer

username: xuxuxu
cooked: <p>Pytorch seems to only aggregate (accumulate) gradients on the same layer.</p>
<pre><code class="lang-auto">optimizer.zero_grad()
loss1.backward(retain_graph=True)
gradient_1 = get_layer_gradient(model, layer_name='common_layer')
optimizer.zero_grad()
loss2.backward(retain_graph=True)
gradient_2 = get_layer_gradient(model, layer_name='common_layer')
optimizer.zero_grad()
loss3.backward(retain_graph=True)
gradient_3 = get_layer_gradient(model, layer_name='common_layer')

optimizer.zero_grad()
loss = loss1 + loss2 + loss3
loss.backward()
gradient_4 = get_layer_gradient(model, layer_name='common_layer')
</code></pre>
<p>I found that <code>gradient_1 + gradient_2 + gradient_3 != gradient_4</code>. Why?<br>
How can I get the exact three gradients by the three losses, which should add up to <code>gradient_4</code> ?</p>
<h2>Feel free to discuss!</h2>

username: xuxuxu
cooked: <p>Specifically, by comparing them step by step, I saw that <code>(gradient_1 + gradient_2 + gradient_3).norm()</code> equals to <code>gradient_4.norm()</code>, however, <code>(gradient_1 + gradient_2 + gradient_3).norm() == gradient_4.norm()</code> is False. This is consistent after hundreds of iterations… Is it because of <strong>numerical instability</strong>? If so, does <i>above code<i> the right way to inspect different gradients by different losses?</i></i></p>

username: Kushaj
cooked: <p>Use <code>torch.allclose</code> to compare two float tensors if you think the issue might be due to numerical instability. On printing <code>grad1+grad2+grad3</code> and <code>grad4</code>, are the values different?</p>

username: xuxuxu
cooked: <p>Hi, I tried <code>torch.allclose(grad1+grad2+grad3, grad4)</code> and it returns <code>True</code>. That means the two gradient matrices are element-wise equal within a tolerance. The printed values of them are the same but they just do not equal each other.</p>
<p>So, how to explain this? Why the addition of <code>gradient_1 + gradient_2 + gradient_3</code> is slightly different from using <code>loss = loss1 + loss2 + loss3</code> to do backpropagation at a time? shouldn’t they differentiate on the same graph three times and accumulate (add) gradients to be <code>gradient_4</code>?</p>

username: Kushaj
cooked: <p>You can only get 6 digits of precision with <code>torch.float32</code>, so if the gradient values are same till 6 digits of precision then it means your observations are a limitation of <code>float32</code>. Maybe adding <code>loss1+loss2+loss3</code> results in the floating point error. Try printing <code>loss</code> and <code>loss1, loss2, loss3</code> individually to see if they add up.</p>
<p>You can use <code>torch.set_printoptions(precision=10)</code> to show 10 digits in float.</p>
Create nn.Module directly on device

username: Dmitrii_Khizbullin
cooked: <p>Is it possible to create a module directly on a specific cuda device?<br>
Normally I create mod=Linear(in_channels, out_channels) and get a module on cpu. Then I have to push a module to a cuda device with mod.to(“cuda:0”). How would I create a linear layer directly on a cuda device without having to transfer tensors through a cuda stream? In my case I do not need this module on cpu at all and would like to save time by directly instantiating my module on a specific device.<br>
In general the task is to create a module that is dummy and does not store parameter tensors. Instead my dummy module will re-allocate its parameter tensors in forward() and then discard them. This will allow me to profile module’s execution wall time if my total model does not fit GPU RAM.<br>
Thanks!</p>

username: ptrblck
cooked: <p>I don’t know, if there is a proper method for built-in <code>nn.Modules</code>, but if you are concerned about profiling or debugging, you could create a custom module and create the parameters directly on the device.</p>

username: Dmitrii_Khizbullin
cooked: <p>Thanks for the reply! Yes, seems that I need to copy-paste all the modules like Linear to my code and add a constructor argument what device to create module’s parameters on. Not as elegant as I wish it were <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Faster RCNN with inceptionv3 backbone very slow

username: LHFO94
cooked: <p>Hi,</p>
<p>I want to train a faster-rcnn model with an InceptionV3 backbone. I have managed to produce code that works, the problem is however that it trains very slow in comparison to other backbones. My code looks as follows:</p>
<pre><code class="lang-auto">def initiate_inceptionv3(num_classes):
        inception = torchvision.models.inception_v3(pretrained=True, aux_logits=False)
        modules = list(inception.children())[:-1]
        backbone = nn.Sequential(*modules)

        for layer in backbone:
            for p in layer.parameters():
                p.requires_grad = False

        backbone.out_channels = 2048

        anchor_generator = AnchorGenerator(sizes=((32, 64, 128, 256, 512),),
                                           aspect_ratios=((0.5, 1.0, 2.0),))

        model = FasterRCNN(backbone, rpn_anchor_generator=anchor_generator,
                           box_predictor=FastRCNNPredictor(1024, num_classes))

        return model
</code></pre>
<p>Any suggestion as to why it runs so slow and how to improve? Thanks in advance</p>