akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Change the one parameter of a mode and cause UserWarning

username: jerly-hjzhou
cooked: <p>I change the first parameter of resnet-18 mannully and use the changed model to infer. And I got this warning information. <code>UserWarning: The .grad attribute of a Tensor that is not a leaf Tensor is being accessed. Its .grad attribute won't be populated during autograd.backward(). If you indeed want the gradient for a non-leaf Tensor, use .retain_grad() on the non-leaf Tensor. If you access the non-leaf Tensor by mistake, make sure you access the leaf Tensor instead. See github.com/pytorch/pytorch/pull/30531 for more informations.   if param.grad is not None:</code>  But I couldn’t understand. Could anyone teach me why or what causes it?<br>
My code is below.</p>
<pre><code class="lang-auto">import torch
import torch.nn.functional as F
from torchvision import datasets, transforms
import time
model = torch.load('resnet-18/resnet_18-cifar_10.pth')  # 加载模型
device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')
print(device)
for layerPara in model.parameters():
    layerPara[0][0][0][0] = -6.0173e+3
    break
model.to(device)
model.eval()
lossSum = 0.0
preLoss = 0.0
curLoss = 0.0
accuracy = 0.0
correctNum = 0
data_transforms = transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))
])
image_datasets = datasets.CIFAR10(root='./data', train=False,
                                  download=True, transform=data_transforms)
test_loader = torch.utils.data.DataLoader(image_datasets, batch_size=500,
                                          shuffle=True, num_workers=0, prefetch_factor=2)
since = time.time()
for data, target in test_loader:
    data = data.to(device)
    target = target.to(device)
    output = model(data)
    lossSum = lossSum + F.cross_entropy(output, target, reduction='sum').item()
    pred = output.data.max(1, keepdim=True)[1]  # get the index of the max log-probability
    correctNum += pred.eq(target.data.view_as(pred)).cpu().sum()  
</code></pre>

username: ptrblck
cooked: <p>I’m not sure where this warning is raised, since I cannot see any usage of the <code>.grad</code> attribute.<br>
However, your current code should fail with:</p>
<pre><code class="lang-python">RuntimeError: a view of a leaf Variable that requires grad is being used in an in-place operation.
</code></pre>
<p>and you would need to manipulate the parameter in a <code>torch.no_grad()</code> block, which seems to work fine:</p>
<pre><code class="lang-python">model = models.resnet18()

with torch.no_grad():
    for layerPara in model.parameters():
        layerPara[0][0][0][0] = -6.0173e+3
        break
    
x = torch.randn(2, 3, 224, 224)
target = torch.randint(0, 1000, (2,))
criterion = nn.CrossEntropyLoss()

out = model(x)
loss = criterion(out, target)
loss.backward()
</code></pre>

username: jerly-hjzhou
cooked: <p>You’re so kind. Thank you!</p>
Resuming a trained model to be trained for longer epochs

username: 887574002
cooked: <p>Hi, I have a segmentation model to segment two datasets, <code>Camvid</code> and <code>Cityscapes</code>. I do not know how long should I train the model. So, I set the model to be trained for 400 epochs and when the training was finished by looking at learning curves I found that the loss is still decreasing and the Dice is improving. So, can I load the trained model and let it to be trained for longer epochs, instead of training it from scratch?</p>

username: pascal_notsawo
cooked: <p>I think so, it’s even one of the essences of saving models if I’m not mistaken.<br>
If your 400 epochs only take a few minutes (I’m not familiar with model segmentation), you can start again without any problem. But when you have to start training models for days, it becomes more than necessary.</p>

username: 887574002
cooked: <p>Ok, thank you for your answer.</p>
Why rescale to [0,1] before normalizing images?

username: FaultyBagnose
cooked: <p>The Pytorch doc says:<br>
“All pre-trained models expect input images normalized in the same way,” … “The images have to be loaded in to a range of <code>[0, 1]</code> and then normalized using <code>mean = [0.485, 0.456, 0.406]</code> and <code>std = [0.229, 0.224, 0.225]</code> .”<br>
Mathematically, the normalization to a given mean and std should give the same result regardless of any prior linear scaling.<br>
Maybe I’m misunderstanding what is meant by “loading in to a range”, but if it means an affine transform that maps the min to 0 and max to 1, performing this operation prior to the normalization should have no effect?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="FaultyBagnose" data-post="1" data-topic="120295">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/f/96bed5/48.png" class="avatar"> FaultyBagnose:</div>
<blockquote>
<p>Maybe I’m misunderstanding what is meant by “loading in to a range”, but if it means an affine transform that maps the min to 0 and max to 1, performing this operation prior to the normalization should have no effect?</p>
</blockquote>
</aside>
<p>That’s correct and you could skip the <code>[0, 1]</code> normalization and use the <code>mean</code> and <code>std</code> stats for the raw input values in <code>[0, 255]</code>. However, since <code>ToTensor()</code> is often used in the transformations to transform a <code>PIL.Image</code> in <code>[0, 255]</code> to a tensor in <code>[0, 1]</code>, I believe this workflow grew organically.</p>
Kornia remap function

username: Alon_Zolfi
cooked: <p>Hey everyone,</p>
<p>I’m trying to convert a code that uses cv2 to a diffrentiable pytorch one.<br>
that code uses cv2.remap function with the following parameters:<br>
src.shape - (3, 418, 818)<br>
map_x.shape - (256,256)<br>
map_y.shape - (256,256)</p>
<p>I tried to use the same parameters with kornia.geomtry.remap (from my understanding the functions should do the same operation), however, the kornia code crushes with the following error:<br>
ValueError: Inputs last two dimensions must match.<br>
which means kornia doesn’t accept different image and maps shapes, unlike cv2.</p>
<p>Is there any workaround?</p>

username: edgarriba
cooked: <p><a class="mention" href="/u/alon_zolfi">@Alon_Zolfi</a> thanks for the feedback.</p>
<p>Definitely we can support that from kornia side. We just need to remove an assertion in the input.</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/kornia/kornia/blob/da3aceb876e0f92b65405a20198836c08ee80d10/kornia/geometry/transform/imgwarp.py#L493-L494" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/kornia/kornia/blob/da3aceb876e0f92b65405a20198836c08ee80d10/kornia/geometry/transform/imgwarp.py#L493-L494" target="_blank" rel="noopener nofollow ugc">kornia/kornia/blob/da3aceb876e0f92b65405a20198836c08ee80d10/kornia/geometry/transform/imgwarp.py#L493-L494</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="493" style="counter-reset: li-counter 492 ;">
<li>if not tensor.shape[-2:] == map_x.shape[-2:] == map_y.shape[-2:]:</li>
<li>    raise ValueError("Inputs last two dimensions must match.")</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>Would you mind to open a ticket in kornia or a quick PR addressing that ?</p>
Dropout2d is too slow

username: AntMorais
cooked: <p>I am running the example at <a href="https://github.com/pytorch/examples/blob/master/cpp/mnist/mnist.cpp" rel="noopener nofollow ugc">https://github.com/pytorch/examples/blob/master/cpp/mnist/mnist.cpp</a> and the training is very slow.<br>
After debugging I found that the Dropout2d layer ( <code>conv2_drop-&gt;forward(x)</code> ) is taking around 20 seconds with CUDA and an RTX 2060. I don’t know why it is taking so long, I’d appreciate if you could help!<br>
Here’s my env:</p>
<pre><code class="lang-auto">PyTorch version: 1.8.1+cu102
Is debug build: False
CUDA used to build PyTorch: 10.2
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.2 LTS (x86_64)
GCC version: (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
Clang version: Could not collect
CMake version: version 3.18.2

Python version: 3.8 (64-bit runtime)
Is CUDA available: True
CUDA runtime version: 11.2.67
GPU models and configuration: GPU 0: GeForce RTX 2060
Nvidia driver version: 460.27.04
cuDNN version: Probably one of the following:
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn.so.8.1.1
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_adv_infer.so.8.1.1
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_adv_train.so.8.1.1
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_cnn_infer.so.8.1.1
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_cnn_train.so.8.1.1
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_ops_infer.so.8.1.1
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_ops_train.so.8.1.1
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.19.2
[pip3] numpydoc==1.1.0
[pip3] torch==1.8.1
[conda] blas                      1.0                         mkl  
[conda] magma-cuda110             2.5.2                         1    pytorch
[conda] magma-cuda112             2.5.2                         1    pytorch
[conda] mkl                       2020.2                      256  
[conda] mkl-include               2020.2                      256  
[conda] mkl-service               2.3.0            py38he904b0f_0  
[conda] mkl_fft                   1.2.0            py38h23d657b_0  
[conda] mkl_random                1.1.1            py38h0573a6f_0  
[conda] numpy                     1.19.2           py38h54aff64_0  
[conda] numpy-base                1.19.2           py38hfa32c7d_0  
[conda] numpydoc                  1.1.0              pyhd3eb1b0_1  
[conda] torch                     1.8.1                    pypi_0    pypi
</code></pre>

username: ptrblck
cooked: <p>How did you profile the code?<br>
Note that CUDA operations are executed asynchronously, so I assume you’ve properly synchronized the code and isolated the bottleneck to the dropout layer or have used other profiling tools such as <code>nsys</code>?</p>
Inplace operation changes is_leaf attribute?

username: wusuowei60
cooked: <p>This question is about tensors, computation graphs and inplace operations.</p>
<p>Here are some operations that try to modify the same underlying data in storage.</p>
<pre><code class="lang-python">c = torch.randn(10, 3, 2, 32, requires_grad=True)
print(c.is_leaf)
# operation 1
try:
    c += 1  # RuntimeError: a leaf Variable that requires grad is being used in an in-place operation.
except RuntimeError:
    print(RuntimeError)
# operation 2
try:
    torch.add(c, 1, out=c)  # RuntimeError: add(): functions with out=... arguments don't support automatic differentiation, but one of the arguments requires grad. Grad will not be copied (only data will be copied).
except RuntimeError:
    print(RuntimeError)
# operation 3
try:
    c.add_(1)  # RuntimeError: a leaf Variable that requires grad is being used in an in-place operation.
except RuntimeError:
    print(RuntimeError)
# operation 4
try:
    c.index_copy_(3, torch.tensor(10), torch.randn(10, 3, 2, 1))  # RuntimeError: a leaf Variable that requires grad is being used in an in-place operation.
except RuntimeError:
    print(RuntimeError)
# operation 5
d = c.detach()
d += 1  # no error, and still leaf
print(c.is_leaf)
# operation 6
c[:] += 1  # no error, but not leaf anymore
print(c.is_leaf)
</code></pre>
<p>There are even more such operations if you consider <code>numpy()</code> and <code>from_numpy()</code>.</p>
<p>What is going on here😨? Why sometimes you can operate the underlying data of tensor <code>c</code> while sometimes not? And why does inplace operation on indexed tensor make <code>is_leaf</code> <code>False</code>.</p>
<p>And what’s the lesson here? Are such behaviors intended and have some special use cases? Or should we try to avoid inplace operation on tensor that requires grad?</p>
Why use 4 GPU with DataParallel strangely slow the speed of inference compared with no DataParallel?

username: jerly-hjzhou
cooked: <p>I’m using resnet-18 to infer test set. When I use one GPU, the time is only 10s. But when I want to use multiple GPUs and only add this code</p>
<pre><code class="lang-auto">if torch.cuda.device_count() &gt; 1:
  print("Let's use", torch.cuda.device_count(), "GPUs!")
  model = torch.nn.DataParallel(model)
</code></pre>
<p>The result look like this<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/b/6/b61279483d750c5c223c4922eaed297d042fd386.png" alt="image" data-base62-sha1="pYGtJ4nBjU8JtGuZbVv3fk1OfkO" width="494" height="65"><br>
which cost 24s.<br>
My whole code is below.</p>
<pre><code class="lang-auto">#coding=gbk
import torch
import torch.nn.functional as F
from torchvision import datasets, transforms
import time
model = torch.load('resnet_18-cifar_10.pth')
device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')
#if torch.cuda.device_count() &gt; 1:
#  print("Let's use", torch.cuda.device_count(), "GPUs!")
#  model = torch.nn.DataParallel(model)
model.to(device)
model.eval()
lossSum = 0.0
preLoss = 0.0
curLoss = 0.0
accuracy = 0.0
correctNum = 0
data_transforms = transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))
])
image_datasets = datasets.CIFAR10(root='../data', train=False,
                                  download=True, transform=data_transforms)
test_loader = torch.utils.data.DataLoader(image_datasets, batch_size=500,
                                          shuffle=True, num_workers=4, prefetch_factor=2)
since = time.time()
for data, target in test_loader:
    data = data.to(device)
    target = target.to(device)
    output = model(data)
    lossSum = lossSum + F.cross_entropy(output, target, reduction='sum').item()
    pred = output.data.max(1, keepdim=True)[1]  
    correctNum += pred.eq(target.data.view_as(pred)).cpu().sum() 
curLoss = lossSum / len(test_loader.dataset
accuracy = correctNum / len(test_loader.dataset)
time_elapsed = time.time() - since
print('Test complete in {:.0f}m {:.0f}s'.format(
    time_elapsed // 60, time_elapsed % 60))
</code></pre>

username: ptrblck
cooked: <p><code>nn.DataParallel</code> will scatter and gather the parameters, gradients, etc. between the devices as explained in <a href="https://medium.com/huggingface/training-larger-batches-practical-tips-on-1-gpu-multi-gpu-distributed-setups-ec88c3e51255">this blog post</a> and is thus adding some overhead. However, it should not slow down the execution assuming you’ve scaled up the batch size.</p>
<p>We generally recommend to use <code>DistributedDataParallel</code> with a single process per device for the best performance.</p>
Adding index to the SHVN dataset

username: cheng_chen1
cooked: <p>Hi, there could anyone help me create a data loader for the SVHN dataset with an index in it, please.</p>

username: ptrblck
cooked: <p>I’m not sure I understand the question correctly, since the <code>index</code> will be used in the <code>__getitem__</code> of the dataset. In case you would like to return the index with the processed data sample and target, you could write a custom <code>Dataset</code> and return all three values in them.<br>
In your custom <code>Dataset</code> you could just index the <code>SVHN</code> dataset using the <code>index</code>.</p>
Generating 28x28 image using DCGAN

username: alexander1709
cooked: <p>I have a python test code which predefines nz=10, ngf= 64</p>
<pre><code class="lang-auto">def test_Generator_shapes():
    nz = 10
    netG = Generator(nz, ngf=64, nc=1)

    batch_size = 32
    noise = torch.randn(batch_size, nz, 1, 1)
    out = netG(noise, verbose=True)

    assert out.shape == torch.Size([batch_size, 1, 28, 28]), f"Bad shape of out: out.shape={out.shape}"
    print('Success')

test_Generator_shapes()
</code></pre>
<p>Now I need to reset the hidden layers and other parameters to be able to output imamges of size 28x28, i.e.- torch.Size([batch_size, 1, 28, 28])</p>
<p>Please can someone suggest what changes I should do in the following code so as to be able to generate images of 28x28 instead of 64x64 presently</p>
<pre><code class="lang-auto">class Generator(nn.Module):
    def __init__(self, nz=10, ngf=28, nc=1, ndf=28):
        """GAN generator.
        
        Args:
          nz:  Number of elements in the latent code.
          ngf: Base size (number of channels) of the generator layers.
          nc:  Number of channels in the generated images.
        """
        ngf=28
        super(Generator, self).__init__()
        self.ngpu = 0
        self.main = nn.Sequential(
            # input is Z, going into a convolution
        
            nn.ConvTranspose2d( nz, ngf * 8, 4, 1, 0, bias=False),
            nn.BatchNorm2d(ngf * 8),
            nn.ReLU(True),
            # state size. (ngf*8) x 4 x 4
            nn.ConvTranspose2d(ngf * 8, ngf * 4, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ngf * 4),
            nn.ReLU(True),
            # state size. (ngf*4) x 8 x 8
            nn.ConvTranspose2d( ngf * 4, ngf * 2, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ngf * 2),
            nn.ReLU(True),
            # state size. (ngf*2) x 16 x 16
            nn.ConvTranspose2d( ngf * 2, ngf, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ngf),
            nn.ReLU(True),
            # state size. (ngf) x 32 x 32
            nn.ConvTranspose2d( ngf, nc, 4, 2, 1, bias=False),
            nn.Tanh()
            # state size. (nc) x 64 x 64
        )
        # YOUR CODE HERE
        #raise NotImplementedError()

    def forward(self, z, verbose=False):
        """Generate images by transforming the given noise tensor.
        
        Args:
          z of shape (batch_size, nz, 1, 1): Tensor of noise samples. We use the last two singleton dimensions
                          so that we can feed z to the generator without reshaping.
          verbose (bool): Whether to print intermediate shapes (True) or not (False).
        
        Returns:
          out of shape (batch_size, nc, 28, 28): Generated images.
        """
        # YOUR CODE HERE
        x = self.main(z)
        print (x.size())
        return x
        #raise NotImplementedError()
</code></pre>
Detect underfitting and overfitting when validation set is small

username: mfoglio
cooked: <p>Hello everyone, I know what you are thinking: “yet again another question on a neural network size and hyperparameters”. My question is specific to a given scenario:</p>
<ul>
<li>I have a small dataset</li>
<li>I am classifying images into two classes</li>
<li>I am afraid of overfitting</li>
<li>I am afraid of underfitting</li>
<li>My neural network reaches 90% accuracy on the validation dataset after one epoch</li>
<li>Adding or removing one or two layers out of 6 does not seem to change performance</li>
</ul>
<p>Does this mean that I am creating a model bigger than needed?<br>
I tried to create a smaller architecture. This smaller architecture reached the same accuracy as a bigger one on the validation set. However, the bigger network (<strong>sometimes</strong> - depending on the training) seems to work better in deployment (no labeled data - just mere human observations). This is not always true: sometimes retraining the bigger network from scratch leads to poorer results. Also, after training for a few epochs the training and validation losses drop and got stuck at some terrible local minima.<br>
My conclusions are that:</p>
<ul>
<li>My validation dataset does not represent a good sample of the real case scenarios</li>
<li>I need to decrease the learning rate</li>
</ul>
<p>However, I am not sure anymore if I am underfitting or overfitting. Based on your experience:</p>
<ol>
<li>
<p><em>Is there still anything I can do, besides collecting more data, to understand if I am underfitting or overfitting the data? I am not sure whether I should decrease the number of layers, or decreasing it.</em></p>
</li>
<li>
<p>This is a simple classification problem that takes in input grayscale images of 200x200 pixels. The goal is to detect images with certain blobs of white color (of different shapes) while discarding others that may contain some noise. I have only a few hundreds of examples. <em>Do you think using 6 layers and about 30k parameters overall could be overkilling?</em> I am mainly using 3x3 convolutions followed by 1x1 to reduce parameters. The challenge is the lack of data.</p>
</li>
</ol>
How to understand conv3d input and oupout?

username: Marcus_Brown
cooked: <p>Hello,</p>
<h2><a name="from-the-documentation-on-3d-convolution-1" class="anchor" href="#from-the-documentation-on-3d-convolution-1"></a>from the documentation on 3d convolution,</h2>
<h2><a name="h-3d659x49uploadgsnabsf1svbhxeka1zvnqp1ggfgpng-2" class="anchor" href="#h-3d659x49uploadgsnabsf1svbhxeka1zvnqp1ggfgpng-2"></a><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/6/76484422bba5ba343fe8b7ccbaa47c88e66f7e4a.png" alt="3d" data-base62-sha1="gSnaBsF1sVbhXeKa1zVNQp1gGfg" width="659" height="49"></h2>
<p>How to understand the D ? in (N, C, D, H, W)?<br>
let’s say for example I have five video frames and I stack the frames along the channel dimension giving me :<br>
a (1, 15, H, W) tensor assuming RGB frames. How do I reshape this tensor to (N, C, D, H, W)</p>

username: Kapil_Rana
cooked: <p>N is the batch size.<br>
H and W are the height and width of the video frame.<br>
Now C and D. I am not sure about which one is the 3 for RGB and 5 for video frames.<br>
I guess 5 would the channels, C=5, and D=3 for the dimension of each video frame.<br>
You cant predefined the batch size in getitem().<br>
try to debug with batch size = 1.</p>

username: Marcus_Brown
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> could you please explain this ? the documentation does not explain the D in (N, C, D, H, W).</p>

username: ptrblck
cooked: <p>The <code>D</code> dimension should define the “depth” of a volume for 3D inputs or parameters.<br>
While 2-dimensional data defines spatial dimensions as height and width, 3-dimensional data uses depth, height, and width.</p>

username: Marcus_Brown
cooked: <p>For my example above, Given a Tensor (N, 15, H, W) in 2d. Its 3D equivalent is : (N, 3, 5, H, W) where 3 is the number of channels for RGB and 5 the number of frames.</p>
Deleting a deep copy from cpu

username: vhalpurohit
cooked: <p>I’m trying to load a large tensor from a file, and save its slices as individual “pt” files. Now since views share the same storage as the original tensors, simply iterating over the tensor and saving the slices doesn’t work (that saves the whole tensor for each iteration), and I have to create a deep copy of the slice during every iteration. Eventually these copies fill my RAM up and the machine crashes.<br>
I tried deleting the deep copies at the end of each iteration using <code>del</code> but that didn’t work. So how do you delete a deep copy?</p>
<pre><code class="lang-auto">            tensor = torch.load(os.path.join(inp_dir,vid_tensor))
            logging.info(f"\n\nSuccessfuly loaded label: {label}")
            logging.info(f"Label shape is: {tensor.shape}\n\n")

            for idx,chunk in enumerate(tensor):
                chunk_name = os.path.join(label_output_dir,f"{label}_{idx}.pt")
                if not os.path.exists(chunk_name):
                    slice = chunk.detach().clone()
                    torch.save(slice,chunk_name)
                    logging.info(f"Successfuly saved {chunk_name}")
                    logging.info(f"chunk has shape {slice.shape}")
                    del slice #doesn't work
</code></pre>

username: eqy
cooked: <p>I tried to repro this with a minimal example:</p>
<pre><code class="lang-auto">import torch

tensor = torch.randn(32768, 32768)
for i in range(tensor.size(1)):
    slice = tensor[:,i].detach().clone()
    torch.save(slice, 'asdf')
</code></pre>
<p>But I only see constant memory usage of about ~4GiB here, which is what we would expect from 2**30 elements * 4 bytes per element.</p>
“torchvision.transforms.ToTensor()‘ didn't scale to 0~1

username: Hao_Meng
cooked: <p>i use this function to preprocess the image of numpy.ndarray type.But it seemed not to divide to 255.<br>
The source code is as follows:</p>
<pre><code class="lang-auto">    if isinstance(pic, np.ndarray):
        # handle numpy array
        if pic.ndim == 2:
            pic = pic[:, :, None]

        img = torch.from_numpy(pic.transpose((2, 0, 1)))
        # backward compatibility
        if isinstance(img, torch.ByteTensor):
            return img.float().div(255)
        else:
            return img
</code></pre>
<p>It looked like it is not the ‘torch.ByteTensor’ type, so didn’t be divided to 255.<br>
What is the reason?</p>

username: eqy
cooked: <p>Looking at the <code>ToTensor</code> implementation, we can see that it converts to the <code>float</code> dtype after scaling. So if a <code>float</code> type tensor appears here or elsewhere in the pipeline, it is usually expected to have already been scaled. You can change the method used to load the images or just add the scaling yourself (there’s no strong reason it has to be in <code>ToTensor</code>)</p>
When do we need padding_idx in nn.Embedding?

username: Zhaofeng_Wu
cooked: <p>If we use pack_padded_sequence and ignore_idx in F.cross_entropy, do we still need set padding_idx in nn.Embedding? If we do, why?</p>

username: yipliu
cooked: <p>Yes,  the same problem bothers me. Do you have any ideas now?</p>
If a transposed convolution layer takes in a 3 channel input and gives out a 2 channel output, how many kernels would it have and why?

username: Jide_Olabisi
cooked: <p>Isn’t the tconv layer supposed to have 6 kernels divided into two? One three kernels for one channel, another three kernels for another channel making two channels because the tconv layer is to give out a two channel output? I’m I right or what?</p>

username: ptrblck
cooked: <p>No, the <code>out_channels</code> value defines the number of kernels, and since it’s set to <code>2</code> the layer would use <code>2</code> kernels.<br>
<a href="https://cs231n.github.io/convolutional-networks/#conv">CS231n - Convolutional Layer</a> explains the kernel shapes with some nice illustrations and <a href="https://arxiv.org/abs/1603.07285">A guide to convolution arithmetic for deep learning</a> is one of the best references when it comes to shape calculations.</p>
RuntimeError: expected scalar type Char but found Float [please help!]

username: David_Kebudi
cooked: <p>I have built my own DGL Dataset using the code below:</p>
<pre><code class="lang-auto">class ORValidationDataset(DGLDataset):

    def __init__(self):

        super().__init__(name='OR')

    def process(self):

        nodes_data = members

        edges_data = interactions

        

        graph_type = torch.from_numpy(nodes_data['topic'].astype('category').cat.codes.to_numpy())

        node_labels = torch.from_numpy(nodes_data['type'].astype('category').cat.codes.to_numpy())

                

        # graph_type = torch.CharTensor(nodes_data['topic'])

        # node_labels = torch.CharTensor(nodes_data['type'])

        

        edge_features = torch.from_numpy(edges_data['Weight'].to_numpy())

        edges_src = torch.from_numpy(edges_data['Src'].to_numpy())

        edges_dst = torch.from_numpy(edges_data['Dst'].to_numpy())

        self.graph = dgl.graph((edges_src, edges_dst), num_nodes=nodes_data.shape[0])

        self.graph.ndata['node_type'] = node_labels

        self.graph.ndata['graph_type'] = graph_type

        self.graph.edata['weight'] = edge_features

    def __getitem__(self, i):

        return self.graph

    def __len__(self):

        return 1
</code></pre>
<p>However when I try to run the following training pipeline to predict the links between the nodes, I get a ‘expected scalar type Char’ error. What am I doing wrong in terms of types? I know that there is a mismatch but I can’t seem to figure it out. Thanks!</p>
<pre><code class="lang-auto">optimizer = torch.optim.Adam(itertools.chain(model.parameters(), pred.parameters()), lr=0.01)

all_logits = []
for e in range(100):
    # forward
    h = model(train_g, train_g.ndata['graph_type'])
    pos_score = pred(train_pos_g, h)
    neg_score = pred(train_neg_g, h)
    loss = compute_loss(pos_score, neg_score)

    # backward
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

    if e % 5 == 0:
        print('In epoch {}, loss: {}'.format(e, loss))

# ----------- 5. check results ------------------------ #
from sklearn.metrics import roc_auc_score
with torch.no_grad():
    pos_score = pred(test_pos_g, h)
    neg_score = pred(test_neg_g, h)
    print('AUC', compute_auc(pos_score, neg_score))
</code></pre>
<p>the error comes from the conv1() in the forward function in the model. Here is the code for that as well:</p>
<pre><code class="lang-auto">class GraphSAGE(nn.Module):
    def __init__(self, in_feats, h_feats):
        super(GraphSAGE, self).__init__()
        self.conv1 = SAGEConv(in_feats, h_feats, 'mean')
        self.conv2 = SAGEConv(h_feats, h_feats, 'mean')

    def forward(self, g, in_feat):
      import pdb; pdb.set_trace()
      h = self.conv1(g, in_feat)
      h = F.relu(h)
      h = self.conv2(g, h)
      return h
</code></pre>
Key Error, custom pytorch dataset class

username: borteley
cooked: <p>Hi all experts,</p>
<p><strong>I’m new to pytorch and I’ve got a KEY ERROR  issue below: Please help me solve it.</strong></p>
<p><strong>I have a created custom dataset class to access category columns for embedding and numerical columns separately, but I get key error when I try to iterate through elements of the data. Please see my code below:</strong></p>
<p>class LoanDataset(Dataset):</p>
<pre><code>def __init__(self, X, Y, emb_cols):
    X = X.copy()
    self.X1 = X.loc[:,emb_cols].copy().values.astype(np.int64) #categorical columns
    self.X2 = X.drop(columns=emb_cols).copy().values.astype(np.float32) #numerical columns
    self.y = Y
    
def __len__(self):
    return len(self.y)

def __getitem__(self, idx):
    return self.X1[idx], self.X2[idx], self.y[idx]
</code></pre>
<p><em><strong>Next, I apply this to my x_train and x_test data like so:</strong></em></p>
<p>train_ds = LoanDataset(x_train, y_train, emb_cols)</p>
<p>test_ds = LoanDataset(x_test, y_test, emb_cols)</p>
<p>batch_size = 64<br>
traindl = DataLoader(x_train, batch_size=batch_size,shuffle=True)<br>
testdl = DataLoader(x_test, batch_size=batch_size,shuffle=True)</p>
<p>I try to view elements of the train_ds with code below and get I key error:1</p>
<p>i = 1<br>
for X1, X2, y in train_ds:<br>
print(‘batch_num:’, i)<br>
i = i+1<br>
print(X1,X2,y)</p>
<p>So I tried to view train_ds[1] and I get key error  1 as well. PLEASE HELP. Would appreciate your support, Thanks</p>

username: eqy
cooked: <p>It’s a little difficult to see what the exact problem is without knowing what the format of x_train, and y_train, etc. are when creating the dataset. It could just be that Y is a wrapped version of the data. You can check this via <code>len(X1)</code>, <code>len(X2)</code>, <code>len(Y)</code>; if one is length 1 then the data needs to be unwrapped.</p>

username: borteley
cooked: <p>hi eqy,<br>
len(X1) = len(X2)=len(y) = 34056 samples/rows</p>
<p>x_train and y_train are data frames, created using sklearn train_test_split function</p>

username: eqy
cooked: <p>It might be that even if the len (e.g., of Y) is correct, it might not like indexing directly. Can you verify the types of X1, X2, y, to check if something like <code>y.values</code> is needed?</p>

username: borteley
cooked: <p>Ok,  found out that y  is a pandas series  while X1 and X2  are nd.arrays</p>

username: eqy
cooked: <p>Right, so you might need something like the .values.astype as done on the other inputs if the dataframe doesn’t support indexing. You can also test indexing each of the self member variables directly to see which one(s) are causing the issue.</p>

username: borteley
cooked: <p>after i changed y from pandas series to nd.array the for loop is running well with no errors,  but I get error in my training loop now troubleshooting to see what the error is</p>
On-the-fly image rotation cpu bottleneck

username: eballest
cooked: <p>I am training a rotation detection model rotating the images during training (instead of as a preprocessing stage). This rotation in the cpu becomes the bottleneck (disabling it reduces epoch time from 74s to 6s and drastically improves GPU utilization). I was looking for suggestions to improve my training performance.</p>
<p>My Dataset class looks something like this:</p>
<pre><code class="lang-python">class RotatingDataset(Dataset):
    ...
    def __getitem__(self, idx):
        target = np.random.uniform(-180, 180)
        image = read_pil_image(idx) #~380x380 image

        # relevant bit
        image = scipy.ndimage.rotate(image, target, reshape=True, mode='nearest')
        image = Image.fromarray(image) # back to PIL

        # apply transforms (mainly resize/crop/normalize)
        image = self.transforms(image)

        return image, target
</code></pre>
<p>Some additional detail:</p>
<ul>
<li>I’m using <code>scipy.ndimage.rotate</code> (instead of PIL or <code>torchvision.transforms.functional.rotate</code>) because I wanted <code>fill-mode='nearest'</code>.</li>
<li>I’ve already implemented the <code>Dataloader</code> optimization “tricks” (<code>num_workers</code>, <code>pin_memory</code>)</li>
<li>I have already resized images on disk to ~380 to reduce io.</li>
<li>The main reason I want to rotate on the fly is so every epoch a given image is rotated a different angle to mitigate overfitting.</li>
</ul>
<p>Thanks for taking the time to read through this! Any suggestion is welcome</p>

username: eqy
cooked: <p>Have you tried alternative rotation implementations (e.g., <a href="https://scikit-image.org/docs/dev/api/skimage.transform.html#skimage.transform.rotate" rel="noopener nofollow ugc">skimage’s rotate</a> or <a href="https://albumentations.ai/docs/api_reference/augmentations/transforms/#albumentations.augmentations.transforms.Rotate" rel="noopener nofollow ugc">albumentations’s rotate</a>)?<br>
Albumentations in particular claims to be very fast for rotation: <a href="https://pypi.org/project/albumentations/#benchmarking-results" rel="noopener nofollow ugc">benchmark</a>.</p>

username: eballest
cooked: <p>wow albumentation is way faster, thanks for the suggestion! Applying it not only to the rotation but also to the transforms I get around 6s per epoch WITH rotation enabled.</p>
Custom forward function of convolution layer

username: Shen
cooked: <p>Hello everyone,</p>
<p>I’m trying to implement a custom convolution layer. However, I only want to custom the forward function, and leave the backward function unchanged.</p>
<p>Following the guidance from <a href="https://pytorch.org/docs/stable/notes/extending.html" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/notes/extending.html</a>, I think the implementation would be something like</p>
<pre><code class="lang-auto">class MyConv2d(Function):

    def forward(ctx, input, weight):
        ctx.save_for_backward(input, weight)
        input_temp, weight_temp = f(input, weight)
        output = g(input_temp, weight_temp)
        return output
    
     def backward(ctx, grad_output):
        input, weight = ctx.saved_tensors
        '''
        Compute backward with original inputs, 
        i.e., compute gradients of output directly to input and weight, 
        and ignore the functions in the middle (i.e., f(·), g(·))
        '''
        return grad_input, grad_weight
</code></pre>
<p>where <code>f(·)</code> and <code>g(·)</code> are custom functions for computing convolutions. One example would be to flatten the input, turning convolutions into matrix multiplications. In this case, <code>f(·)</code> could be <code>unfold(·)</code>. As the backward path of <code>unfold(·)</code> is very slow, I don’t want this operation to be part of the computational graph. In other words, no matter what <code>f(·)</code> and <code>g(·)</code> are, I would like the <code>autograd</code> always treats this layer as <code>output = nn.conv2d(input, weight)</code>, making the computational graph only include <code>input</code>, <code>weight</code>, and <code>output</code> without any middle variables and functions.</p>
<p>The pseudocode above requires customizing the gradient calculation of convolution. I tried several approaches, but none of them can reach the same speed as the original <code>conv2d</code> function. I’m wondering:</p>
<blockquote>
<ol>
<li>How to implement the backward function of conv layer as fast as the original one in Pytorch?</li>
<li>Otherwise, are there any other simple ways to handle this problem?</li>
</ol>
</blockquote>
<p>Please help and thanks!!</p>

username: Shen
cooked: <p>I noticed one way to solve this problem is to compute <code>output=nn.functional.conv2d(input, weight)</code> first, and then perform any custom computations to get an <code>output_temp</code>. Finally, replace the values of <code>output</code> with <code>output_temp</code>.</p>
<p>However, the drawback is that I have to compute convolution operations twice, which slows down the training a lot. Are there any other strategies available? Thanks!</p>
W-GAN - Discriminator/Critic - Anomaly detection possible?

username: MaxM
cooked: <p>Hello guys,</p>
<p>I`m quite new to the topic of using GANs, in particular the discriminator of them, to classify images whether they are normal or abnormal regarding a before defined and trained on normal dataset. I started first with a normal Wasserstein-GAN-GP, coded by Aladdin Persson, that is provided on the below linked Git repository:</p>
<p><a href="https://github.com/aladdinpersson/Machine-Learning-Collection/tree/master/ML/Pytorch/GANs/4.%20WGAN-GP" class="inline-onebox" rel="noopener nofollow ugc">Machine-Learning-Collection/ML/Pytorch/GANs/4. WGAN-GP at master · aladdinpersson/Machine-Learning-Collection · GitHub</a></p>
<p>I am able to produce nice looking fake images based on the probability distribution of my custom data set. But I dont know how to understand the underlying loss function of the critic (discriminator) and how I can use this critic after training to classify some never seen before test data whether it is normal or abnormal (meaning if it fits to the trained data distribution). How do I get the critic to provide an Anomaly Score, where a particular threshold defines whether an input image is normal or abnormal.</p>
<p>Is this possible?</p>
<p>Many thanks in advance!</p>
Attention weighted aggregation

username: Aldebaran
cooked: <p>Let the tensor shown below be the representation of two sentences (batch_size = 2) composed with 3 words (max_lenght = 3) and each word being represented by vectors of dimension equal to 5 (hidden_size = 5) obtained as output from a neural network:</p>
<pre><code class="lang-python">net_output
# tensor([[[0.7718, 0.3856, 0.2545, 0.7502, 0.5844],
#          [0.4400, 0.3753, 0.4840, 0.2483, 0.4751],
#          [0.4927, 0.7380, 0.1502, 0.5222, 0.0093]],

#         [[0.5859, 0.0010, 0.2261, 0.6318, 0.5636],
#          [0.0996, 0.2178, 0.9003, 0.4708, 0.7501],
#          [0.4244, 0.7947, 0.5711, 0.0720, 0.1106]]])

</code></pre>
<p>Also consider the following attention scores:</p>
<pre><code class="lang-python">att_scores
# tensor([[0.2425, 0.5279, 0.2295],
#         [0.2461, 0.4789, 0.2751]])
</code></pre>
<p>Which efficient approach allows obtaining the aggregation of vectors in <code>net_output</code> weighted by <code>att_scores</code> resulting in a vector of shape (2, 5)?</p>
Torch/all.h: No such file or directory

username: offset-null1
cooked: <p>I have included <code>torch/extension.h</code>. But while building I’m getting the following error:</p>
<blockquote>
<p>/usr/local/include/torch/extension.h:4:10: fatal error: torch/all.h: No such file or directory<br>
<span class="hashtag-raw">#include</span> &lt;torch/all.h&gt;<br>
^~~~~~~~~~~~~<br>
compilation terminated.<br>
src/CMakeFiles/model_lib.dir/build.make:62: recipe for target ‘src/CMakeFiles/model_lib.dir/load.cpp.o’ failed<br>
make[2]: *** [src/CMakeFiles/model_lib.dir/load.cpp.o] Error 1<br>
CMakeFiles/Makefile2:1018: recipe for target ‘src/CMakeFiles/model_lib.dir/all’ failed<br>
make[1]: *** [src/CMakeFiles/model_lib.dir/all] Error 2<br>
Makefile:116: recipe for target ‘all’ failed<br>
make: *** [all] Error 2</p>
</blockquote>
<p>Can anyone please help me figure out what’s wrong?</p>

username: Ajinkya_Ambatwar
cooked: <p>The file is present in <code>torch/csrc/api/include/torch</code></p>
PyPy Vs TorchScript

username: Ahmed_m
cooked: <p>Hi there,</p>
<p>I would like to know whether TorchScript can be considered as a <a href="https://www.pypy.org/" rel="noopener nofollow ugc">PyPy</a> but for Pytorch so that we can use TorchScript instead of PyPy?!</p>
<p>Is it possible to simultaneously use PyPy to speedup non-pytorch code snippets while TorchScript is used for pytorch module?</p>
<p>Best,<br>
Ahmed</p>

username: googlebot
cooked: <p>I believe pytorch c++ module makes it impossible to switch to PyPy. There is some remote similarity between PyPy and TorchScript, yes.</p>
Torch.fx modify a copy of the Graph

username: nscotto
cooked: <p>Hi,</p>
<p>Let me explain the problem in its simplest form.<br>
I have a rule for block replacement (block1 → block2), and given a model I would like to generate all possible variations applying this rule.</p>
<p>So what I would like to do is I find all occurences of block1 in the model graph and keep <em>some sort of<br>
reference</em> to the nodes, then I iterate over the binary selection block1/block2 of combinations available and modify the graph accordingly (keep block1 occurence vs delete block1 occurence and insert block2).<br>
The problem is that the graph is modified in place, and I am using memory references to nodes, so after the first iteration I deleted an occurence of block1 I won’t be able to access the node again.<br>
It would be nice if I could modify a copy of the graph, but if I make a copy I loose track of the occurences in the new graph (because of the use of references).</p>
<p>Anyone has a better idea for this?</p>

username: Chillee
cooked: <p>Sorry, I don’t quite understand what your issue is. In particular, I don’t understand the problem here:</p>
<blockquote>
<p>so after the first iteration I deleted an occurence of block1 I won’t be able to access the node again.</p>
</blockquote>
<p>Could you elaborate a bit more?</p>

username: nscotto
cooked: <p>Sorry I mad it really complex, it is a reference problem.<br>
I am using object reference to reference the nodes composing the block I want to replace, however because modifications are done in place, those references won’t be valid if I wan’t to reuse them on a copy of the model to replace by another block.<br>
I think using node name should be fine for referencing, as long as each node have a unique name, do they?</p>
Evidence lower bound loss function and its gradient in pytorch

username: zezo
cooked: <p>How to implement evidence lower bound ELBO loss function and its gradient in pytorch. I have been using KL divergence as following:</p>
<pre><code># KL Divergence loss function
loss       = nn.KLDivLoss(size_average=False, log_target=True)                                    
out        = loss(Gaussian.log_prob(x), Gaussian.log_prob(xtrue))      
gr          = autograd.grad(outputs=[out], inputs=[x])[0]
</code></pre>
<p>How can I implement ELBO/gradient?</p>
Best way to compute the average of multiple models

username: omarfoq
cooked: <p>Hello,</p>
<p>I have a list of models (<code>nn.Module</code> instances) with same architecture and I need to compute the average model obtained from them, i.e., having the same architecture and where the weights are the average of input models weights.</p>
<p>An easy solution would be to iterate over the keys in <code>state_dict</code> of the target model and fill it using the weights from input models, corresponding to the same key (see code below). I am not sure that this is best way to do it because we need to iterate over the keys (or the modules) of the model which is probably not optimal, and was wondering if there is a faster way to do.</p>
<p>We can do the same, by iterating over the <code>modules</code> instead of the <code>state_dict</code>, but it doesn’t seem to help.</p>
<pre><code class="lang-auto">    for key in target_state_dict:
        if target_state_dict[key].data.dtype == torch.float32:
            target_state_dict[key].data.fill_(0.)
            for model_id, learner in enumerate(models):
                state_dict = model.state_dict()
                target_state_dict[key].data +=  state_dict[key].data.clone()
</code></pre>

username: ptrblck
cooked: <p>I think averaging the values in the <code>state_dicts</code> is a valid approach and I’ve also suggested it <a href="https://discuss.pytorch.org/t/average-each-weight-of-two-models/77008/2">here</a> in the past.<br>
You should of course check, if this approach is valid at all from the point of view of the trained models.<br>
E.g. if you are trying to calculate the “average model” using completely different training runs, each model could converge to different parameter sets, and I would assume that the average model would yield a bad performance.</p>

username: omarfoq
cooked: <p>Yes I remember this solution from <a href="https://discuss.pytorch.org/t/average-each-weight-of-two-models/77008/7">here</a>. I think it’s good in the case of two models, but I don’t think it’s the most efficient in the case of a sequence of many models. Because as you see, both in the code you suggested and in the one I pasted with this question, we are having a double loop, so this is very slow, because of the fact that loops in python are slow.</p>
<p>Normally, having some option that enables averaging models, can be very useful, for example for <a href="https://arxiv.org/abs/1803.05407" rel="noopener nofollow ugc">Stochastic Weighted Averaging</a> or other variations of this method, and I would be surprised if pytorch doesn’t support this feature.</p>

username: ptrblck
cooked: <p>PyTorch does support <a href="https://pytorch.org/blog/stochastic-weight-averaging-in-pytorch/">Stochastic Weight Averaging</a> through some util methods, so you might want to take a look at it.</p>