akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Batching most of current operations

Zhaoyi-Yan: <p>Sometimes, current operations tend to hardly meet our needs as they are not ‘vectorized enough’.<br>
For example, here I have to <code>tensor.eq(float)</code> to get each <code>cur_sp_mask</code>. So I have to use <code>for-loop</code> to enumerate each blob.</p>
<pre><code class="lang-python"> import torch

 bz = 2
 a = (torch.rand(bz, 3, 4)*4).int()
 index = torch.tensor([[0, 1, 0, 2],[0, 1, 2, 3], [1, 2, 2, 3]])

 a = a.view(bz, -1)
 index_vec = index.view(-1)

 for i in range(bz):
     max_index = torch.max(index_vec)
     a_vec = a[i]
     rec = torch.zeros_like(a_vec.float())
     # I have to enumerate each blob
     for j in range(max_index + 1):
         cur_sp_mask = index_vec.eq(j)
         n_pixels = torch.sum(cur_sp_mask).float()
         rec[cur_sp_mask] = a_vec[cur_sp_mask] + j * n_pixels
     rec = rec.view(3, 4)

     print('a[i]')
     print(a_vec.view(3,4))
     print('index')
     print(index.view(3,4))
     print('result')
     print(rec)
     print('------')
</code></pre>
<p>Another example, I want to create a list of tensors, each of which can be created using <code>torch.arange</code>:</p>
<pre><code class="lang-python"># Something like this:
#
import torch
x= torch.tensor([20, 53, 98]).long()
y= torch.tensor([110, 120, 132]).long()

# now I have to
a = []
for i in range(3):
    a.append(torch.arange(x[i], y[i]))

# However, I hope something like this can be
# more efficient than `for-loop`.
a = torch.arange(x, y)
# a is a list full of tensors
</code></pre>
<p>However, <code>torch.arange</code> does not support “batching” operation.</p>
<p>It is definetely huge work to make most operations powerful enough to support so advanced “batching”. However, maybe it is possible:</p>
<pre><code class="lang-python">import torch
x= torch.tensor([20, 53, 98]).long()
y= torch.tensor([110, 120, 132]).long()


warp_x = warp_op(x)
warp_y = warp_op(y)

# When tensors are warpped,
# then they can be more efficient executed than `for-loop`.
a = torch.arange(warp_x , warp_y)
# a is a list full of tensors
</code></pre>

ptrblck: <p>You suggestion seems to come close to what <code>vmap</code> could do and there are a few proofs of concept (<a href="https://github.com/pytorch/pytorch/pull/32558">#32558</a>, <a href="https://github.com/pytorch/pytorch/pull/32836">#32836</a>) as well as an open <a href="https://github.com/pytorch/pytorch/pull/38723">PR</a> with more information regarding the roadmap.</p>
Autoencoder in Pytorch to encode features/categories of data

ayn: <p>My question is regarding the use of autoencoders (in PyTorch). I have a tabular dataset with a categorical feature that has 10 different categories. Names of these categories are quite different - some names consist of one word, some of two or three words. But all in all I have 10 unique category names. What I’m trying to do is to create an autoencoder which will encode names of these categories - for example, if I have a category named  ‘Medium size class’ , I want to see if it is possible to train autoencoder to encode this name as something like  ‘mdmsc’ or something like that. The use of it would be to found out which data points are hard to encode or not typical or something like that. I tried to adapt autoencoder architectures from various tutorials online however nothing seems to work for me or I simply do not know how to use them as they are all about images. Maybe someone has any idea how this type of autoencoder might be accomplished if it is at all possible? Or maybe someone has a suggestion of tutorial to use?</p>
<p>Thank you in advance!</p>

ptrblck: <p>If I understand the use case correctly, you are dealing with names, which are corresponding to a category, i.e. they have a unique label.<br>
You would like to pass these names (as labels or in any representation) into your model and would like to get the abbreviation of this name as the latent tensor?</p>

ayn: <p>Yes, exactly. What I tried to do is to use LabelEncoder() and OneHotEncoder() to give these names numerical form to pass them into the simple autoencoder model however I’m not sure if it’s the right way as I can’t manage to get the output I want.</p>

ayn: <p>Here is what I have so far for my model however I am not sure if it suits the task I want to do. In the end, after training, I get the same result as an input and I am not sure what I should fix. Maybe you could help me with suggestions what should I do with my model or how should I preprocess my categorical data or something else? Thanks in advance!</p>
<pre><code class="lang-auto">class Autoencoder(nn.Module):

def __init__(self, input_shape, encoding_dim):
    super(Autoencoder, self).__init__()

    self.encode = nn.Sequential(
        nn.Linear(input_shape, 128),
        nn.ReLU(True),
        nn.Linear(128, 64),
        nn.ReLU(True),
        nn.Linear(64, encoding_dim),
    )

    self.decode = nn.Sequential(
        nn.Linear(encoding_dim, 64),
        nn.ReLU(True),
        nn.Linear(64, 128),
        nn.ReLU(True),
        nn.Linear(128, input_shape)
    )

def forward(self, x):
    x = self.encode(x)
    x = self.decode(x)
    return x

model = Autoencoder(input_shape=10, encoding_dim=5)
</code></pre>

ptrblck: <p>Unfortunately I don’t know, how you could approach this problem.<br>
The desired target sounds like a sequential task, i.e. you would pass in a sequence of letters and would expect your model to output another sequence (the abbreviation of the input), which would be similar to e.g. a translation use case.<br>
However, I’m sure how this could be combined with an autoencoder-like architecture.</p>
R2D2 in PyTorch

Muppet: <p>Hi, does anyone have an R2D2 example for PyTorch? Preferrably pong or some basic env? Would love to see a small efficient implementation</p>

Isaac_Kargar: <p>Did you check <a href="https://github.com/astooke/rlpyt" rel="nofollow noopener">rlpyt</a>? You can check <a href="https://github.com/jinbeizame007/pytorch-r2d2-DPG" rel="nofollow noopener">this</a> too. If you know another clean implementation, I’m interested to know too.<br>
Thanks.</p>
Confused about NN input

smu226: <p>Hello! This is probably something silly but I am confused about why the input to my NN doesn’t work. This is the network.</p>
<pre><code class="lang-auto">class SimpleNet(nn.Module):
    def __init__(self, ni):
        super().__init__()
        self.linear1 = nn.Linear(ni, 128)
        self.bn1 = nn.BatchNorm1d(128)
        self.linear2 = nn.Linear(128, 128)
        self.bn2 = nn.BatchNorm1d(128)
        self.linear3 = nn.Linear(128, 64)
        self.bn3 = nn.BatchNorm1d(64)
        self.linear4 = nn.Linear(64,64)
        self.bn4 = nn.BatchNorm1d(64)
        self.linear5 = nn.Linear(64,1)

    def forward(self, x):
        x = F.tanh(self.bn1(self.linear1(x)))
        x = F.tanh(self.bn2(self.linear2(x)))
        x = F.tanh(self.bn3(self.linear3(x)))
        x = F.tanh(self.bn4(self.linear4(x)))
        x = self.linear5(x)
        return x

n_variables = 2
model = SimpleNet(n_variables).cuda()
</code></pre>
<p>If I run</p>
<pre><code class="lang-auto">dt = torch.tensor([[1, 2],[3, 4]]).float().cuda()
model(dt)
</code></pre>
<p>it works fine. But for my case I need to pass one input at a time (after training) and when I do this:</p>
<pre><code class="lang-auto">dt = torch.tensor([[1, 2]]).float().cuda()
model(dt)
</code></pre>
<p>I am getting this error: <code>Expected more than 1 value per channel when training, got input size torch.Size([1, 128])</code></p>
<p>What am I doing wrong? Thank you!</p>

ptrblck: <p>The batchnorm layers cannot calculate the running estimates from a single sample, and raise this issue.<br>
Your input activation would have the shape <code>[1, 128]</code> without a temporal dimension, where 128 are the input channels.</p>
<p>You could either disable the batchnorm layers via <code>model.eval()</code> or increase the batch size.<br>
Also, using batchnorm layers in general for a small batch size might yield skewed running estimates.</p>
CUDNN_STATUS_INTERNAL_ERROR for large image sizes on RTX GPU

gfotedar: <p>I’m facing a CUDNN_STATUS_INTERNAL_ERROR on machines that have an RTX Quadro 8000 GPU.<br>
This happens only when I set cudnn.benchmark equal to True.<br>
I do not face the same error on other machines that have the same python/pytorch/cuda/cudnn versions installed.</p>
<p>I was able to reproduce the error using a simple code that is given below:</p>
<pre><code class="lang-auto">import time
import torch
from torch import nn, optim
import torch.utils.data as data_utils
import torchvision.models as models
from torch.backends import cudnn
from torch.nn import functional as F

cudnn.enabled = True
cudnn.benchmark = True
device = torch.device("cuda")
torch.cuda.set_device(2)
batch_size=1
img_size=1024
N=320
lr=0.001
channels=3

train_data = torch.randn(N, channels, img_size, img_size)
train_labels = torch.ones(N).long()

train = data_utils.TensorDataset(train_data, train_labels)
train_loader = data_utils.DataLoader(train, batch_size=batch_size, shuffle=True, pin_memory=True)

criterion = nn.CrossEntropyLoss().cuda()

model = models.densenet161().cuda()
#model = models.resnet18().cuda()

model.train()
optimizer = optim.Adam(model.parameters(), lr=lr)
for x,y in train_loader:
    x, y = x.to('cuda',non_blocking=True), y.to('cuda',non_blocking=True)
    pred = model(x)
    print('forward done')

    loss = criterion(pred, y)

    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
</code></pre>
<p>Some observations while running the above code on the machines that give the error:</p>
<ol>
<li>When I decrease img_size to 512 I do not face the same error</li>
<li>When I change model to resnet I do not face the same error</li>
<li>Changing batch size does not seem to have any effect</li>
</ol>
<p>I would appreciate any help or information to help debug the source of the problem.</p>
<p>Library versions:<br>
Python 3.7.5<br>
Pytorch1.3.1<br>
Cuda 10.1<br>
Cudnn 7.6.3<br>
Ubuntu 18.04</p>
<p>Also I understand from other posts that cudnn.benchmark looks for the best implementation for the particular hardware and image size and then uses that throughout the training. However, I could not find a good explanation of what cudnn.enabled does. I see that even if I set cudnn.benchmark as False I do get a boost in performance by just setting cudnn.enabled as True. How does it help?</p>

ptrblck: <p><code>torch.backends.cudnn.enabled = True</code> uses cudnn, while <code>... = False</code> disables it and falls back to the native PyTorch implementations.</p>
<p><code>cudnn.benchmark = True</code> could fail to find a fast kernel, but should fall back to a (slower but) working algorithm, which is not the case for your workload.<br>
Could you update PyTorch to the latest stable version (<code>1.5</code>) or the nightly binaries, as we’ve recently implemented another fallback mechanism, which should raise a proper error messge.</p>
New layer parameters initialization for pre-trained model

nicehzj: <p>I want to modify the pre-trained model and train it.<br>
For example, I use the resnet50 model in torchvision.<br>
I cancel the last fc layer<br>
Then I use a new transposed convolution layer and a new maxpool layer and a new classifier layer.</p>
<p>My question: are these three new layers’ parameters already initialized?<br>
Or I need to initialize them with method like “Xavier” ?</p>
<p>I checked the values of these three layers, and they are not zero. I don’t know whether they are already initialized.<br>
And why they are not equal to zero? Where do these values come from?</p>
<p>Thanks for your help!</p>

bing: <p>Initializing new layers with new method will override any value present, if it exists.</p>

nicehzj: <pre><code class="lang-auto">import torchvision.models as models
import torch.nn as nn

class Net(nn.Module):
    def __init__(self , model):
        super(Net, self).__init__()
        
        self.resnet_layer = nn.Sequential(*list(model.children())[:-2])
        
        self.transion_layer = nn.ConvTranspose2d(2048, 2048, kernel_size=14, stride=3)
        self.pool_layer = nn.MaxPool2d(32)  
        self.Linear_layer = nn.Linear(2048, 8)
        
    def forward(self, x):
        x = self.resnet_layer(x)
 
        x = self.transion_layer(x)
 
        x = self.pool_layer(x)
 
        x = x.view(x.size(0), -1) 
 
        x = self.Linear_layer(x)
        
        return x

res = models.resnet50(pretrained=True)

#model = Net(res)
#fc_features = res.fc.in_features

#res.fc = nn.Linear(fc_features, 10)

print(res.state_dict()['fc.weight'].shape)
</code></pre>
<p>I use this code and I don’t know which part do the initialization.<br>
The three new layers are not exist in the pre-trained model.</p>
<p>Thanks!</p>
How to transform a tensor to uniform distribution

tideng777: <p>I have a tensor in size [15, 28, 28]. I want to transform it to uniform distribution<br>
U(0,1) with same size. What tool can I use? thanks.</p>

vainaijr: <p>maybe,</p>
<pre><code class="lang-auto">import torch, seaborn as sns
x = torch.randn(15, 28, 28)
torch.nn.init.uniform_(x, a=0, b=1)
sns.distplot(x, kde=False)
</code></pre>
<p>we will get something like,<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/1/3/134ef2dc6970357cee2046a76d70e7b4cac9d328.png" alt="image" data-base62-sha1="2KOdyFpY2UT6K6XNgOWrvyqSRCM" width="375" height="248"></p>

tideng777: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/8/a8157a695dae476b9f320320256bce2a73522065.png" data-download-href="https://discuss.pytorch.org/uploads/default/a8157a695dae476b9f320320256bce2a73522065" title="Screenshot 2020-05-23 at 10.12.12 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/8/a8157a695dae476b9f320320256bce2a73522065_2_689x419.png" alt="Screenshot 2020-05-23 at 10.12.12 PM" data-base62-sha1="nYWen38QwvkkM7emQMIBDJWXzXD" width="689" height="419" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/8/a8157a695dae476b9f320320256bce2a73522065_2_689x419.png, https://discuss.pytorch.org/uploads/default/optimized/3X/a/8/a8157a695dae476b9f320320256bce2a73522065_2_1033x628.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/a/8/a8157a695dae476b9f320320256bce2a73522065.png 2x" data-dominant-color="F6F7F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2020-05-23 at 10.12.12 PM</span><span class="informations">1148×698 53.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>maybe I say not clearly. I have a distribution like this, and want to transform it in U(0,1) with same size.</p>
Calculate Autoencoder Accuracy based on recon. error

_joker: <p>I am using autoencoders to reconstruction images and then based on a certain threshold value I calculate the detection accuracy using;</p>
<pre><code class="lang-auto"># threshold = 95th percentile
# flag_list = {0, 1} Meaning -&gt; 0: Benign image, 1: adversarial image

def detection_accuracy(total_recon, flag_list, threshold):
  recon_np, label_np = np.asarray(total_recon), np.asarray(flag_list)
  recon_label_np = np.stack((recon_np, label_np), axis=1)
  recon_label_np = recon_label_np[recon_label_np[:,0].argsort()]
  FN, TN, TP, FP = 0, 0, 0, 0
  for i in range(recon_label_np.shape[0]):
    if recon_np[i] &lt; threshold:
      if label_np[i]!=flag:
        FP += 1
      else:
        TP += 1
    else:
      if label_np[i]!=flag:
        TN += 1
      else:
        FN += 1
  acc = 100.*(TP + TN) / (TP + TN + FP + FN)
  print('Accuracy: ', acc)
  return acc
</code></pre>
<p>I am doubtful about the way I do, as the accuracy doesn’t come out well as per the theoretical expectations. Can anyone tell a better way to find the accuracy based on reconstruction error? I tried passing the reconstruction through a resnet50 classifier and calculate the classification accuracy but that doesn’t come out well and that does not even support the objective.</p>
Will this code correctly calculate a dataset's mean and std?

ProGamerGov: <p>Does this code calculate the correct mean and std to use for <code>torchvision.transforms.Normalize()</code>?</p>
<pre><code class="lang-auto">import torch
import torchvision
import torchvision.transforms as transforms

# Calculate the mean and std of a dataset
def calc_data_mean(data_path):
    traindata = torchvision.datasets.ImageFolder(
        root=data_path,
        transform=transforms.Compose([transforms.ToTensor()])
    )
    image_means = torch.stack([torch.mean(t, dim=(1, 2)) for t, c in traindata])
    image_means = image_means.mean(0)
    print('RGB mean:', image_means)
	
    image_stds = torch.stack([torch.std(t, dim=(1, 2)) for t, c in traindata])
    image_stds = image_stds.std(0)
    print('RGB std:', image_stds)
    quit()
</code></pre>