akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
LibTorch for mobile devices

Jay_Fartiyal: <p>I want to compile my pytorch c++ Frontend code locally in my x86_64 Ubuntu but using a g++ arm compiler. Using the so file which I get after compilation to use in Android applications. Now, I am unable to use the compiled file as LibTorch file is compiled in x86_64. Is there a way to download LibTorch file for arm processor.</p>
<p>And the only way I can build that is building LibTorch source file from arm server. Is there any other way I can do that</p>

tom: <p>Libtorch cross-compiles well for Android, but not with g++ (Android stopped supporting this).</p>
<p>Best regards</p>
<p>Thomas</p>

Jay_Fartiyal: <p>Hi Tom,</p>
<p>But, this means that I have to put the libtorch.so file in my android which would be unnecessary. It takes huge space, {read from the discussion forum}. I want to build a complete so file locally and to place that in the android, rather than writing code in android and building from there.</p>
<p>I am pretty beginner in this, could you please help me to figure out the best way to do that ?</p>

tom: <p>I’d strongly recommend to</p>
<ul>
<li>try out the standard way first,</li>
<li>see what android gives you (the installed binary is <em>much</em> smaller than the one you install into the development environment),</li>
<li>follow the “dieting” tutorial (removing unneeded ops).</li>
</ul>
<p>Most likely after these, you already have something that is very reasonable.</p>
<p>Best regards</p>
<p>Thomas</p>
I want confusion matrix for this training process

Muhammad_Izaz: <p>for epoch in range(num_epochs):<br>
for i ,(images, labels) in enumerate(train_data):<br>
# Run the forward pass<br>
images = images.unsqueeze(1).float()<br>
outputs = model(images)<br>
loss = criterion(outputs, labels)<br>
loss_list.append(loss.item())</p>
<pre><code>    # Backprop and perform Adam optimisation
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
#optimizer update the weight parameters to minimize the loss function

    # Track the accuracy
    total = labels.size(0)
    _, predicted = torch.max(outputs.data, 1)
    correct = (predicted == labels).sum().item()
    acc_list.append(correct / total)

    if (i + 1) % 35 == 0:
        print('Epoch [{}/{}], Step [{}/{}], Loss: {:.4f}, Accuracy: {:.2f}%'
              .format(epoch + 1, num_epochs, i + 1, total_step, loss.item(),
                      (correct / total) * 100))</code></pre>

TGI_Fernando: <p>Please refer</p>
<p><a href="https://github.com/tgi25/pytorch/blob/master/Lesson10%20-%20LeNet.ipynb" rel="nofollow noopener">https://github.com/tgi25/pytorch/blob/master/Lesson10%20-%20LeNet.ipynb</a> - [4  Predictions]</p>
Torchvision install Error

Shahrullo_Lutfulloxo: <p>I had to create additional environment for ‘cv2’, after that I could easily install pytorch.<br>
Unfortunately, when it comes to install torchvision. I got this Error:</p>
<p>Collecting torchvision<br>
Downloading torchvision-0.5.0-cp37-cp37m-win_amd64.whl (1.2 MB)<br>
Requirement already satisfied: pillow&gt;=4.1.1 in c:\users\fachri\anaconda3\envs\opencv\lib\site-packages (from torchvision) (7.0.0)<br>
ERROR: Could not find a version that satisfies the requirement torch==1.4.0 (from torchvision) (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2)<br>
ERROR: No matching distribution found for torch==1.4.0 (from torchvision)</p>

ptrblck: <p>This issue is usually raised, if your conda/pip environment wasn’t updated for some time.<br>
Could you update <code>conda/pip</code> or create a new environment and retry the installation?</p>

Shahrullo_Lutfulloxo: <p>I tried both of them. Unfortunately couldn’t help</p>

peterjc123: <p>Please try out the install commands on <a href="https://pytorch.org" rel="nofollow noopener">https://pytorch.org</a>. You could not install torchvision without pytorch using <code>pip install torchvision</code> because the Windows wheels are not hosted on PyPI.</p>

Shahrullo_Lutfulloxo: <p>I installed everything on my basic anaconda environment, both pytorch and torchvision is working well. cv2 didn’t work there, that’s why I created new environment only for cv and pytorch. In new environment cv2 and pytorch is working, unfortunately when I tried to install torchvision, it is getting ERROR as mentioned above.</p>
Custom kernel like matmul can not be dispatched

harryjing: <p>Some ops are implemented as a composition of other ATen functions like matmul, it consists of expand , view, and bmm in the at::native::matmul. If I implement the kernel of matmul and register it in the following way:</p>
<blockquote>
<p>torch::RegisterOperators().op(…“aten::matmul…”).impl_unboxedOnlyKernel&lt;…&gt;…</p>
</blockquote>
<p>I wonder how to skip the stitching under at::native::matmul  to reach my custom kernel directly without modification of the at::native::matmul source code ?</p>
Can validation code block affect training?

Aptha_Gowda: <p>I tried to seed everything so that I can reproduce the same result in PyTorch by using this block of code</p>
<pre><code class="lang-auto">def seed_everything(SEED):
    np.random.seed(SEED)
    torch.manual_seed(SEED)
    torch.cuda.manual_seed(SEED)
    torch.backends.cudnn.deterministic = True
    torch.backends.cudnn.benchmark = False
</code></pre>
<p>and It was successful. But the problem arose when I removed the validation part(I just removed the validation code block, not the validation data) in my train function. I thought the result of each training epoch should match with my previous kernel where I used both the training and validation part in my train function. But it was not and the results were different.</p>
<p>My question is how come</p>
<pre><code class="lang-auto">model.eval()
with torch.no_grad():
</code></pre>
<p>is affecting my training as these lines turn off everything?</p>
<p>I didn’t use any metric which decides learning rate depending upon validation loss/accuracy.</p>

KFrank: <p>Hello Aptha!</p>
<aside class="quote no-group" data-username="Aptha_Gowda" data-post="1" data-topic="77604" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/aptha_gowda/48/23015_2.png" class="avatar"> Aptha_Gowda:</div>
<blockquote>
<p>My question is how come</p>
<pre><code class="lang-auto">model.eval()
with torch.no_grad():
</code></pre>
<p>is affecting my training as these lines turn off everything?</p>
</blockquote>
</aside>
<p>Well, any number of things that could affect your training could be hiding<br>
in the code you allude to.</p>
<p>One possibility is that if both your training and validation code consume<br>
pseudo-random numbers, turning off your training could cause different<br>
pseudo-random numbers to be fed to your training, affecting the training<br>
results.</p>
<p>For example, if both your training and validation involve using a random<br>
<code>DataLoader</code>, you could see this kind of issue.</p>
<p>Best.</p>
<p>K. Frank</p>

Aptha_Gowda: <p>So basically by removing the validation code block, the random number generated is different compared with the previous version where there is validation code.<br>
I think it makes sense as I noticed the difference in the 2nd epoch and not the 1st epoch.</p>
<p>Thanks, K. Farnk</p>
Dataloader giving incorrect result

aditya_k: <p>I am trying to return variable length array from Dataset, Dataloader is returning incorrect results.</p>
<pre><code class="lang-auto">class TS(Dataset):
    
    def __len__(self):
        return 10
    
    def __getitem__(self,idx):
        return {'token':[1]*idx,'in':idx,'text':"hello"}

ds = TS()
dl = DataLoader(ds,batch_size=3,shuffle=False)

for dic in dl:
    print(dic)

</code></pre>
<p>Output</p>
<blockquote>
<p>{‘token’: <span class="chcklst-box fa fa-square-o fa-fw"></span>, ‘in’: tensor([0, 1, 2]), ‘text’: [‘hello’, ‘hello’, ‘hello’]}<br>
{‘token’: [tensor([1, 1, 1]), tensor([1, 1, 1]), tensor([1, 1, 1])], ‘in’: tensor([3, 4, 5]), ‘text’: [‘hello’, ‘hello’, ‘hello’]}<br>
{‘token’: [tensor([1, 1, 1]), tensor([1, 1, 1]), tensor([1, 1, 1]), tensor([1, 1, 1]), tensor([1, 1, 1]), tensor([1, 1, 1])], ‘in’: tensor([6, 7, 8]), ‘text’: [‘hello’, ‘hello’, ‘hello’]}<br>
{‘token’: [tensor([1]), tensor([1]), tensor([1]), tensor([1]), tensor([1]), tensor([1]), tensor([1]), tensor([1]), tensor([1])], ‘in’: tensor([9]), ‘text’: [‘hello’]}</p>
</blockquote>
<p>If i change the Dataset to</p>
<pre><code class="lang-auto">class TS(Dataset):
    
    def __len__(self):
        return 10
    
    def __getitem__(self,idx):
        return [1]*idx, idx
</code></pre>
<p>Output:</p>
<blockquote>
<p>[<span class="chcklst-box fa fa-square-o fa-fw"></span>, tensor([0, 1, 2])]<br>
[[tensor([1, 1, 1]), tensor([1, 1, 1]), tensor([1, 1, 1])], tensor([3, 4, 5])]<br>
[[tensor([1, 1, 1]), tensor([1, 1, 1]), tensor([1, 1, 1]), tensor([1, 1, 1]), tensor([1, 1, 1]), tensor([1, 1, 1])], tensor([6, 7, 8])]<br>
[[tensor([1]), tensor([1]), tensor([1]), tensor([1]), tensor([1]), tensor([1]), tensor([1]), tensor([1]), tensor([1])], tensor([9])]</p>
</blockquote>

ptrblck: <p>You would need a custom collate function to use variable input shapes:</p>
<pre><code class="lang-python">def my_collate(batch):
    data = [item for item in batch]
    return data

class TS(Dataset):
    
    def __len__(self):
        return 10
    
    def __getitem__(self,idx):
        return {'token':[1]*idx,'in':idx,'text':"hello"}

ds = TS()
dl = DataLoader(ds,batch_size=3,shuffle=False, collate_fn=my_collate)

for dic in dl:
    print(dic)
</code></pre>
What is changing python to c ++?

sangwon: <h2>python code</h2>
<p>import torch<br>
import torch.nn.functional as F<br>
from torchvision import transforms<br>
from torchvision.models import resnet18<br>
from torch import nn<br>
from PIL import Image</p>
<p>make_model_path = “d:\model.pt”</p>
<p>image_path = “D:\1.png”<br>
image = Image.open(image_path).convert(‘RGB’)</p>
<p>transformer = transforms.Compose([<br>
transforms.Resize(256),<br>
transforms.CenterCrop(224),<br>
transforms.ToTensor(),<br>
transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])<br>
])</p>
<p>image = transformer(image)<br>
image =image.unsqueeze(0)</p>
<p>device1 = torch.device(‘cpu’)<br>
modelll  = torch.jit.load(make_model_path);<br>
outputs = modelll(image.to(device1))<br>
_, predicted = torch.max(outputs.data, 1)</p>
<p>print(“end”)</p>
<h3>c++ code</h3>
<pre><code>torch::jit::script::Module module;

std::ifstream is("d:\\model.pt", std::ifstream::binary);
module = torch::jit::load(is);


cv::Mat image;
image = cv::imread("D:\\1.png", 1);

cv::cvtColor(image, image, 4); 
cv::Mat img_float;
image.convertTo(img_float, CV_32F, 1.0 / 255);
cv::resize(img_float, img_float, cv::Size(224, 224));

auto img_tensor = torch::from_blob(img_float.data, { 1, 224, 224, 3 });// .to(torch::kCUDA);
img_tensor = img_tensor.permute({ 0, 3, 1, 2 });
img_tensor[0][0] = img_tensor[0][0].sub(0.485).div(0.229);
img_tensor[0][1] = img_tensor[0][1].sub(0.456).div(0.224);
img_tensor[0][2] = img_tensor[0][2].sub(0.406).div(0.225);
auto img_var = torch::autograd::make_variable(img_tensor, false);

std::vector&lt;torch::jit::IValue&gt; inputs;
inputs.push_back(img_var);


torch::Tensor output = module.forward(inputs).toTensor().cpu();
auto accessor = output.accessor&lt;float, 2&gt;();
</code></pre>
<p>please…</p>

ptrblck: <p>OpenCV loads images in BGR format by default, while <code>PIL</code> uses RGB.<br>
I cannot find a transformation between these color formats in your C++ code, so besides the wrong normalization of the tensor you are passing it in a flipped format to the model.</p>
Nv-nsight-cu-cli Does not work in NGC Pytorch container 20.03

Navdeep_Kumar: <p>When I try to use nsight compute in the NGC container 20.03 on pytorch scripts, I get the following error.<br>
free(): invalid pointer.<br>
The profiler works on simple *.cu files but not on  pytorch sripts.</p>

ptrblck: <p>It’s a known <a href="https://forums.developer.nvidia.com/t/nv-nsight-cu-cli-pytorch-throws-free-invalid-pointer-error/110299/4">issue</a>.<br>
We’ve isolated, fixed it, and will release the fixed version in the next container release.</p>

Navdeep_Kumar: <p>OK That’s great thanks. BTW may I know what was causing it, Because I tried to install pytorch from source in the container and it also didn’t work. However I installed a nightly pytroch binary and it worked.<br>
For my purposes I actually need to make some changes in pytorch source and use that version. But this issue prevents me to do that. Is there any way?</p>
Detection2 Mask head resolution question

jackck: <p>I am studying Detection2 implementation of maskrcnn. The groundtruth mask prediction is a 28x28 binary tensor created by:</p>
<p>File masks.py:</p>
<pre><code class="lang-auto">def polygons_to_bitmask(polygons: List[np.ndarray], height: int, width: int) -&gt; np.ndarray:
    """
    Args:
        polygons (list[ndarray]): each array has shape (Nx2,)
        height, width (int)

    Returns:
        ndarray: a bool mask of shape (height, width)
    """
    assert len(polygons) &gt; 0, "COCOAPI does not support empty polygons"
    rles = mask_utils.frPyObjects(polygons, height, width)
    rle = mask_utils.merge(rles)
    return mask_utils.decode(rle).astype(np.bool)
</code></pre>
<p>The default mask heigh and width is 28.  Now, a 28x28 binarized mask is very low resolution, yet the model is able to predict much higher fidelity masks.  I am struggling to understand how a severely down-samplied ground truth mask used for training can produce a higher resolution masks.  Can someone please enlighten me?</p>

ptrblck: <p>I’m not completely sure, but I would assume that the output might be interpolated to the input shape as seen <a href="https://github.com/pytorch/vision/blob/a6a926bc8bdb8a32d910943c9c78e231879ff989/torchvision/models/segmentation/_utils.py#L25">here</a>.</p>
How to make an imbalance mnist dataset with index?

TanmDL: <p>How to make an imbalance mnist dataset with three outputs like data, target, and index?</p>

ptrblck: <p>I’ve created <a href="https://github.com/pytorch/tutorials/pull/236">this tutorial</a> a while ago, which uses the CIFAR10 dataset to create an artificially imbalanced dataset, so it might be a good starter for MNIST.</p>
Iterating through trainloader gives error after random no of iterations, everytime the file is run

ANKITA_DAS: <pre><code class="lang-auto">transform=transforms.Compose([transforms.RandomCrop(224),transforms.ToTensor(),transforms.Normalize((0.5,0.5,0.5),(0.5,0.5,0.5))])
batch_size=20 
train_data=datasets.ImageFolder(root="/data/dog_images/train",transform=transform)
test_data=datasets.ImageFolder(root="/data/dog_images/test",transform=transform)
valid_data=datasets.ImageFolder(root="/data/dog_images/valid",transform=transform)

train_list=np.arange(0,len(train_data))
print(train_list)
test_list=np.arange(0,len(test_data))
valid_list=np.arange(0,len(valid_data))
train_sampler = SubsetRandomSampler(train_list)
valid_sampler = SubsetRandomSampler(valid_list)
test_sampler=SubsetRandomSampler(test_list)


train_loader=torch.utils.data.DataLoader(dataset=train_data,batch_size=batch_size,sampler=train_sampler)
valid_loader=torch.utils.data.DataLoader(dataset=valid_data,batch_size=batch_size,sampler=valid_sampler)
test_loader=torch.utils.data.DataLoader(dataset=test_data,batch_size=batch_size,sampler=test_sampler)
print(len(train_loader))  #334

for data,target in train_loader:
    print("data:",len(data))
    print("Target:",len(target))
</code></pre>
<p>Output:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/d/4dad56cb6549247aa743454c730338c44ec339e0.png" data-download-href="https://discuss.pytorch.org/uploads/default/4dad56cb6549247aa743454c730338c44ec339e0" title="Screenshot from 2020-04-21 06-34-05"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/d/4dad56cb6549247aa743454c730338c44ec339e0_2_690x346.png" alt="Screenshot from 2020-04-21 06-34-05" data-base62-sha1="b5a8PpE6dHSGC7oNYuwSSfre1ZS" width="690" height="346" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/d/4dad56cb6549247aa743454c730338c44ec339e0_2_690x346.png, https://discuss.pytorch.org/uploads/default/original/3X/4/d/4dad56cb6549247aa743454c730338c44ec339e0.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/4/d/4dad56cb6549247aa743454c730338c44ec339e0.png 2x" data-dominant-color="F5F7F6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2020-04-21 06-34-05</span><span class="informations">774×389 41.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><strong>Error message: ValueError: empty range for randrange() (0,-3, -3)</strong></p>
<p>Note:This time the control iterated over the trainloader <em>twice</em> and then showed an error,but the number of iterations after which the error is produced is varying everytime.</p>
<p>Where am I going wrong?</p>

ptrblck: <p>Are you mounting the <code>/data</code> folder?<br>
If so, could you create a local copy and rerun your code?<br>
If the local copy works, I would assume your server / network / mount might have some connectivity issues.</p>
RuntimeError: The size of tensor a (4) must match the size of tensor b (14) at non-singleton dimension 1

aayush: <p>I am getting the following error:<br>
Background:<br>
Each image is associated with 14 labels, with a batch size of 4.<br>
So the size of labels is (4,14)<br>
But when i get the predictions each image will have a single class prediction.<br>
So the size of preds is(4,)<br>
So how shall I rectify the following error.<br>
running_corrects += torch.sum(preds == labels.data) RuntimeError: The size of tensor a (4) must match the size of tensor b (14) at non-singleton dimension 1</p>

ptrblck: <p>I assume that <code>preds</code> are the class predictions created via e.g. <code>torch.argmax(outputs, 1)</code>.<br>
Also, it seems that your target is one-hot encoded.<br>
If that’s the case, you should get the class indices via:</p>
<pre><code class="lang-python">labels = torch.argmax(labels, 1)
</code></pre>
<p>Note, that the usage of <code>.data</code> is discouraged, as it might have unwanted side effects.</p>
Flatten cannot be found in nn Module

John_Deterious: <p>My Pycharm linter complains about Flatten not being found in nn (it is actually there and the code runs smoothly but the linter cannot find it). I know this is trivial, but how can I solve it?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/8/88062c9ede9edf2df380f296cb851b2242deb958.png" data-download-href="https://discuss.pytorch.org/uploads/default/88062c9ede9edf2df380f296cb851b2242deb958" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/8/8/88062c9ede9edf2df380f296cb851b2242deb958.png" alt="image" data-base62-sha1="jpkabMDBoLlPCa2Oef4GEXEF4CY" width="690" height="129" data-dominant-color="171824"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">987×185 11.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Get weights as list from a CNN

gss: <p>I am working on this CNN:</p>
<pre><code class="lang-auto">  class CustomConvNet(nn.Module):
    def __init__(self, num_classes):
        super(CustomConvNet, self).__init__()

        self.layer1 = self.conv_module(3, 64)
        self.layer2 = self.conv_module(64, 128)
        self.layer3 = self.conv_module(128, 256)
        self.layer4 = self.conv_module(256, 256)
        self.layer5 = self.conv_module(256, 512)
        self.gap = self.global_avg_pool(512, num_classes)
        #self.linear = nn.Linear(512, num_classes)
        #self.relu = nn.ReLU()
        #self.softmax = nn.Softmax()

    def forward(self, x):
        out = self.layer1(x)
        out = self.layer2(out)
        out = self.layer3(out)
        out = self.layer4(out)
        out = self.layer5(out)
        out = self.gap(out)
        out = out.view(-1, 4)
        #out = self.linear(out)

        return out

    def conv_module(self, in_num, out_num):
        return nn.Sequential(
            nn.Conv2d(in_num, out_num, kernel_size=3, stride=1, padding=1),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=(2, 2), stride=None))

    def global_avg_pool(self, in_num, out_num):
        return nn.Sequential(
            nn.Conv2d(in_num, out_num, kernel_size=3, stride=1, padding=1),
            #nn.BatchNorm2d(out_num),
            #nn.LeakyReLU(),

            nn.ReLU(),
            nn.Softmax(),
            nn.AdaptiveAvgPool2d((1, 1)))
</code></pre>
<p>Is there a way of getting the weights of the first layer as a list? I know one can save the model as a pth file, but I need a list with the weights. Thanks!</p>

jean-marc: <p>In your case your first layer has 1 Conv2d, 1 ReLU, and 1 MaxPool2d<br>
You can get the weights of your Conv2d of layer1 by doing</p>
<pre><code class="lang-auto"># Just a sample CustomConvNet
net = CustomConvNet(num_classes=10)
# Print the weights of Conv2d of layer1
print(net.layer1[0].weight.data)
# Print the bias of Conv2d of layer1
print(net.layer1[0].bias.data)
</code></pre>
<p><strong>[0]</strong> because <strong>nn.Conv2d</strong> is the first in your <strong>nn.Sequential</strong><br>
So if you <code>print(net.layer1)</code> you will see</p>
<pre><code class="lang-auto">Sequential(
  (0): Conv2d(3, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (1): ReLU()
  (2): MaxPool2d(kernel_size=(2, 2), stride=(2, 2), padding=0, dilation=1, ceil_mode=False)
)
</code></pre>
Negative AP in TorchVision 0.3 Object Detection finetuning tutorial

azer: <p>I have been using this tutorial with a few datasets. I have found that sometimes while the AP for the bb increases the AP for the segmentation drops more than double. I have also noticed that running the Penn-Fudan dataset the precision for the small areas is negative. Is this supposed to be ok to have negative AP values❓</p>
<p>Also for the learning scheduler, is it ok for it to be placed before the evaluation on the test set?</p>
Forward Block in Inception Network for creating BasicConv2d.?

sgaur: <p>HI</p>
<p>I just want to know while creating the Inception Network  “<strong>BasicConv2d</strong>” why author implemented with forward block inside class <strong>BasicConv2d</strong> why cant’t we use just a Function as defined below named <strong>def conv3x3</strong>??</p>
<p>Is there any special need to define class <strong>BasicConv2d(nn.Module)</strong> instead of <strong>def conv3x3</strong>.</p>
<blockquote>
<p>class BasicConv2d(nn.Module):</p>
<pre><code>def __init__(self, in_channels, out_channels, **kwargs):
    super(BasicConv2d, self).__init__()
    self.conv = nn.Conv2d(in_channels, out_channels, bias=False, **kwargs)
    self.bn = nn.BatchNorm2d(out_channels, eps=0.001)

def forward(self, x):
    x = self.conv(x)
    x = self.bn(x)
    return F.relu(x, inplace=True) 
</code></pre>
</blockquote>
<blockquote>
<p>def conv3x3(in_planes, out_planes,kernel_size,**kwargs):<br>
“”“3x3 convolution with padding”“”<br>
return nn.Conv2d(in_planes, out_planes, kernel_size, **kwargs)</p>
</blockquote>

ptrblck: <p>The <code>BasicConv2d</code> module defines a batchnorm layer besides the conv layer and also applied a relu to the output., while <code>conv3x3</code> just refines a conv layer.<br>
Both approaches can be used in one way or the other, but they define different abstraction levels as explained before.</p>

sgaur: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a>  for replying.</p>
<p>If i use below function instead of class it will also works?</p>
<blockquote>
<p>def conv2d():<br>
conv2d = nn.Sequential(nn.Conv2d(in_channels,out_channels,**kwargs)<br>
,nn.BatchNorm2d(out_channels,eps = .001)<br>
,nn.ReLU()<br>
,nn.Dropout(0.15))<br>
return conv2d</p>
</blockquote>

ptrblck: <p>Yes, this should return the <code>nn.Sequential</code> container with the specified modules.</p>
Help with model parameters

Muhsinka: <p>data_transforms = {<br>
‘train’: transforms.Compose([<br>
transforms.RandomResizedCrop(224),<br>
transforms.RandomHorizontalFlip(),<br>
transforms.ToTensor(),<br>
transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])<br>
]),<br>
‘val’: transforms.Compose([<br>
transforms.Resize(224),<br>
transforms.CenterCrop(256),<br>
transforms.ToTensor(),<br>
transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])<br>
]),<br>
}</p>
<p>class Net(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(Net, self).<strong>init</strong>()<br>
self.conv1 = nn.Conv2d(3, 64, kernel_size=6, padding=2)<br>
self.pool = nn.MaxPool2d(2, 2)<br>
self.conv2 = nn.Conv2d(64, 128, kernel_size=6, padding=2)<br>
self.fc1 = nn.Linear(256 * 6 * 6, 4096)<br>
self.fc2 = nn.Linear(4096, 4096)<br>
self.fc3 = nn.Linear(4096, 2)</p>
<pre><code>def forward(self, x):
    x = self.pool(F.relu(self.conv1(x)))
    x = self.pool(F.relu(self.conv2(x)))
    x = x.reshape(x.size(0), -1)
    x = F.relu(self.fc1(x)) 
    x = F.relu(self.fc2(x))
    x = self.fc3(x)
    return x
</code></pre>
<p>RuntimeError: size mismatch, m1: [4 x 387200], m2: [9216 x 4096]</p>
<p>how can I set parameters for this model?</p>

ptrblck: <p>Since the number of features of the input activation to <code>self.fc1</code> is <code>387200</code>, you should set <code>in_features</code> of <code>self.fc1</code> also to this value.<br>
Your current model expects <code>256*6*6</code> input features, which doesn’t match the activation shape for your current input.</p>

Muhsinka: <p>where is that 387200 is coming from</p>

ptrblck: <p>It’s coming from the number of features of the activation you are trying to pass to <code>self.fc1</code>.<br>
I.e. <code>x</code> would have the shape here:</p>
<pre><code class="lang-python">x = x.reshape(x.size(0), -1)
print(x.shape) # should be [batch_size, 387200]
x = F.relu(self.fc1(x)) 
</code></pre>

Muhsinka: <p>So how would you write this model?</p>

ptrblck: <p>Either change <code>in_features=387200</code> in <code>self.fc1</code> as posted before.<br>
Alternatively you could add more conv/pooling operation to reduce the spatial size of the activation or pass a smaller input tensor to the model.<br>
The current shape is defined by your input shape and the conv/pool layers.</p>
RuntimeError: CUDA error: device-side assert triggered error

SandPhoenixX: <p>hello,<br>
i trained my model and everything went cool but when testing it this error shows up<br>
RuntimeError: CUDA error: device-side assert triggered</p>

albanD: <p>Hi,</p>
<p>You can run your code with <code>CUDA_LAUNCH_BLOCKING=1 python your_script.py</code> to make it return a better error message.<br>
Also it should have printed in the terminal more info about the assert no?</p>

SandPhoenixX: <p>am using jupyter notebook</p>

albanD: <p>You should make sure that you set the environment variable just after reseting the kernel and before running any cuda-related code (before importing torch if possible).</p>

SandPhoenixX: <p>I did, Cuda works fine in the training phase and just after that when the testing phase starts the problem pops up</p>

albanD: <p>Can you give the full log please?<br>
And if you can get a small code repro (around 30 lines) that would be very helpful.</p>

SandPhoenixX: <p>----&gt; 2 images=images.to(device)<br>
3 labels=labels.to(device)<br>
RuntimeError:CUDA error: device-side assert triggered error</p>
<p>that is exactly the error I am getting<br>
and this is the code causing it:</p>
<blockquote>
<p>total_loss = 0<br>
total_correct=0<br>
for batch in testloader:<br>
images, labels = batch<br>
images=images.to(device)<br>
labels=labels.to(device)<br>
preds = net(images)<br>
loss = loss_function(preds,labels)<br>
total_loss += loss.item()<br>
total_correct += get_correct(preds, labels)</p>
</blockquote>

albanD: <p>I would be very surprised if that line threw that. This is most likely another that is responsible and the async API is pointing at the wrong line.<br>
Are you sure that you set <code>CUDA_LAUNCH_BLOCKING</code> properly?</p>

SandPhoenixX: <p>Sadly I am sure xD<br>
I wasn’t able figure out a solution because I couldn’t know how that line causes such an error especially that I am usining the same line above in the code and it went fine</p>

albanD: <p>One thing to know is that once a GPU sends an assert, it goes in a bad state and the whole process needs to be restarted, in your case, you need to restart the kernel from scratch. Otherwise, you will see random errors every time you try to use the GPU (this is a limitation on CUDA side).</p>
<p>You should double check also is that every place where you using indices (targets for loss, or when indexing), check that these indices are in the proper range.</p>
<p>If both of these are done, you will have to remove code to try and pinpoint the reason for the failure I’m afraid.</p>
What is memory_format in torch.zeros_like?

shinx: <p>In the implementation of Adam theres a line like</p>
<pre><code class="lang-auto">state['exp_avg'] = torch.zeros_like(p, memory_format=torch.preserve_format)
</code></pre>
<p>Where as the documentation has the following options</p>
<pre><code class="lang-auto">torch.zeros_like(input, dtype=None, layout=None, device=None, requires_grad=False)
</code></pre>
<p>What is this <code>memory_format</code> argument ? And what does <code>torch.preserve_format</code> do ?</p>

albanD: <p>Hi,</p>
<p>You can find it if you look at the latest doc here: <a href="https://pytorch.org/docs/master/torch.html#torch.zeros_like">https://pytorch.org/docs/master/torch.html#torch.zeros_like</a>  (note the master in the top left of the website)</p>
Transfer Learning Imagenet

Ashish_Gupta1: <p>I am running a experiment on pretrained models. I am wondering if I can have pretrained models trained on some other data other then imageNet. Can I use pretrained models on some other data?<br>
Meanwhile please suggest me if I can download a small part of ImageNet(I need only few examples but 1000 classes(like the same way the pytorch models are trained))</p>
Reading Videos through PyTorch DataLoader

shakeel608: <p>I am facing issue in Pytorch DataLoader.<br>
I created a custom video data-loader in pytorch but the issue arises that every time I am trying to read a video file, the code doesn’t show any error but it doesn’t read any data regarding the viedo read like its frames, NumFrames etc. (No problem in reading video file names)</p>
<p>But Whenever I try to read the video files with same code snippet (without Pytorch DataLoader), it runs smoothly as per the need</p>
<p>Could you please help here?</p>
<pre><code class="lang-auto">in __getitem__() function ::

 vid_file    = self.csv_vid_metadata["FileName"][item]            
            vis_data    = cv2.VideoCapture(vid_file)
            print(vis_data.get(cv2.CAP_PROP_FRAME_COUNT))
            numFrames   = int(vis_data.get(cv2.CAP_PROP_FRAME_COUNT))
            frameWidth  = int(vis_data.get(cv2.CAP_PROP_FRAME_WIDTH))
            frameHeight = int(vis_data.get(cv2.CAP_PROP_FRAME_HEIGHT))
            frame_buffer= np.zeros((numFrames, frameHeight, frameWidth, 3), np.dtype('uint8'))

            print("vid_file, numFrames, frameWidth, frameHeight:::::",vid_file, numFrames, frameWidth, frameHeight)
</code></pre>
Adding a layer to an RNN

nicolasJ: <p>Hi there,</p>
<p>So I’ve been trying to make a certain kind of Neural Network to work. My layout is the following: I want my input first to go into a hidden layer of <em>x</em> neurons then it goes through another layer where it gives the output (so a two layer network). Now the “tricky” (for a newly introduced to PyTorch as myself) part is that I want to give the output at time <em>t</em> as a new feature in the input, so that my input constitutes of x(t),y(t-1).  My instinct was that it is some kind of Recurrent Neural Net but as I tried to implement that I have no possibility of adding an <em>x</em>-sized layer into the RNN. Am I wrong ? As I see it the RNN has only one layer of which you cannot determine the hidden layer size. How should I proceed ?</p>
<p>Thank you very much if you have the time to help <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>

googlebot: <p>To use multiple inputs, you can concatenate them on last dimension (torch.cat).</p>
<p>Note that you may not need x/y role split in the middle of the network, unless you have residual architecture, multiple outputs/losses or something like that - your second hidden layer can produce hidden x representation that is fed to rnn, and Y can be created later.</p>
Loss not decreasingn LSTM classification

Fethallah_Walid: <p>hi guys, I’ve finally been able to shape my data an start training LSTM network but the Loss doesn’t seem to drop.</p>
<p>My data is 1000 records of ECG signal each belong to one of 17 classes, each of length [3600], I’ve reshaped it to [180,20]. the Adam optimizer keeps giving me a loss that sometimes increase instead of decreasing, I tried SGD optimizer and the best result I had so far with (learning rate=0.01, momentum=0.5, weight decay=0.1,neterov=True) are shown in the pic, the network has 7 layers with 0.1 dropout and hidden size of 200.</p>
<p>If anyone has any Idea about what’s going on please let me know.<br>
P.S: I’s a newbie and this is my first project so please bare with me.<img src="https://discuss.pytorch.org/uploads/default/original/3X/1/e/1e713a7f46c6e8f544cf1ead1f0ff3df9c114789.png" alt="2" data-base62-sha1="4liUJvME5s9bRxnPXYVYba1uta9" width="640" height="480"></p>
How does pytorch computes on GPU?

Faruk_Nane: <p>Hi, I have been writing <a href="https://github.com/faruknane/DeepLearningFramework" rel="nofollow noopener">a deep learning library</a>. I think a lot about how the way of gpu and cpu computing should be and wonder how pytorch computes actually and achieves building the fastest library. Before continuing on telling, I want to say that I tried to find the answer by looking at the codes of CNTK. However, It’s so complicated, I couldn’t understand at all. So after CNTK, I gave up searching for pytorch codes.</p>
<p>My library has Layers which all have sequence length as outer dimension, like below. Each box of a layer is called a Term.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://user-images.githubusercontent.com/37745467/79687009-0408ce00-824d-11ea-96bb-3c17dbfa3f3b.png" title="image" rel="nofollow noopener"><img src="https://user-images.githubusercontent.com/37745467/79687009-0408ce00-824d-11ea-96bb-3c17dbfa3f3b.png" alt="image" width="690" height="342"><div class="meta">
<svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use xlink:href="#far-image"></use></svg><span class="filename">image</span><span class="informations">910×452</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use xlink:href="#discourse-expand"></use></svg>
</div></a></div></p>
<p>One way is, I can continue with one main thread on the cpu and parallelize computing of a term in gpu. This way, I only compute one Term at a time. However, that term may not fill the entire gpu load. So gpu utilization could be so low.</p>
<p>The second approach I thought is, Whenever I am to compute a Term, first I compute the necessary Terms that my Term will need, simultaneously. Then after computing the necessary terms, I can use them and compute my term. This way, I can push more than one Term to Gpu to compute. But there might a problem here as well which I don’t know for sure.</p>
<p>I wonder how pytorch computes those. Can someone just give a little idea?</p>
<p>Thank you</p>
Training code of pretrain models in torchvision

huangbiubiu: <p>PyTorch release popular models in <code>torchvision</code> with pre-trained checkpoints.</p>
<p>I want to reproduce the training process of these models, e.g. <code>mobilenetv2</code>, are there any official training code for those pre-trained models?</p>
<p>I see there is an example <a href="https://github.com/pytorch/examples/tree/master/imagenet" rel="nofollow noopener">https://github.com/pytorch/examples/tree/master/imagenet</a> to train networks with ImageNet, but I don’t know how to set these arguments for different networks.</p>

ptrblck: <p>You can define different models by passing the <code>--arch</code> argument with the desired model name.<br>
<a href="https://github.com/pytorch/examples/blob/391be73bbd191a08776c9ba05d29ef1e05c0c83b/imagenet/main.py#L131-L137">These lines of code</a> will then create the model (and <code>state_dict</code>, if you want to use a pretrained model).</p>

huangbiubiu: <p>But how can I know the other parameters used to train the network?<br>
For example, as mentioned in the MobileNet v2 paper, they train their network with RMSProp and some special learning rate schedule. Some implementations use extra data augmentation (e.g. label smoothing). I want to know about these training details when train the pretrain network.</p>
How to export to Tensorflow when model contains torch.atan2()?

drscotthawley: <p>I want to export my model to Tensorflow, which as I understand it means I need to export to ONNX first. (is that right?)</p>
<p>My model contains a torch.atan2() operation, which is not on the <a href="https://pytorch.org/docs/stable/onnx.html#supported-operators" rel="nofollow noopener">list of supported operators for ONNX export.</a>    Indeed, the <a href="https://github.com/onnx/onnx/blob/master/docs/Operators.md" rel="nofollow noopener">list for ONNX itself</a> only includes atan but not atan2, but Tensorflow has atan2.</p>
<p>So, if this were numpy I could just write my own little “my_atan2()” function that calls atan() and then uses some kind of <code>where()</code> to decide how many factors of pi  to add appropriately… maybe something like this?</p>
<pre><code class="lang-auto">def my_atan2(y, x):
    pi = torch.from_numpy(np.array([np.pi])).to(y.device, y.dtype)
    ans = torch.atan(y/x)
    ans = torch.where( (y&gt;0)*(x&lt;0), ans+pi, ans)   # upper left quadrant
    ans = torch.where( (y&lt;0)*(x&lt;0), ans+pi, ans)   # lower left quadrant
    # upper right quadrant and lower right quadrant, do nothing
    return ans
</code></pre>
<p>…But I’m guessing that won’t satisfy Autograd.  And looking at the code for <code>torch.atan2.backward()</code>… yea I don’t understand what’s going on there.</p>
<p>Any suggestions?</p>

drscotthawley: <p>Whoa, actually that seems to work!  But now I have <em>other</em> functions that also throw “undefined” errors, such as <code>torch.flip()</code> not being supported.</p>
<p>And my new DIY implementation of <code>torch.flip()</code> relies on <code>torch.arange()</code> which throws an ONNX export error about “index” not existing.</p>
<p>…bah.  This is no fun.  Is ONNX still the only way to get to Tensorflow from PyTorch?</p>
<p>(This is for a web app.  We may just go with Bokeh Server.)</p>
Transfer uint8_t[] to at::Tensor@CUDA and back

gebbissimo: <p>I am writing a small c++ module, whose input and output interface are images in uint8_t[] format. Currently, I am performing a lot of conversion steps. Can this be done any more efficient?</p>
<ul>
<li>at::Tensor t = torch::from_blob(uint8_t[] data_in, at::kByte)</li>
<li>t = t.to(at::kCUDA)</li>
<li>t = t.to(at::kFloat)</li>
<li>t = at::tranpose(t,1,2)</li>
<li>t = model.forward(std::vector<a>torch::jit::IValue</a> {t}).toTensorList().get(0)</li>
<li>t = t.round()</li>
<li>t = at::tranpose(t,2,1)</li>
<li>t = t.to(at::kByte)</li>
<li>t = t.to(at::kCPU)</li>
<li>memcpy(uint8_t[] data_out, t.data_ptr(), t.numel()*sizeof(uint8_t))</li>
</ul>
Function scripted by jit.export in с++

AZuev: <p>Hello. I scripted by torch.jit.export function init_state() but can’t use it in c++ code.</p>
<p>Example:<br>
class my_class(torch.nn.Module):<br>
def forward …</p>
<pre><code>@torch.jit.export
def init_state():
       pass
</code></pre>
<p>a = torch.jit.script(my_class)<br>
a.init_state().  <span class="hashtag">#is</span> working<br>
a.save(‘test.pt’)</p>
<p>In c++:<br>
auto b =torch:jit:load(“test.pt”);<br>
b.init_state(); //error</p>
<p>‘Struct torch:jit:Module’ has no member "init-state’’<br>
Please help me use init_state in c++.</p>

AZuev: <p>Works, without () in the end. Thank you very much)))</p>

AZuev: <p>Thank you. I understand. My class and function are more complicate than i show, i can’t run program, but it compiles with run_method and with get_method without (). I see now, that init_state not need jit.export in my case(forward use init_state)<br>
get_method with () //error<br>
no match for call to ‘(torch::jit::method)()’</p>

AZuev: <p>Thank you. It is working! Excellent)</p>
How to use different dataset every epoch?

lunasdejavu: <p>My mentor wants me to do the image novelty detection by normal two classes classification.the positive dataset is around 800 and the negative dataset genereted by a classification inference result for 2 millions.So I want to do generate a dataset which is combined by all the 800 positive dataset and 800 sampled from negative dataset with heavy data augmentation every epoch . I know there is  randomsampler function in pytorch but it seems can’t sample only one class randomly.</p>
Using one trained model as a extractor in another model

JakeAndFinn: <p>I am using a trained model as an extractor in another model and it seems like it is working ok, but I don’t know if I am doing it right or now.  So this is my model and how it is declared.</p>
<pre><code class="lang-auto">class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        image_modules = list(ae_model.children())[:-2] #get only the encoder layers
        self.modelA = nn.Sequential(*image_modules)
        # Shape of max pool = 4, 112, 112
        self.fc1 = nn.Linear(4*112*112, 500)
        self.fc2 = nn.Linear(500,500)
        self.out = nn.Linear(500, 10)
        
        self.drop = nn.Dropout(0.3)
        
    def forward(self, x):
        x = self.modelA(x)
        x = x.view(x.size(0),4*112*112)
        x = torch.relu(self.fc1(x))
        x = self.drop(x)
        x = torch.relu(self.fc2(x))
        x = self.drop(x)
        x = self.out(x)
        return x
</code></pre>
<p>Now I have done this for a Resnet50 model but it did not look like this. My extractor that I am using is this</p>
<pre><code class="lang-auto"># define the NN architecture
class ConvAutoencoder(nn.Module):
    def __init__(self):
        super(ConvAutoencoder, self).__init__()
        ## encoder layers ##
        self.conv1 = nn.Conv2d(3, 16, 3, padding=1)  
        self.conv2 = nn.Conv2d(16, 4, 3, padding=1)
        self.pool = nn.MaxPool2d(2, 2)
        
        ## decoder layers ##
        self.t_conv1 = nn.ConvTranspose2d(4, 16, 2, stride=2)
        self.t_conv2 = nn.ConvTranspose2d(16, 3, 2, stride=2)
    def forward(self, x):
        ## encode ##
        x = torch.relu(self.conv1(x))
        x = self.pool(x)
        x = torch.relu(self.conv2(x))
        x = self.pool(x)  # compressed representation
        ## decode ##
        x = torch.relu(self.t_conv1(x))
        x = torch.sigmoid(self.t_conv2(x))
        return x
</code></pre>
<p>I think this is working but I would like to get a second opinion on it.<br>
Thank you for any help with this.</p>

ptrblck: <p>I assume that <code>ae_model</code> is an instance of <code>ConvAutoencoder</code>.<br>
If that’s the case, note that <code>self.modelA</code> will only use the child modules of <code>ConvAutoencoder</code> and will not apply the functional API calls in its <code>forward</code> such as <code>torch.relu</code> and <code>torch.sigmoid</code> and will also use the max pooling layer once.</p>
<p><code>nn.Sequential</code> is a simple container to call modules in a sequential way. You could e.g. add the activation operations as modules (<code>nn.ReLU()</code> and <code>nn.Sigmoid()</code>) to the sequential model.<br>
I think the better way would be to create a new model with the desired submodules and just call it in <code>MyModel</code>.</p>

JakeAndFinn: <p>I see I should explain. I want just the encoder part so I want to output of the maxpooling layer only. So would what I have run the image through the encoder part and output the maxpooling layer part. From printing the output I think that is what is happening.<br>
Thank you again.</p>
Is there an error in all VGG architectures?

Valerio_Biscione: <p>Hello. I was trying to understand the architecture of VGG19_bn and I stumbled upon something interesting.<br>
After the convolution layers,  there is an adaptive average pooling(7,7). My understanding is that 7x7 is the requested output of the average pooling. However, the feature maps that feed into the average pooling are n x 7 x 7, meaning that the average will happen across each single element (so average of a 1x1 element, result in the element itself).  I checked stepping into the code that the output of the average pooling is exactly the same as it’s input, both in terms of size and in terms of valus.<br>
Why is the average pooling even there, since it’s not doing anything? Is this a mistake?<br>
Thanks</p>

ptrblck: <p>The adaptive pooling layers were added to relax the condition on specific input shapes.<br>
By default a lot of models are trained on images of <code>[batch_size, 3, 224, 224]</code>.<br>
As you said, for this shape the original architecture should be used.<br>
However, if you would like to increase the spatial size to e.g. <code>350x350</code> pixels, you would run into a shape mismatch error and would have to resize the image to <code>224x224</code>.<br>
To relax this condition and make these models more flexible, the adaptive pooling layers were added, which will return the desired output shape so that the number of input features of this activation will match the specified <code>in_features</code> of the following linear layers.</p>

Valerio_Biscione: <p>Got it! Thanks! Since we are at it, let me ask another question. I really like the ‘adaptive’ feature of the average pooling. Is there anything similar for the conv layer, where we specify the size of the output feature map, and the kernel size is automatically computed? Thanks</p>

ptrblck: <p>Unfortunately, there isn’t such a method for conv layers, as you would have to manipulate the trainable parameters for each different input shape.<br>
Since pooling layers do not have trainable parameters, it’s not problematic to change the kernel shape and setup.<br>
However, the filters in conv layers have a certain shape and if you would like to increase their spatial size, you would have to reinitialize the new elements (or use any other methods such as cloning/mirroring certain values).</p>