akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Hypernetwork implemnentation

Rabeeh_Karimi: <p>Hi<br>
I want to generate the weights for network 2 using network 1, for this assuming I have MLP for network 1, and then network 2 is a linear layer then we have:</p>
<p>Linear.weight = weight generated from network 2, requires_grad=False</p>
<p>is this the correct implementation of hyper-net?<br>
thanks</p>
How to feed a convolutional neural network with images already split into three channels?

Saveman: <p>Greetings to all. I am newbie to this framework and therefore have no idea what to do. I have a set of images. Every three images in it are the same object, but in different channels. I want to build a classifier that receives similar images as input and then defines them as one object of a specific class. I don’t understand how to feed three separate images to the input so that the network understands that these are parts of one whole. If I just give her specially grouped images by me, and then concatenate the result, does it work?</p>

NicolasChaverou: <p>Hey Saveman,<br>
May I understand that your images are stored as 3 files, each of which storing the R / G / B channel ?<br>
Are you planning to use an existing classifier architecture or make your own architecture ?<br>
The answer will probably be concatenating your 3 tensors (r / g / b) together but just want to make sure depending on what you need / want to do <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

Saveman: <p>Hi, Nicolas. Yes, one image is stored as three files, but not really RGB. These are images from a special instrument (imaging flow cytometer). Two images are Red and Green, and the third is the so-called brightfield image. As I understand it, an additional complication is that in all channels the images are grayscale. Let me explain. We tint the cage with special dyes and illuminate them with a laser. The device fixes the intensity of a particular color in different channels and the usual image in the optical range, but the pictures themselves remain gray. I will attach an example of how the image of one cell looks like this. Answering the second question - I would like to first use some modification of the standard architecture.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/b/a/ba3a8142c06939ee2d819f56841fb815498a99e1.jpeg" alt="9_Ch1.ome" data-base62-sha1="qzs9EC3JuU4jlDpKTIYD4fComWZ" width="344" height="75"></p>
Is Data Parallel(or DDP) equivalent to larger batch

rui_zhang_331: <p>Suppose I have batch size 256, and if I have 2 GPUs to use Data parallel, I can split a 512 batch data into two 256 batches, but in final optimization, It uses sum up of individual loss gradients which equals to gradient of loss sum up. Ex: (f(x1) + f(x2))’ = f’(x1) + f’(x2) . dose that mean it is same as a single 512 batch size?</p>
<p>Since our model may not perform well in large batch size, but we have large training data, so would like to know whether we should consider DP or DDP</p>

PistonY: <p>You can regard DP as a single 512 batch size, but DDP is different.</p>

rui_zhang_331: <p>can you elaborate more on why DDP is different? they still use All reduce to “average” gradient?</p>

PistonY: <aside class="quote" data-post="1" data-topic="77891">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/deeply/48/3840_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/dataparallel-vs-distributeddataparallel/77891">DataParallel vs DistributedDataParallel</a> <a class="badge-category__wrapper " href="/c/distributed/12"><span data-category-id="12" data-drop-close="true" class="badge-category "><span class="badge-category__name">distributed</span></span></a>
  </div>
  <blockquote>
    Hi, 
what is the difference between 
model = nn.parallel.DistributedDataParallel(model, device_ids=[args.gpu]) 
and 
model = nn.DataParallel(model, device_ids=[args.gpu]) 
?
  </blockquote>
</aside>

<p>I think this should be helpful.</p>

rui_zhang_331: <p>Yes, I checked this thread, I knew DDP is better than DP as it dose fully All reduce  and multi processing, instead DP need to calculate the loss in main GPU thus in balanced GPU utilization.</p>
<p>But here my question is if # of GPU is equivalent to<code> batch_size * #</code> in DP ,  is it same in DDP? since in DDP “At the end of the backwards pass, every node has the averaged gradients” , so to me seems both are doing some “average” here</p>

PistonY: <p>No, they are different. DDP will calculate loss on each device and get grad then do “average”, DP just do it in “main” device and update model.</p>

rui_zhang_331: <p>Thanks, I understand, but why DP is same as large batch size, but DDP is not? essentially they do some “average”, “sum” for all gradient calculated in each GPUs, so model side should be same?</p>

PistonY: <p>DP will all gather NN result to main device and then all operations are doing in it, that may contain calculate loss get grad and update model param.<br>
DDP keep result on each device until get grad, and do you say <code>average</code> then update model on each device.</p>
Access scale and zero_point from the weight within QuantizedLinearReLU

ninfueng: <p>Hello,</p>
<p>I try to access scale and zero_point from the weight within QuantizedLinearReLU  by using methods <code>scale</code> and <code>q_scale</code>.  The results are failed as shown in below:</p>
<pre><code class="lang-auto">model.fc1
Out[7]: QuantizedLinearReLU(in_features=4, out_features=4, scale=0.04960649833083153, zero_point=0, qscheme=torch.per_channel_affine)

model.fc1.weight().scale
Out[8]: AttributeError: 'Tensor' object has no attribute 'scale'

model.fc1.weight().q_scale()
Out[9]: RuntimeError: Expected quantizer-&gt;qscheme() == kPerTensorAffine to be true, but got false.  (Could this error message be improved?  If so, please report an enhancement request to PyTorch.)
</code></pre>
<p>The scale and zero_point that I want to access are <strong>zero_point=tensor([0, 0, 0, 0]), axis=0 and scale=tensor([0.0145, 0.0016, 0.0132, 0.0124], dtype=torch.float64)</strong> as below:</p>
<pre><code class="lang-auto">model.fc1.weight()
Out[9]: 
tensor([[-0.1880, -1.3018,  0.8100,  1.8369],
        [-0.0033,  0.1172, -0.0065, -0.2083],
        [-0.9236,  0.3299, -1.6889, -1.3195],
        [-0.2718,  0.4078,  1.0997,  1.5693]], size=(4, 4), dtype=torch.qint8,
       quantization_scheme=torch.per_channel_affine,
       scale=tensor([0.0145, 0.0016, 0.0132, 0.0124], dtype=torch.float64),
       zero_point=tensor([0, 0, 0, 0]), axis=0)
</code></pre>
<p>Is there any way to access these parameters without manually copy from the prompt? Thank you.</p>

dskhudia: <p>For per_channel_affine quant scheme, please use q_per_channel_scales() and q_per_channel_zero_points()</p>

ninfueng: <aside class="quote no-group" data-username="dskhudia" data-post="2" data-topic="105117" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/d/ebca7d/48.png" class="avatar"> dskhudia:</div>
<blockquote>
<p>For per_channel_affine quant scheme, please use q_per_channel_scales() and q_per_channel_zero_points()</p>
</blockquote>
</aside>
<p>It works perfectly,  thank you <a class="mention" href="/u/dskhudia">@dskhudia</a>.</p>
Multiply each scalar in matrix A with tensor B

Bob_Li: <p>Suppose I have a matrix e.g. <code>A = tensor([[0, 1, 2], [3, 4, 5]]) </code>, and I have another tensor B e.g. <code>B = torch.tensor([1, 5, 2, 4])</code>, how can I multiply each scalar in A with B, to get C of shape [2, 3, 4] in this example?</p>

KFrank: <p>Hi Bob!</p>
<aside class="quote no-group" data-username="Bob_Li" data-post="1" data-topic="105190" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/bob_li/48/29699_2.png" class="avatar"> Bob_Li:</div>
<blockquote>
<p>Suppose I have a matrix e.g. <code>A = tensor([[0, 1, 2], [3, 4, 5]]) </code>, and I have another tensor B e.g. <code>B = torch.tensor([1, 5, 2, 4])</code>, how can I multiply each scalar in A with B, to get C of shape [2, 3, 4] in this example?</p>
</blockquote>
</aside>
<p>I believe you are asking for the generalized <em>outer product.</em>  Please<br>
try <code>torch.ger()</code> (<code>torch.outer</code>) with <code>.reshape()</code> or <code>torch.einsum</code>:</p>
<pre data-code-wrap="python"><code class="lang-python">ABa = torch.ger (A.reshape([6]), B).rehshape ([2, 3, 4])
ABb = torch.einsum ('ij, k -&gt; ijk', A, B)
</code></pre>
<p>These two versions should give you the same result.  Do they do what<br>
you want?</p>
<p>Best.</p>
<p>K. Frank</p>

Bob_Li: <p>Wow! Thanks a lot, works like charm~</p>
MNIST Classifier in Pytorch - using prior to exclude class

catubc: <p>Hello everyone</p>
<p>I’ve written pytorch code to us ResNet to predicting digits 0-9 from the standard MNIST database. It works pretty great. But I want to improve it. In particular, in my application, I usually have prior information which tells me that a particular image is not a particular value. For example, I know that an image being classified is not the number “6”.</p>
<p>Is there any known approach to use this prior information to restrict the classification so that the classifier output will give a 0% probability of the image being 6? I obviously know that I can just zero out the classifier output for label 6 and renormalize the other classes (i.e 0-9 excluding 6). But this is just a linear scaling on the final layer output.</p>
<p>I’d like to use this prior information at the input layer (?) or somewhere in the network to take advantage of all the nonlinear scaling of the network. I’m ok with writing a toy CNN from scratch (i.e. bypassing ResNet to begin).</p>
<p>Any advice is much appreciated. I know this is a bit off topic for pytorch forums only, but I’m hoping someone has some general advice.</p>
Distributing one dimension over others

m.hassanin: <p>Hi guys,</p>
<p>I need  to unsqueeze a tensor by one dimension provided that this dimension is added to the following dim. Can I do without for loop?</p>
<p>For example, A= [1,2,4,4], I need to make it [1,5,5].</p>
<p>Any thoughts?</p>

alwynmathew: <p>Tensor A with 1 x 2 x 4 x 4 has 32 element. You cant squeeze it to 1 x 5 x 5 which will be 25 elements.</p>
<p>If you just want to sqeeze dim 1 somehow, <code>A = A.mean(dim=1)</code> gives 1 x 4 x 4 tensor</p>

m.hassanin: <p>Thanks Alwyn for your prompt reply.</p>
<p>Okay, We can say that  I need it in the form A=[1,8,4].</p>
<p>This what I had done with the loop,</p>
<p>x1=[]<br>
for i in range(x.shape[1]):<br>
x_t = x[:,i,:,:]<br>
x1.append(x_t)</p>
<pre><code>    x_t = torch.cat(x1, dim=1)</code></pre>

alwynmathew: <p>You can use <a href="https://pytorch.org/docs/stable/tensors.html#torch.Tensor.view" rel="noopener nofollow ugc">view</a> like below:</p>
<pre><code class="lang-auto">A =  = torch.rand((1, 2, 4, 4)) # 1 x 2 x 4 x 4
batch, channel, height, width = A.shape
A = A.view(batch, channel*height, width) # 1 x 8 x 4
</code></pre>
About the way to get each layer of a module and number of layers in C++

Chen0729: <p>Hi, All</p>
<p>I am trying to apply the relu to the output of each layer in C++</p>
<p>The python code is like:<br>
for i in range(len(self.layers)):<br>
m = self.layers[i]<br>
x_f = F.relu(m(x))</p>
<p>where the layers are defined as<br>
self.layers = torch.nn.ModuleList([<br>
LinearWeightNorm(input_dim, 1000),<br>
LinearWeightNorm(1000, 500),<br>
LinearWeightNorm(500, 250),<br>
LinearWeightNorm(250, 250),<br>
LinearWeightNorm(250, 250)]<br>
)<br>
self.final = LinearWeightNorm(250, output_dim, weight_scale=1)</p>
<p>However, in C++ code, if I wrote</p>
<p>for (int64_t i = 0; i &lt; 5; i++)<br>
{</p>
<pre><code>  torch::Tensor x_f = torch::nn::functional::relu(layers[i]-&gt;as&lt;LinearWeightNorm&gt;()-&gt;forward(x));
</code></pre>
<p>I met an error in “layers[i]” .<br>
as I checked layers-&gt;size() is 0,</p>
<p>Any comments for better solutions please ?<br>
Thanks.</p>

ptrblck: <p>Based on the error it seems you are not initializing the <code>ModuleList</code> properly.<br>
Have a look at <a href="https://github.com/pytorch/pytorch/blob/bdd7dbfd4b75e66a88d393993b41c77f576f74fc/test/cpp/api/modulelist.cpp#L26-L45">this test</a> to see the usage of <code>ModuleList</code> in C++.</p>

Chen0729: <p>Thanks.</p>
<p>When I change to<br>
auto a=layers[0+i]-&gt;as()-&gt;forward(x);</p>
<p>The error reports “index  out of range”<br>
terminate called after throwing an instance of ‘c10::Error’<br>
I defined the layers as<br>
struct DiscriminatorImpl:torch::nn::Module {<br>
torch::Tensor x;</p>
<pre><code>torch::nn::ModuleList layers;
LinearWeightNorm final=nullptr;

int64_t input_dim=28*28;
Scalar output_dim=10;
Scalar weight_scale=1;
DiscriminatorImpl(){}
 DiscriminatorImpl(int64_t input_dim, int64_t output_dim)
{
    layers = register_module("layer", torch::nn::ModuleList(
        
            LinearWeightNorm(input_dim, 1000, true, 1, 0.1),
            LinearWeightNorm(1000, 500,true, 1, 0.1),
            LinearWeightNorm(500, 250, true, 1, 0.1),
            LinearWeightNorm(250, 250, true, 1, 0.1),
            LinearWeightNorm(250, 250, true, 1, 0.1)
    ));
    final= register_module("final",LinearWeightNorm(250, output_dim, true,1, 0.1));
}
</code></pre>
<p>May I ask how shall I initialize the ModuleList properly in this case please ?</p>
<p>Thanks.</p>
Concatenate two dimensions inside one tensor

m.hassanin: <p>Hi all,</p>
<p>I wanna concatenate two columns inside one tensor without any loops, Any thoughts?</p>
<p>e.g.  A=[2,4,5,5], How it becomes A=[2,20,5]?</p>
<p>Thanks</p>

KFrank: <p>Hi Mohammad!</p>
<aside class="quote no-group" data-username="m.hassanin" data-post="1" data-topic="105193" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/m.hassanin/48/31935_2.png" class="avatar"> m.hassanin:</div>
<blockquote>
<p>A=[2,4,5,5], How it becomes A=[2,20,5]?</p>
</blockquote>
</aside>
<p>I’m not sure what you mean by “concatenate.”  I’m guessing that the<br>
values in brackets (<span class="chcklst-box fa fa-square-o fa-fw"></span>) indicate the shapes of the tensors.  Does:</p>
<pre data-code-wrap="python"><code class="lang-python">A.reshape ([2, 20, 5])
</code></pre>
<p>give you what you are looking for?</p>
<p>If not, could you give an explicit numerical example of what your<br>
result tensor should be?</p>
<p>Best.</p>
<p>K. Frank</p>

m.hassanin: <p>Hi Frank,</p>
<p>Thanks for your answer.</p>
<p>After I posted I done it as you suggest exactly.</p>
<p>Thanks</p>
How to use data parallel in Bert

XiMing_Dong: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/0/f0ac46669803d00678407fc58c3a600ca1e315be.png" data-download-href="https://discuss.pytorch.org/uploads/default/f0ac46669803d00678407fc58c3a600ca1e315be" title="Screen Shot 2020-12-04 at 7.50.40 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/0/f0ac46669803d00678407fc58c3a600ca1e315be_2_443x500.png" alt="Screen Shot 2020-12-04 at 7.50.40 PM" data-base62-sha1="yl5GNTuL4OuiCrDhfJEqtBXCj1k" width="443" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/0/f0ac46669803d00678407fc58c3a600ca1e315be_2_443x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/0/f0ac46669803d00678407fc58c3a600ca1e315be_2_664x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/f/0/f0ac46669803d00678407fc58c3a600ca1e315be.png 2x" data-dominant-color="EBECEC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2020-12-04 at 7.50.40 PM</span><span class="informations">733×826 102 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>File “/content/drive/My Drive/DuplicateBugFinder/proposed.py”, line 62, in forward<br>
bert_long = self.bert_pro(bert_long)<br>
/nn/functional.py", line 1688, in linear<br>
if input.dim() == 2 and bias is not None:<br>
TypeError: ‘int’ object is not callable</p>
<p>I do not know what wrong with my code, can you help me thanks</p>
What would my training model look like? How would I define it?

kelsin77: <p>Based on this, what would be the most appropriate way to define my training model so that I could then record loss and accuracy?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/2/82d23292991fb8e725f4056440e7f5380c1e6153.png" data-download-href="https://discuss.pytorch.org/uploads/default/82d23292991fb8e725f4056440e7f5380c1e6153" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/2/82d23292991fb8e725f4056440e7f5380c1e6153_2_690x491.png" alt="image" data-base62-sha1="iFipSdFhvC0ut71eat3YheytcuD" width="690" height="491" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/2/82d23292991fb8e725f4056440e7f5380c1e6153_2_690x491.png, https://discuss.pytorch.org/uploads/default/optimized/3X/8/2/82d23292991fb8e725f4056440e7f5380c1e6153_2_1035x736.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/8/2/82d23292991fb8e725f4056440e7f5380c1e6153_2_1380x982.png 2x" data-dominant-color="F1F1F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">2278×1624 339 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>The model definition is independent from recording the loss and accuracy so could you explain the issue a bit more, please?</p>
<p>PS: you can post code snippets by wrapping them into three backticks ```, which would make debugging easier. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
What are the expected model sizes on the GPU of #vision resnets?

maikefer: <p>I’m currently chasing an OOM-Error thrown in the forward pass on my GPU (Tesla V100-SXM2-32GB). Loading the resnext50 from torchvision onto the cpu uses 27GB already. I wonder if that is as expected and how much additional space is needed in the forward pass. Is there any overview on model sizes and needed memory?</p>

ptrblck: <p>The memory usage depends not only on the stored parameters and activations but also on the used libraries. E.g. if you are using cudnn with its benchmark mode (via <code>torch.backends.cudnn.benchmark = True</code>), the first iteration will profile different algorithms and will select the fastest one, which might use more memory.<br>
The available kernels depend on the cudnn version as well as the used GPU, so I think the best way to profile the model would be to check the max. batch size for your current setup.</p>
<p>That being said, if you disable e.g. cudnn and use the native implementations, you might be able to calculate the approx. memory usage for the current PyTorch version.</p>
Parameters supplied to CrossEntropyLoss not working

rol227: <pre><code class="lang-auto">import torch
from torch import nn, optim
import torch.nn.functional as F
import torchvision


train_loader = torch.utils.data.DataLoader(
    torchvision.datasets.MNIST('/files/', train=True, download=True,
                               transform=torchvision.transforms.Compose([
                                   torchvision.transforms.ToTensor(),
                                   torchvision.transforms.Normalize(
                                       (0.1307,), (0.3081,))
                               ])), batch_size=64, shuffle=True)

test_loader = torch.utils.data.DataLoader(
    torchvision.datasets.MNIST('/files/', train=False, download=True,
                               transform=torchvision.transforms.Compose([
                                   torchvision.transforms.ToTensor(),
                                   torchvision.transforms.Normalize(
                                       (0.1307,), (0.3081,))
                               ])), batch_size=1000, shuffle=True)


class Net(nn.Module):

    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(1, 8, kernel_size=3)
        self.conv1_bn = nn.BatchNorm2d(8)
        self.conv2 = nn.Conv2d(8, 16, kernel_size=3)
        self.conv2_bn = nn.BatchNorm2d(16)
        self.conv3 = nn.Conv2d(16, 32, kernel_size=3)
        self.conv3_bn = nn.BatchNorm2d(32)
        self.fc1 = nn.Linear(1, 100)
        self.fc2 = nn.Linear(100, 10)

    def forward(self, x):
        x = F.relu(F.max_pool2d(self.conv1_bn(self.conv1(x)), 2))
        x = F.relu(F.max_pool2d(self.conv2_bn(self.conv2(x)), 2))
        x = F.relu(F.max_pool2d(self.conv3_bn(self.conv3(x)), 2))
        x = F.relu(self.fc1(x))
        return self.fc2(x)


net = Net()
optimizer = optim.SGD(net.parameters(), lr=0.01)
criterion = nn.CrossEntropyLoss()


for epoch in range(3):
    running_loss = 0.0
    for i, data in enumerate(train_loader, 0):
        inputs, labels = data

        optimizer.zero_grad()

        outputs = net(inputs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

        running_loss += loss.item()
        if i % 2000 == 1999:
            print('[%d, %5d] loss: %.3f' %
                  (epoch + 1, i + 1, running_loss / 2000))
            running_loss = 0.0

print('Finished Training')
</code></pre>
<p>I get the following RuntimeError: only batches of spatial targets supported (3D tensors) but got targets of dimension: 1</p>

ptrblck: <p>The shape of the model outputs doesn’t match the target shape.<br>
Based on your model definition it seems you are dealing with a multi-class classification and your model should return an output in the shape <code>[batch_size, nb_classes]</code>.<br>
However, since you are never flattening the activation in <code>forward</code>, I assume the output might have the shape <code>[batch_size, *, *, 10]</code>.<br>
Add <code>x = x.view(x.size(0), -1)</code> before passing <code>x</code> to <code>self.fc1</code> and it might work.</p>
Transformations don't apply to labels

CrlSolitu: <p>Hi, since I updated torch to 1.7 the transformations on my dataset don’t apply to my labels, even though I did not change anything. The transformations work fine on the normal data. Maybe somebody has a quick fix, here’s the code:</p>
<pre><code class="lang-auto">batch_size = 1

path_file = "data.csv"

train_inputs, train_labels, val_inputs, val_labels = BatchMaker.BatchMaker(path_file)


transform = transforms.Compose([
    transforms.ToPILImage(),
#     transforms.Resize((165, 220)),
    transforms.RandomRotation(degrees=random.randint(0,30)),
    transforms.RandomHorizontalFlip(p=0.5),
    transforms.RandomVerticalFlip(p=0.5),

])

class CreateDataset(Dataset):
    def __init__(self, inputs, labels, transform=transform):
        self.inputs = torch.FloatTensor(inputs)
        self.labels = torch.FloatTensor(labels)

        self.transform = transform
        
    def __getitem__(self, index):
        x = self.inputs[index]
        y = self.labels[index]
        
        if self.transform:
            seed = np.random.randint(2147483647) 
            random.seed(seed)

            x = self.transform(x)
            y = self.transform(y)
            if random.random() &gt; 0.5:
                x = TF.adjust_brightness(x, random.uniform(0.4,0.6))

            if random.random() &gt; 0.5:
                x = TF.adjust_contrast(x, random.uniform(0.4,0.6))
            x = TF.to_tensor(x)

            random.seed(seed)
            
            y = TF.to_tensor(y)
            
        y = y/np.sum(np.array(y))
        
        return x.view(1,180,240), y.view(1,180,240)

    def __len__(self):
        return len(self.inputs)

    
# Get the data, transform it
data = {
   'train':
   CreateDataset(train_inputs, train_labels),
   'val':
   CreateDataset(val_inputs, val_labels, transform=None),
#     'test':
#    CreateDataset(test_inputs, test_labels, transform=None)
}

# Load Data in batches, shuffled
dataloaders = {
   'train': DataLoader(data['train'], batch_size=batch_size, shuffle=True, drop_last=True),
   'val': DataLoader(data['val'], batch_size=batch_size, shuffle=False, drop_last=True),
#     'test': DataLoader(data['test'], batch_size=batch_size, shuffle=False, drop_last=True),
}
</code></pre>

ptrblck: <p>Could you explain the issue a bit more?<br>
Are these two lines of code not executed:</p>
<pre><code class="lang-python">y = TF.to_tensor(y)
y = y/np.sum(np.array(y))
</code></pre>
<p>and the dataset returns unexpected objects for <code>y</code>?</p>
Split image into Non-overlapping sub-blocks

learned_fool: <p>Hi Guys!</p>
<p>I am basically trying to split an MNIST Image Into smaller non-overlapping square blocks of size  <code>block_size</code> and trying to shuffle pixel values in it randomly. However, a naive implementation from my using nested for loops causes the function to be terribly slow. I have tried reading the docs for some functions in <code>torch</code> that can help me out; I couldn’t find any. Could Someone please help me with doing this using slicing? Any help would be highly appreciated! Thanks in advance! I am attaching the code for reference.</p>
<pre><code class="lang-python">#takes a batch of images (BxCxHXW) and shuffles pixels
#I was executing this on mnist, hence I didn't consider the channel dimension
def Pixel_Shuffle(batch,block_size=2):

  it = int(batch.shape[2]/block_size)

  for img in batch :

    for i in range(it**2) :

      row = int(i/it)
      row = row*2
      col = i%it
      col = col*block_size
      img[0,row:row+block_size,col:col+block_size] = Block_Shuffle(img[0,row:row+block_size,col:col+block_size])

return batch

def Block_Shuffle(block):
  ord = torch.randperm(block.numel())
  block = block.reshape(-1)[ord].reshape(block.shape)
  return block
</code></pre>

ptrblck: <p>You could use <code>tensor.unfold</code> to create the patches and reshape them back as explained in <a href="https://discuss.pytorch.org/t/unfold-a-tensor/101922/4">this post</a>.</p>
DDP+ SYNC BN always?

Scott_Hoang: <p>Is it always recommended to use SYNC BN every time you use DDP? Is there any exception?</p>

pritamdamania87: <p>Usually SYNC BN is used because for large training runs the batch size per GPU is pretty small and you can’t gather enough statistics independently for batch normalization. If you have a large enough batch size per GPU, you might be able to get away without SYNC BN.</p>
<p>Although, a lot of this depends on your model and you should try with and without SYNC BN to see if your model converges fine in both cases. If your model converges fine without SYNC BN, I’d recommend avoiding SYNC BN since there is a perf overhead while running with SYNC BN due to synchronization among processes.</p>
Saving checkpoints with DistributedDaraParallel

Seo: <p>Hi everyone,<br>
I was trying to implement a distributed training for my model and I have found to have an issue when a new checkpoint (best model) has to be saved on disk. Whenever I try to execute my training with 2 processes (one process per GPU) using DistributedDataParallel, the process with rank 1 stops without any error in the output while the master process continues to work for a while (I think then it will be stopped since the process 1 is no more able to sync).<br>
This problem arises whenever I use the following piece of code to save my checkpoint:</p>
<pre><code class="lang-auto">        if rank == 0 and loss_trend['dev'][-1] &lt; best_loss:
            best_loss = loss_trend['dev'][-1]
            if cfg.LOGGING.CHECKPOINTS:
                assert ckp_file, 'Checkpoint file not defined'
                state_dict = model.module.cpu().state_dict()
                torch.save(state_dict, ckp_file)
            model.cuda(gpu)
            logger.log('best model saved at: {}'.format(ckp_file))
</code></pre>
<p>The problem is fixed I use deepcopy on my model without touching (without moving to cpu) the original one:</p>
<pre><code class="lang-auto">        if rank == 0 and loss_trend['dev'][-1] &lt; best_loss:
            best_loss = loss_trend['dev'][-1]
            if cfg.LOGGING.CHECKPOINTS:
                assert ckp_file, 'Checkpoint file not defined'
                state_dict = copy.deepcopy(model.module).cpu().state_dict()
                torch.save(state_dict, ckp_file)
            logger.log('best model saved at: {}'.format(ckp_file))
</code></pre>
<p>Since the deepcopy introduces some overhead, I want to know why it works in this case and if there are available other methods to solve my problem. Here my model is wrapped using <code>DistributedDataParallel</code>.<br>
Thank you</p>

pritamdamania87: <p>What is the error that you encounter when you don’t deepcopy the model? Does rank 0 just get stuck and rank 1 exits successfully? If so, do you know where rank 0 is stuck?</p>
Apply indicator function to pytorch tensor

pascal_notsawo: <p>Hello.<br>
I would like to implement the indicator function of a set with pytorch (pytorch in particular because I need to use it as an activation function for one of my models).</p>
<p>I take the case of the derivative of Parameterised ReLU (parameterised by a real a), which is 1 for positive numbers and a elsewhere. I would like to be able to implement this derivative so that it can support batch sizes greater than 1.<br>
Here is my example for relu.</p>
<pre><code class="lang-auto">import torch
activation_function = torch.relu
deriv_activation_function = lambda x : (x &gt; 0).float()
</code></pre>

sigma_x: <p>why not just use the tensor product, for example, when you want to apply penalty if the sum of the output exceeds 1:</p>
<pre><code class="lang-auto">import torch

v=torch.rand(3)
penalty = torch.sum()*(torch.sum()&gt;1)
</code></pre>
The Hogwild example doesn't use my GPU when executing

jlopez: <p>I implemented the Hogwild example program from here. My PC has a GPU, but when I execute the program it does not use the GPU but all the CPU cores, why? How can I indicate I want it to be run with the GPU instead?</p>

ptrblck: <p>You can use the GPU by passing the <code>--cuda</code> argument as seen <a href="https://github.com/pytorch/examples/blob/a3f28a26851867b314f4471ec6ca1c2c048217f1/mnist_hogwild/main.py#L30-L31">here</a>.</p>

jlopez: <p>Thanks, but I get an error: RuntimeError: Cannot pickle CUDA storage; try pickling a CUDA tensor instead, do you know how I can fix it?</p>

ptrblck: <p>Could you try to move <a href="https://github.com/pytorch/examples/blob/a3f28a26851867b314f4471ec6ca1c2c048217f1/mnist_hogwild/main.py#L74-L75">these two lines</a> right after the <code>args = parser.parse_args()</code> call?<br>
I needed to move them as I got an error as:</p>
<pre><code class="lang-python">RuntimeError: context has already been set
</code></pre>
<p>I’ll create an issue with a fix if I can reproduce it using the nightly (my current PyTorch build is a source build from a few weeks ago).</p>

jlopez: <p>I did the moving of those lines and after running the program I got the context error, were you able to fix it?</p>

ptrblck: <p>I got the before changing anything. After I moved the lines of code it worked.<br>
Are you only seeing the error after moving?</p>

jlopez: <p>Hi, the context error displays when moving the lines as you told me, the pickle error occurs when running the posted code as is.</p>
How to backpropagate beyond the distance between two models' weights

Xingchen_Zhao: <p>The formula in the screenshot is what I want to implement.<br>
Basically, there are two models U and W.</p>
<ol>
<li>we train the W on source data, then we multiply a trainable a to the loss and update the weights.</li>
<li>we train the U on training data, then we add the loss to the l2 norm of the “distance between U’s weight and W’s weight”, and update the U.</li>
<li>we update the trainable a by minimizing the loss of U on the validation dataset.</li>
</ol>
<p>Therefore, to optimize the trainable a, we need three steps. It is like a meta-learning problem, so I tried higher library. However, I got the problem that I cannot optimize the “a”, because it seems that l2 norm of “distance between U and W” is not on the computation graph. Is there any way to fix this problem? Also I wonder does the algorithm make sense? Thanks!<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/f/d/fd090efbf46c91daa0cb59c3c7d8121bed332bb0.png" alt="Screen Shot 2020-12-02 at 22.27.31" data-base62-sha1="A6scSVQoj2AS3SbNWnhNvpftp8Q" width="562" height="106"></p>
Unable to find torch_shm_manager when importing pytorch with python 3.7

Jimmy2027: <p>Hello, while pytorch works fine for me with python3.6, I get this error when trying to import it in python3.7 :</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "/usr/lib/python3.7/site-packages/torch/__init__.py", line 427, in &lt;module&gt;
    _C._initExtension(manager_path())
  File "/usr/lib/python3.7/site-packages/torch/__init__.py", line 422, in manager_path
    raise RuntimeError("Unable to find torch_shm_manager at " + path)
RuntimeError: Unable to find torch_shm_manager at /usr/lib/python3.7/site-packages/torch/bin/torch_shm_manager
</code></pre>
<p>Why is torch looking for torch_shm_manager under <code>/usr/lib/python3.7/site-packages</code>? I find torch_shm_manager under <code>/usr/bin/torch_shm_manager</code>.<br>
Is there anything wrong with my install?</p>

ptrblck: <p>How did you install PyTorch? Did you build it from source or did you install the conda/pip binaries?</p>

Jimmy2027: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> I built it from source via the gentoo package manager: portage</p>

ptrblck: <p>I’m not familiar with the Gentoo and this package manager, but are you able to install and run it without these tools?</p>
What is the best way to implement this loss regularization term in Pytorch C++?

Carol_Ye_Liu: <p>I am working on a loss regularization term which looks like this: <img src="https://discuss.pytorch.org/uploads/default/original/3X/7/f/7f5044fb1adfbc2c2aab6ec2c3797275bdc62283.png" alt="image" data-base62-sha1="iagC94LPJ8YTIdAaIsSAJaCSnyX" width="326" height="51"> and by taking advantage of this identity<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/b/a/ba2e6a15f9ed3c9e99c09fddeac7ff011f5ea972.png" alt="image" data-base62-sha1="qz2fCgYA9C2eZH7Bw5RZvyYU6m6" width="260" height="36"> , I implement the loss as follows(y is my noise, w is my latent, fake image is my g(w)).<br>
std::vector<a>torch::Tensor</a> g_path_regularize(const torch::Tensor&amp; fake_img, const torch::Tensor&amp;<br>
latents,  const torch::Tensor&amp; mean_path_length, Scalar decay=0.01){<br>
std::vector<a>torch::Tensor</a> res;<br>
auto noise = torch::randn_like(fake_img) / sqrt(<br>
fake_img.size(2) * fake_img.size(3)<br>
);</p>
<p>auto grad = torch::autograd::grad(<br>
{(fake_img * noise).sum()}, {latents}, {}, true, true)[0];</p>
<p>auto path_lengths = torch::sqrt(torch::mean(torch::sum(torch::mul(grad, grad), 2),1));</p>
<p>auto path_mean = mean_path_length + decay * (path_lengths.mean() - mean_path_length);//tensor</p>
<p>auto path_penalty = torch::mean(torch::mul(path_lengths - path_mean, path_lengths - path_mean) );</p>
<p>res={path_penalty, path_mean.detach(), path_lengths};<br>
return res;<br>
}</p>
<p>And in the training process, i have a backward step on my loss regularization term and that can take a relative long time. I wonder if there are better way to implement this. Thanks!</p>
Transformer model to encode the set of sentences

NadiaMe: <p>Here is the code that attempts to learn to encode for paragraphs (the set of sentences that are themselves encoded).</p>
<pre><code class="lang-auto">class TransformerEnc(nn.Module):
    def __init__(self):
        super(TransformerEnc, self).__init__()
        d_model = 1024
        self.posenc = PositionalEncoding(d_model)
        encoder_layer = nn.TransformerEncoderLayer(d_model, nhead=8)
        self.model = nn.TransformerEncoder(encoder_layer, num_layers=6)

    def forward(self, x):
        x = self.posenc(x)
        output = self.model(x)
        output = torch.sum(output, 1)
        return output
</code></pre>
<p><code>forward</code> function gets tensor <code>x</code> of shape<code> [64, 20, 1024]</code>.<br>
Where 20 is the max number of sentences in a paragraph and 1024 is encoded sentence.<br>
Now I need the <code>output</code> of the size<code> [64, 1024]</code> where 20 sentences got embedded in one 1024 dim vector.</p>
<p>When I train a model with this embedding I get a random accuracy. If I replace this code with LSTM it trains well. So clearly there is some issue.<br>
Any ideas what might be the problem?</p>
Datasets and Dataloaders from single channel images stored in numpy arrays

Aseem: <p>I am trying to train a CNN using images stored in numpy that have a single channel. I am using Grayscale(num_output_channels=3) to transform them into 3 channels so as to be able to use transfer learning but keep getting the error shown below. Any help will be greatly appreciated.</p>
<pre><code class="lang-auto"># Load training images and labels
image_train = np.load('data/Hela_Hoechst_GM130/x_train.npy') #(1848, 135, 135, 3)
label_train = np.load('data/Hela_Hoechst_GM130/y_train.npy') #(1848,)

# Load test images and labels
image_valid = np.load('data/Hela_Hoechst_GM130/X_test.npy') #(113, 135, 135, 3)
label_valid = np.load('data/Hela_Hoechst_GM130/Y_test.npy') #(113,)

# Extracting DNA dataset from  channel 1
train_ds_DNA = image_train[:,:,:,0]
valid_ds_DNA = image_valid[:,:,:,0]

train_tfms = tt.Compose([tt.Grayscale(num_output_channels=3)])
valid_tfms = tt.Compose([tt.Grayscale(num_output_channels=3)])

class MyDataset(Dataset):
    def __init__(self, data, target, transform):
        self.data = torch.from_numpy(data).float()
        self.target = torch.from_numpy(target).long()
        self.transform = transform
        
    def __getitem__(self, index):
        x = self.data[index]
        y = self.target[index]
        
        if self.transform:
            x = self.transform(x)
        
        return x, y
    
    def __len__(self):
        return len(self.data)

 

train_ds = MyDataset(train_ds_DNA, label_train, train_tfms)
train_dl = DataLoader(
    train_ds,
    batch_size=128,
    shuffle=False,
    num_workers=2,
    pin_memory=torch.cuda.is_available())

for i, (images, labels) in enumerate(train_dl):
    print(type(images))
</code></pre>
<p>I am getting an error <strong>TypeError: Input image tensor should have at least 3 dimensions, but found 2</strong></p>

Alexey_Demyanchuk: <p>Hi, after you’re converting from numpy to tensor, make sure you have an additional channel dimension (channel first), as of this transformation requires it if it is working with tensors. You can do it with: <code>x = x.unsqueeze(0)</code>, this adds channel dimension to be the first one.<br>
From <a href="https://pytorch.org/docs/stable/torchvision/transforms.html" rel="noopener nofollow ugc">docs.</a></p>
<blockquote>
<p>The image can be a PIL Image or a Tensor, in which case it is expected to have […, 3, H, W] shape, where … means an arbitrary number of leading dimensions</p>
</blockquote>

Aseem: <p>Thanks for your response. The shape of <strong>train_ds_DNA</strong>  is (1848, 135, 135). Using x = x.unsqueeze(0) after x = self.data[index] in the above code, I get this to  (1848, 1, 135, 135). The reason for using tt.Grayscale is to get the shape to (1848, 3, 135, 135) with same data in the three channels. But this transform seems to require three channels in the input image. Is that not what it is supposed to do? The error now is <strong>Input image tensor should 3 channels, but found 1</strong><br>
Thanks again for your time.</p>
Backprop through Jacobian

ppower1: <p>I would like to construct my loss function so that it is a function of the jacobian as follows.</p>
<p>model = nn.Sequential(<br>
nn.Linear(1, 1),<br>
nn.ReLU())<br>
opt = torch.optim.SGD(model.parameters(), lr = 0.001)<br>
z = torch.tensor([2.])<br>
det = torch.autograd.functional.jacobian(model, z)<br>
loss = (torch.abs(det))**2<br>
loss.backward()</p>
<p>I would appreciate any suggestions on how I could get this to work. Thanks</p>

albanD: <p>Hi,</p>
<p>You just need to pass the <code>create_graph=True</code> flag to the call to <code>jacobian</code> and it will work.</p>
Pass all parameters to optimizer instead of only nonfrozen parameters

cosmosaa: <p>I’m curious what happens in this scenario. If we set <code>requires_grad = False</code> to some parameters, but accidentally pass these to the optimizer, are they skipped over or still optimized? As in the following:</p>
<pre><code class="lang-auto">model_conv = torchvision.models.resnet18(pretrained=True)
for param in model_conv.parameters():
    param.requires_grad = False

# Parameters of newly constructed modules have requires_grad=True by default
num_ftrs = model_conv.fc.in_features
model_conv.fc = nn.Linear(num_ftrs, 2)

model_conv = model_conv.to(device)

criterion = nn.CrossEntropyLoss()

optimizer_conv = optim.SGD(model_conv.parameters(), lr=0.001, momentum=0.9)
</code></pre>
<p>Note that <code>model_conv.parameters()</code> should actually be <code>model_conv.fc.parameters()</code>.</p>
<p>Are all of <code>model_conv</code> parameters optimized or only <code>model_conv.fc</code>?</p>

albanD: <p>Hi,</p>
<p>Assuming no gradients were computed for them before and their .grad field is None to begin with. Then the optimizer will just ignore them because they don’t have any gradient (as the backward won’t populate them).</p>
Accumulate the gradients

111433: <p>Hello. Nowadays I notice that importance of batch size when optimize the models.<br>
So I tried to push the envelope of memory limitation of CUDA.<br>
And followed the kind explanation<br>
<a href="https://discuss.pytorch.org/t/why-do-we-need-to-set-the-gradients-manually-to-zero-in-pytorch/4903/20">this post</a> .</p>
<p>For my case, I could train on 16 batch. But couldn’t 32 batch.<br>
So I accumulate gradient by backward and<br>
optimizer.step() when even numbers epoch.</p>
<p>But I encountered that<br>
File “train.py”, line 1065, in <br>
main()<br>
File “train.py”, line 641, in main<br>
multi_label=args.multi_label)<br>
File “train.py”, line 848, in train_epoch<br>
optimizer.step()<br>
File “/home/<strong>/anaconda3/envs/tresnet/lib/python3.6/site-packages/torch/autograd/grad_mode.py", line 49, in decorate_no_grad<br>
return func(<em>args, <strong>kwargs)<br>
File "/home/</strong></em>/adahessian.py", line 115, in step<br>
self.set_hessian()<br>
File "/home/</strong>/anaconda3/envs/tresnet/lib/python3.6/site-packages/torch/autograd/grad_mode.py”, line 49, in decorate_no_grad<br>
return func(*args, <strong>kwargs)<br>
File "/home/</strong>/adahessian.py", line 98, in set_hessian<br>
grads, params, grad_outputs=zs, only_inputs=True, retain_graph=i &lt; self.n_samples - 1)<br>
File “/home/**/anaconda3/envs/resnet/lib/python3.6/site-packages/torch/autograd/<strong>init</strong>.py”, line 157, in grad<br>
inputs, allow_unused)<br>
RuntimeError: CUDA out of memory. Tried to allocate 26.00 MiB (GPU 0; 10.76 GiB total capacity; 9.38 GiB already allocated; 16.56 MiB free; 9.77 GiB reserved in total by PyTorch)</p>
<p>The above is encountered at first time of optimizer.step().<br>
So I tried with 8 batch and optimizer.step() when multiples of 4 epochs.<br>
But nothing changed even decrease to 1 batch.</p>
<p>I wonder which allocate the  9.38 GiB already.<br>
Or I wonder It is problem with second order optimization.<br>
Would anybody help me?</p>

albanD: <p>You are using option 2 from there right?<br>
Do you see the memory usage increase at each iteration?</p>
<p>Also depending on which optimizer you use it might need to allocate a fairly big state. You need to make sure that 1step + optim works fine before doing more steps.</p>
Getting the associated input of each of the trained embedding matrix rows

leoentersthevoid: <p>Hi,</p>
<p>I trained and saved a model that has two embedding layers.<br>
One had a vocab V of ~ 22k items, that were encoded using a latent dimension D of 8.</p>
<p>If this is my trained embedding matrix (V x D)</p>
<pre><code class="lang-auto">tensor([[ 0.6163,  0.9769,  0.3950,  ...,  1.2966,  0.3279,  1.4990],
        [-0.4985, -0.6366,  0.6025,  ...,  0.1584, -0.9755, -0.7621],
        [-0.7265,  0.0665, -1.8310,  ..., -0.4401,  0.8690, -0.7261],
        ...,
        [-0.5039,  0.8430, -0.7346,  ..., -0.1686, -0.0024, -0.9600],
        [ 0.9195,  0.3476,  0.0367,  ..., -0.9595,  0.1659,  1.1200],
        [ 1.1634, -0.1817, -1.1437,  ..., -1.2055, -0.5795, -1.6404]])
</code></pre>
<p>is there a way to map each of the rows to its original input (which is an integer)?</p>
<p>I am thinking that the embedding matrix is one-to-one to the initial size of the vocab, but I don’t know how are they mapped because the <code>num_embeddings</code>  param asks for the length of the vocab only. So if the encoding is done with something like a dictionary, then what is the first key? Is it 0 or 1, and if I would plug into that dictionary the key 0 would I get the first row of the embedding matrix?</p>

ptrblck: <p>The input index is directly used to index the weight matrix as seen here:</p>
<pre><code class="lang-python">emb = nn.Embedding(4, 2)
x = torch.arange(4)
print(emb.weight)
&gt; Parameter containing:
tensor([[-0.5812,  0.8916],
        [ 1.9390, -0.6156],
        [ 0.2126,  0.0578],
        [-1.1099,  0.0964]], requires_grad=True)

out = emb(x)
print(out)
&gt; tensor([[-0.5812,  0.8916],
          [ 1.9390, -0.6156],
          [ 0.2126,  0.0578],
          [-1.1099,  0.0964]], grad_fn=&lt;EmbeddingBackward&gt;)
</code></pre>

leoentersthevoid: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>, thank you, this was my intuition but I wasn’t completely sure that this is right. However, I assumed it is so and ran a test that confirmed it empirically.</p>
what is the reason this eror

mohammadreza: <hr>
<p>UnboundLocalError                         Traceback (most recent call last)<br>
 in ()<br>
2   inputs, labels = data<br>
3   optimizer.zero_grad()<br>
----&gt; 4   outputs = net(inputs)<br>
5   loss = criterion(outputs, labels)<br>
6   loss.backward()</p>
<p>1 frames<br>
 in forward(self, X)<br>
9     self.fc3 = nn.Linear(84,10)<br>
10   def forward(self,X):<br>
—&gt; 11     x =  self.pool(F.relu(self.conv1(x)))<br>
12     X = self.pool(F.relu(self.conv2(x)))<br>
13     X = X.view(-1,16<em>5</em>5)</p>
<p>UnboundLocalError: local variable ‘x’ referenced before assignment</p>

utkuumetin: <p>Can you give full code? I think error due to variable assignment</p>

ptrblck: <p>It seems you are passing <code>X</code> (uppercase) to <code>forward</code>, while using <code>x</code> (lowercase) in the modules.</p>

mohammadreza: <p>yes exactly what was the problem. it was solved with your help bro.thanks alot</p>

mohammadreza: <p>thanks alot bro it was solved</p>