akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How parametric Relu implemented internally

username: dhakshiin
cooked: <p>i am trying to multiply the input vector of size (4,5,32,32)<br>
here<br>
4 - batch  size<br>
5 - channels<br>
32 x 32 feature size</p>
<p>As mentioned in the parametric channelwise relu  i want to multiply a vector to each of the channels</p>

username: ptrblck
cooked: <p>Please don’t tag certain people specifically. They will answer, if they have time.<br>
Also, this might discourage other users to post an answer.</p>

username: dhakshiin
cooked: <p>Sure. Can u pls provide suggestion for my question</p>

username: ptrblck
cooked: <p>Could you explain the parametric channelwise ReLU?<br>
How would you like to multiply the input of <code>[4, 5, 32, 32]</code> with a tensor, i.e. what shape would the internal tensor have?</p>

username: dhakshiin
cooked: <p>The internal tensor will be of 1x channel number. For 5 channel input we would have 1x5 as the internal tensor.</p>
<p>For [4 , 5, 32, 32]<br>
We need to multiply 1 value for each of the [32,32] in each batch.<br>
So 5 different values are used for each of the 5 channels in 32 x 32</p>
Freezing weights for some layers, for selective elements of a batch?

username: melodyguan
cooked: <p>Does anyone know how to freeze weights at a deep layer (to prevent backpropagation), for only some elements of a batch? I want all elements to still backpropagate through the final layer, and some but not all to backprop beyond that.<br>
I know how to selectively freezing some elements of a batch (with a mask), as well as how to freeze a deep layer (deep_layer.requires_grad = False), but I can’t think of a way to do both simultaneously. Thanks a lot!</p>
Same weight init, learning_rate, data_order but different gradient

username: Juna
cooked: <p>Hi!!<br>
I’m doing some experiments and I found that even though I fixed starting weight(I used pre-initialized weight), learning_rate and data order(I set shuffle option to False), the gradient changed when I try this two times! I got gradients of my model throughout 5 epochs and I did this for two times. Then I compared those gradients each other. They get further and further as epoch gets bigger.</p>
<p>Is there any other way that gradients can be changed?</p>

username: spanev
cooked: <p>Hi,</p>
<p>Are you running your model on the GPU? Some op used by cuDNN are non-deterministic  (especially backward accumulation with <code>atomicAdd</code> with floating points).</p>
<p>Please find more info about it <a href="https://pytorch.org/docs/stable/notes/randomness.html#cudnn" rel="nofollow noopener">here</a></p>

username: Juna
cooked: <p>Oh wow thanks for quick replying LOL.<br>
I found out that I have to do all three things in the link to make gradients same.<br>
Thanks so much for the information!!<br>
Have a good one!</p>
Stagnant Loss and Output in Multi-Class classification

username: Krish
cooked: <p>I have a multi-class classification problem but when I train it, the loss becomes stagnant after few epochs at <code>1.7918</code>.<br>
The output is:</p>
<pre><code class="lang-auto">[[-0.1240, -0.1240, -0.1240, -0.1240, -0.1240, -0.1240]]
</code></pre>
<p>for every class and doesn’t change a bit after that.<br>
The model:</p>
<pre><code class="lang-auto"># Feature Encoder 
class Enc(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(14, 12)
        self.fc2 = nn.Linear(12, 10)
        self.fc3 = nn.Linear(10, 8)
        self.fc4 = nn.Linear(8, 6)
        
    
    def forward(self, x):
        fn = nn.ReLU()

        hidden = fn(self.fc2(fn(self.fc1(x))))
        out = self.fc4(fn(self.fc3(hidden)))
        return out

model = Enc().to(device)
</code></pre>
<p>The training loop:</p>
<pre><code class="lang-auto">criterion = nn.CrossEntropyLoss().to(device)
optimizer = torch.optim.SGD(model.parameters(), lr = 0.01, momentum = 0.09)

for epoch in range(10000):
    inputs, labels = data
    optimizer.zero_grad()
    outputs = model(inputs)
    loss = criterion(outputs, labels)
    loss.backward()
    optimizer.step()
    if epoch%1999 == 1:
        print(loss) 
</code></pre>
<p>What should I do?</p>
LARS LAMB optimizers

username: dejanbatanjac
cooked: <p>I searched inside <a href="https://pytorch.org/docs/master/optim.html" rel="nofollow noopener">https://pytorch.org/docs/master/optim.html</a> but haven’t found any Layer-wise AdaptiveRate Scaling optimizer <a href="https://arxiv.org/pdf/1708.03888.pdf" rel="nofollow noopener">LARS </a> implemented. Same for LAMB. Are there any plans soon? Should we add FR?</p>

username: spanev
cooked: <p>Hi,<br>
LARC is not available directly in PyTorch but it is in Apex <a href="https://github.com/NVIDIA/apex/blob/master/apex/parallel/LARC.py" rel="nofollow noopener">https://github.com/NVIDIA/apex/blob/master/apex/parallel/LARC.py</a></p>
How to create custom dataset excluding a list of images

username: mohamadal
cooked: <p>Hi,<br>
I am trying to create a custom dataset using the structure of ImageFolder, where the images that appear in a list are excluded from the dataset. I have modified the ImageFolder class (its getitem) to output the path to each image in the folder, and then have been trying to set an if clause that checks if each image path is in my exclusion list, otherwise to include in the output dataset. However, I am not super comfortable with the init and getitem methods and have been stuck and wanted to know what I should change in ImageFolder to implement this.<br>
BTW, my use case is active learning to iteratively select a list of images out of a large pool, use them for retraining, and exclude them from the pool in the next round of candidate selection.<br>
Thanks in advance for your help.</p>
Out of Memory in Multi GPU -- Model Parallelization

username: Animesh_Kumar_Paul
cooked: <p>I am facing an error “Out of memory” when trying to load 100 MySmallModel in “cuda:0”.</p>
<p>I have two GPU with 12GB.</p>
<blockquote>
<p><strong>RuntimeError: CUDA out of memory. Tried to allocate 28.00 MiB (GPU 0; 11.91 GiB total capacity; 11.18 GiB already allocated; 19.38 MiB free; 34.67 MiB cached)</strong></p>
</blockquote>
<p>Can any please help me how can I solve this problem?/ How can I divide “self.fc1 = nn.Linear(70000, 3000)” layer into multiple layers and then do the computation?</p>
<pre><code class="lang-auto">class MySmallModel(nn.Module):
    def __init__(self,nodes):
        super(MySmallModel, self).__init__()
            self.fc1 = nn.Linear(70000, 3000)
            self.fc3 = nn.Linear(3000, 1000)
        
        self.fc1.cuda(0)
        self.fc3.cuda(1)
        
    def forward(self, x):
        
        x = F.relu(self.fc1(x))
        x = x.cuda(1)
        x = F.relu(self.fc3(x))

        return x

class Classifier(nn.Module):
    def __init__(self,input_nodes):
        super(Classifier, self).__init__()
        
        self.networks = nn.ModuleList([MySmallModel() for i in range(100)])

        self.sharedlayer = nn.Sequential(
            nn.Linear(30000, 300),
            nn.ReLU(),
            nn.Linear(300, 100),
            nn.ReLU(),
        )
        self.sharedlayer.cuda(1)
   
    def forward(self, input_):
        x_list=[]
        x_list.append(F.relu(self.networks[i](tensor_Data)))
        x = torch.cat((x_list), 1)
        
        x = x.cuda(1)
        h_shared = self.sharedlayer(x)
        
        return h_shared 

================================================
criterion = nn.MSELoss()
model = Classifier(input_nodes)
optimizer = optim.SGD(model.parameters(), lr=0.01)

trainloader = torch.utils.data.DataLoader(NN_data_train, batch_size=1, shuffle=True)
for epoch in range(n_epochs):
	running_loss = 0
	i = 0
	model.train()
	for data, label in trainloader:
		 out1 = model(data.cuda(0))
		 output = torch.cat([out1], 1)
		 loss = criterion(output.cuda(1), label.cuda(1))
		 optimizer.zero_grad()
		 loss.backward()
		 optimizer.step()
</code></pre>
<p>I have already reduce the batch size to 1. But still same memory problem.<br>
Please help.</p>
CTC loss vs NLL/CrossEntropy performance differences

username: tasubo
cooked: <p>Hi all,</p>
<p>I am having trouble finding research on models that used CrossEntropy vs CTC and their performance. It seems that if you have some kind of seq2seq task, it makes a lot of sense to use CTC but I would like to see what kind of difference I can expect. Any hints are welcome.</p>
<p>Thanks</p>
Why does the prebuilt pytorch for macos only use one core?

username: Jonas_Bulow
cooked: <p>Hi,</p>
<p>I’m new to pytorch. Why does the prebuilt pytorch for macos only use one core when using CPU? Is there a documented way to get a better performing build when only CPU is available.</p>
<p>torch.<strong>config</strong>.show() reports:</p>
<p>‘PyTorch built with:\n  - GCC 4.2\n  - clang 9.0.0\n  - Intel® MKL-DNN v0.18.1 (Git Hash 7de7e5d02bf687f971e7668963649728356e0c20)\n  - NNPACK is enabled\n  - Build settings: BLAS=MKL, BUILD_NAMEDTENSOR=OFF, BUILD_TYPE=Release, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -Wno-deprecated-declarations -DUSE_FBGEMM -DUSE_QNNPACK -O2 -fPIC -Wno-narrowing -Wall -Wextra -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -Wno-invalid-partial-specialization -Wno-typedef-redefinition -Wno-unknown-warning-option -Wno-unused-private-field -Wno-inconsistent-missing-override -Wno-aligned-allocation-unavailable -Wno-c++14-extensions -Wno-constexpr-not-const -Wno-missing-braces -Qunused-arguments -fcolor-diagnostics -faligned-new -fno-math-errno -fno-trapping-math -Wno-unused-private-field -Wno-missing-braces -Wno-c++14-extensions -Wno-constexpr-not-const, DISABLE_NUMA=1, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, USE_CUDA=False, USE_EIGEN_FOR_BLAS=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=OFF, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=OFF, USE_NNPACK=ON, USE_OPENMP=OFF, \n’</p>
<p>I installed pytorch in a virtual environment using pip.</p>
Conv based resnet

username: slavavs
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/4/42d91dbc44b8d21a366c956439e52b9acb206b17.png" data-download-href="https://discuss.pytorch.org/uploads/default/42d91dbc44b8d21a366c956439e52b9acb206b17" title="resnet-neural-e1548772388921"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/4/42d91dbc44b8d21a366c956439e52b9acb206b17_2_690x387.png" alt="resnet-neural-e1548772388921" data-base62-sha1="9xmGiCFG4jWGtnXDRLiAt4xfsKH" width="690" height="387" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/4/42d91dbc44b8d21a366c956439e52b9acb206b17_2_690x387.png, https://discuss.pytorch.org/uploads/default/optimized/2X/4/42d91dbc44b8d21a366c956439e52b9acb206b17_2_1035x580.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/4/42d91dbc44b8d21a366c956439e52b9acb206b17.png 2x" data-dominant-color="EFEFEF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">resnet-neural-e1548772388921</span><span class="informations">1130×635 31.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>How can I implement resnet as in the picture? How to add X to the next layer?</p>

username: slavavs
cooked: <p>It seems that this can be done using pooling.</p>

username: ptrblck
cooked: <p>Have a look at the <a href="https://github.com/pytorch/vision/blob/19315e313511fead3597e23075552255d07fcb2a/torchvision/models/resnet.py#L69" rel="nofollow noopener">resnet implementation</a> in <code>torchvision</code>.<br>
In particular the definition of <code>BasicBlock</code> and <code>Bottleneck</code> might be helpful.</p>
Keyword arguments in torch.nn.Sequential

username: cherepanovic
cooked: <p>Hello at all!</p>
<p>a question regarding keywords in torch.nn.Sequential, it is possible in some way to forward keywords to specific models in a sequence?</p>
<pre><code class="lang-auto">    model = torch.nn.Sequential(model_0, MaxPoolingChannel(1))
    res = model(input_ids_2, keyword_test=mask)
</code></pre>
<p>here, keyword_test should be forwarded only to the first model.</p>
<p>Thank a lot and best regards!</p>

username: albanD
cooked: <p>Hi,</p>
<p>You can pass any single input to the sequential. So you can pass a tuple or dict if you want and unpack them in your first model.<br>
The implementation is simple and can be found <a href="https://github.com/pytorch/pytorch/blob/59094c409ea552001d0f5356df9fdb0086ccee12/torch/nn/modules/container.py#L90-L93" rel="nofollow noopener">here</a> if you want to see the forward method of the Sequential</p>
Pytorch version conflict on Colab

username: LaleR
cooked: <p>Dear all,</p>
<p>I tried to run PyTorch transfer learning codes on Colab for my project. The codes need Pytorch 0.3.1 and torchvision 0.2.0. I have uninstalled the current version of PyTorch and try to install the needed version via:<br>
!pip3 install torch==0.3.1 torchvision==0.2.0 -f <a href="https://download.pytorch.org/whl/torch_stable.html" rel="nofollow noopener">https://download.pytorch.org/whl/torch_stable.html</a></p>
<p>however, the Cuda did not install. I have searched for several days to find the solution, but unfortunately, I did not get anything useful. Would somebody please help me with that? Thank you so much in advance.</p>
How to write a loss function with mask?

username: yangxi
cooked: <p>What I want is fairly simple: a MSE loss function, but able to mask some items:</p>
<pre><code class="lang-auto">def masked_mse_loss(a, b, mask):
    sum2 = 0.0
    num = 0
    for i in len(range(a)):
        if mask[i] == 1:
            sum2 += (a[i] - b[i]) ** 2.0
            num += 1
    return sum2 / num
</code></pre>
<p>Due to backward issue, I believe such a straightforward implementation would not make pytorch happy, but I have no idea on how to make it in correct way.</p>

username: Oli
cooked: <p>I was under the belief that pytorch would accept this kind of loss function. What error does it give you?</p>

username: yangxi
cooked: <p>I wrote the loss class:</p>
<pre><code class="lang-auto">class MaskedMSELoss(torch.nn.Module):
    def __init__(self):
        super(MaskedMSELoss, self).__init__()

    def forward(self, input, target, mask):
        diff2 = (torch.flatten(input) - torch.flatten(target)) ** 2.0
        sum2 = 0.0
        num = 0

        flat_mask = torch.flatten(mask)
        assert(len(flat_mask) == len(diff2))
        for i in range(len(diff2)):
            if flat_mask[i] == 1:
                sum2 += diff2[i]
                num += 1

        return sum2 / num
</code></pre>
<p>What would be default backward looks like?</p>

username: zhl515
cooked: <p>You can call out.backward() to backpropagate the error.</p>
<pre><code class="lang-auto">predict = torch.tensor([1.0, 2, 3, 4], dtype=torch.float64, requires_grad=True)
target = torch.tensor([1.0, 1, 1, 1], dtype=torch.float64,  requires_grad=True)
mask = torch.tensor([1, 0, 0, 1], dtype=torch.float64, requires_grad=True)
out = torch.sum(((predict-target)*mask)**2.0)  / torch.sum(mask)
out.backward()
</code></pre>

username: yangxi
cooked: <p>I modified the algorithm to a more simple form that removed for cycle and if, and it works properly:</p>
<pre><code class="lang-auto">diff2 = (torch.flatten(input) - torch.flatten(target)) ** 2.0 * torch.flatten(mask)
result = torch.sum(diff2) / torch.sum(mask)
return result
</code></pre>
<p>Thanks!</p>
Using optimizer for word_language_model example

username: pooya_khandel
cooked: <p>Is there any necessary reason PyTorch word language model doesn’t use any optimizer for training the model?<br>
The weights in the source example are updated like:</p>
<pre><code class="lang-auto">        for p in model.parameters():
            p.data.add_(-lr, p.grad.data)
</code></pre>
<p>I have tried different optimizers such as Adam, RMSprop,… with different learning rate for updating the weights instead, but all of them result in high loss and perplexity. They can not help the model train better.</p>
AttributeError: module 'torch' has no attribute 'where

username: solsol
cooked: <p>Hi.</p>
<p>I used this line in my main pytorch code:</p>
<p>torch.where(F.softmax(presence_vector.view(-1), dim=0) &gt;= (1/len(m_vector))</p>
<p>I faced this error: AttributeError: module ‘torch’ has no attribute ‘where’.<br>
'torch version is: ‘0.3.1’</p>
<p>Is there any help?<br>
Thanks</p>

username: spanev
cooked: <p>Hi <a class="mention" href="/u/solsol">@solsol</a></p>
<p><code>torch.where</code> was added in v0.4.0 <a href="https://github.com/pytorch/pytorch/releases/tag/v0.4.0" class="inline-onebox" rel="noopener nofollow ugc">Release Trade-off memory for compute, Windows support, 24 distributions with cdf, variance etc., dtypes, zero-dimensional Tensors, Tensor-Variable merge, , faster distributed, perf and bug fixes, CuDNN 7.1 · pytorch/pytorch · GitHub</a></p>
<aside class="quote" data-post="8" data-topic="1329">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pete_florence/48/3338_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-can-i-do-the-operation-the-same-as-np-where/1329/8">How can I do the operation the same as `np.where`?</a> 
  </div>
  <blockquote>
    <a class="mention" href="/u/truenicoco">@truenicoco</a>, torch.where() looks like it’'s in 0.4! <a href="http://pytorch.org/docs/master/torch.html#torch.where" rel="noopener nofollow ugc">http://pytorch.org/docs/master/torch.html#torch.where</a> 
thanks others for examples of other options for 0.3 and before
  </blockquote>
</aside>
Is pytorch 1.2 compatible with cudatoolkit 9.0？

username: RomanShen
cooked: <p>Hello, I’m trying to upgrade to pytorch 1.2. I’ve found that official guidance seems to need cudatoolkit version to be 9.2 or higher. Is it possible to use pytorch 1.2 with cudatoolkit 9.0? Thanks for any help!</p>

username: ptrblck
cooked: <p>You should be able to compile PyTorch using CUDA9.0.<br>
However, the binaries dropped the support for CUDA8.0 and CUDA9.0 in the latest release.</p>

username: RomanShen
cooked: <p>Thank you! Sorry for the late reply. I have tried to build pytorch 1.2 from source following the instruction on the official github website. I came across the following problem: “ninja: build stopped: subcommand failed.” I can’t provide the detailed error report now because our server is in the progress of update. When it’s done, I’ll try it again. Thanks for your help!</p>
Using eval() when I want to use the net as a loss

username: rmokady
cooked: <p>Hi All,<br>
I want to use a net as part of loss function, but the problem is that the net outputs different values when I use eval() and train(), it actually outputs much better when I use eval().<br>
But, when I use eval(), I get the following error when I try to use the net as part of a loss:<br>
<code>RuntimeError: cudnn RNN backward can only be called in training mode</code></p>
<p>So I want to get the same net output as using eval() and also avoid the mentioned error.<br>
Any help will be appreciated.</p>

username: rmokady
cooked: <p>I found some workaround. declare net.train() then for any Dropout and BatchNorm layers:</p>
<pre><code>    net.dropout.p = 0
    net.batchNorm.training = False
</code></pre>
<p>It seems to work.</p>
Did I misunderstand data.DataSet.__getitem__()?

username: 11130
cooked: <p>This is very weird. I have loaded CIFAR10 dataset:</p>
<pre><code class="lang-auto">import torchvision.datasets as datasets
import torchvision.transforms as transforms
dataloader = datasets.CIFAR10
transform_train = transforms.Compose([
    transforms.RandomCrop(32, padding=4),
    transforms.RandomHorizontalFlip(),
    transforms.ToTensor(),
    transforms.Normalize((0.4914, 0.4822, 0.4465), (0.2023, 0.1994, 0.2010)),
])
trainset = dataloader(root='./data', train=True, download=True, transform=transform_train)
</code></pre>
<p>Then I find the image returned by trainset.<strong>getitem</strong> and the one accessed directly are different.</p>
<p>The <strong>getitem</strong> in class CIFAR10(data.Dataset) says</p>
<pre><code class="lang-auto">    def __getitem__(self, index):
        img, target = self.data[index], self.targets[index]

        # doing this so that it is consistent with all other datasets
        # to return a PIL Image
        img = Image.fromarray(img)

        if self.transform is not None:
            img = self.transform(img)

        if self.target_transform is not None:
            target = self.target_transform(target)

        return img, target

</code></pre>
<p>But if I implement <strong>getitem</strong>() directly and run this</p>
<pre><code class="lang-auto">by_get_item = trainset[0][0]
directly = trainset.transform(Image.fromarray(trainset.data[0]))
print(torch.sum(torch.abs(
    by_get_item - directly
)))
</code></pre>
<p>it prints a non-zero and the value keeps changing. Why does this happen?</p>

username: ptrblck
cooked: <p>Since you are using random transformations, you’ll get randomly transformed samples every time.<br>
Remove <code>RandomHorizontalFlip</code> and <code>RandomCrop</code> and you should get the same outputs.</p>

username: 11130
cooked: <p>Thanks a lot. I’m not familiar with torchvision transforms</p>
Torch.jit isuue with submodules(Resnet)

username: kcw4875
cooked: <p>I created a model that uses two Resnets(pre-trained) as feature extractors<br>
In my model I used a pre-trained Resnet as a sub module.</p>
<p>Training and testing work well in pytorch, but<br>
If you use simple torch.jit.trace while converting torch, the following error occurs.</p>
<p><strong>could not export python function call Scatter. Remove calls to Python functions before export. Did you forget add <span class="mention">@script</span> or <span class="mention">@script_method</span> annotation? If this is a nn.ModuleList, add it to  <strong>constants</strong> .:</strong></p>
<p>if i want to use sub_module, it looks like i need to use trace and script together, as shown in the code below from the official pytorch homepage.<br>
##################################</p>
<p>import torch<br>
import torchvision<br>
class MyScriptModule(torch.nn.Module):<br>
def <strong>init</strong>(self):<br>
super(MyScriptModule, self).<strong>init</strong>()<br>
self.means = torch.nn.Parameter(torch.tensor([103.939, 116.779, 123.68])<br>
.resize_(1, 3, 1, 1))<br>
self.resnet = torch.jit.trace(torchvision.models.resnet18(),<br>
torch.rand(1, 3, 224, 224))<br>
def forward(self, input):<br>
return self.resnet(input - self.means)<br>
my_script_module = torch.jit.script(MyScriptModule())</p>
<p>########################################################</p>
<p>but, This also causes an error.</p>
<p>Please advise if this approach is correct or if you can make a simple trace.</p>
<p>plz…</p>

username: driazati
cooked: <p>The code snippet from the website you posted works fine for me (on PyTorch 1.2, are you up to date?), can you post a full code example that results in the <code>could not export Python function..</code> error?</p>
Is `setattr` something we need when creating custom NN with changing layers?

username: Brando_Miranda
cooked: <p>I was trying make a model that built a model with many layers depending on how many the user had. I saw somewhere (<a href="https://github.com/MorvanZhou/PyTorch-Tutorial/blob/master/tutorial-contents/504_batch_normalization.py" rel="nofollow noopener">https://github.com/MorvanZhou/PyTorch-Tutorial/blob/master/tutorial-contents/504_batch_normalization.py</a>) that <code>setattr</code> can be used. Is it stricly necessary?</p>
<p>for example:</p>
<pre><code class="lang-auto">class MyNet(nn.Module):
    ## 5 conv net FC
    def __init__(self,C,H,W, Fs, Ks, FC):
        super(MyNet, self).__init__()
        self.nb_conv_layers = len(Fs)
        ''' Initialize Conv layers '''
        self.convs = []
        out = Variable(torch.FloatTensor(1, C,H,W))
        in_channels = C
        for i in range(self.nb_conv_layers):
            F,K = Fs[i], Ks[i]
            conv = nn.Conv2d(in_channels,F,K) #(in_channels, out_channels, kernel_size)
            self.convs.append(conv)
            ##
            in_channels = F
            out = conv(out)
        ''' Initialize FC layers'''
        CHW = out.numel()
        self.fc = nn.Linear(CHW,FC)
</code></pre>
<p>vs</p>
<pre><code class="lang-auto">class MyNet(nn.Module):
    ## 5 conv net FC
    def __init__(self,C,H,W, Fs, Ks, FC):
        super(MyNet, self).__init__()
        self.nb_conv_layers = len(Fs)
        ''' Initialize Conv layers '''
        self.convs = []
        out = Variable(torch.FloatTensor(1, C,H,W))
        in_channels = C
        for i in range(self.nb_conv_layers):
            F,K = Fs[i], Ks[i]
            conv = nn.Conv2d(in_channels,F,K) #(in_channels, out_channels, kernel_size)
            setattr(self,f'conv{i}',conv)
            self._set_init(conv)
            self.convs.append(conv)
            ##
            in_channels = F
            out = conv(out)
        ''' Initialize FC layers'''
        CHW = out.numel()
        self.fc = nn.Linear(CHW,FC)
</code></pre>

username: jpeg729
cooked: <p>Any submodule that isn’t declared as a direct attribute of the class, will not be registered as a submodule of the model and its parameters won’t show up in model.parameters() so it won’t get trained.</p>
<p>For example, if you do this…</p>
<pre><code class="lang-auto">self.convs = []
self.convs.append(conv)
</code></pre>
<p>then your model doesn’t know that it has <code>conv</code> as a submodule.</p>
<p>I know of three possible fixes…</p>
<ol>
<li>use <code>settattr()</code> as you pointed out.</li>
<li><code>self.add_module(f'conv{i}', conv)</code></li>
<li>replace <code>self.convs = []</code> with <code>self.convs = nn.ModuleList()</code>. Then treat self.convs like an ordinary python list and anything you append to it will be properly registered.</li>
</ol>
<p>The third solution is the most elegant in my opinion.</p>

username: lonelykid
cooked: <p>But with the third solution, you won’t be able to assign a name to a layer</p>
How to get pre-trained 3D model?

username: piaoliusihai1
cooked: <p>If I’m working on video or medical images, 3D model is a better choice.<br>
Then here comes a question? How can I get a pre-trained model such like Densenet, Resnet on ImageNet?<br>
Looking forward for your reply!</p>

username: thadar
cooked: <p>Hi,</p>
<p>You can check this repository on github:<br>
<a href="https://github.com/kenshohara/3D-ResNets-PyTorch" rel="nofollow noopener">https://github.com/kenshohara/3D-ResNets-PyTorch</a></p>
<p>Most of the models there are pretrained on Kinetics dataset.</p>
<p>Hope this helps,<br>
Tal</p>

username: piaoliusihai1
cooked: <p>Thank you very much!</p>

username: nabsabs
cooked: <p>hi <a class="mention" href="/u/piaoliusihai1">@piaoliusihai1</a>, just curious if you found these models pretrained on kinetics useful for medical stuff!</p>
Building a Modular Model: State dict vs Chaining Models

username: mgpoirot
cooked: <p>Hi all,</p>
<p>I’m working on a distributed learning system where I’m splitting a big model into smaller parts.<br>
I call this big network a chain consisting of smaller networks called links.<br>
The use case is that the links can be a client in a distributed learning system.</p>
<p>Currently I’m initializing separate models, and forwarding the output of each link to the next link in the chain, this works well.</p>
<p>When I would like to replace a link in the chain for another, I’m currently simply calling a different model. For implementation sake, it would be easier to just replace the state dict of several layers in the chain by the state dict of another link.</p>
<p><strong>My question: Does the state dict of layers encompass all the layers properties/attributes/links?</strong><br>
(I’m familiar with most, weights, biases, gradients, but I’ve switched to PyTorch quite recently, and would not know what other information might stay linked.)</p>
Batching Smoothgrad

username: DDong
cooked: <p>I’m trying to adjust <a href="https://github.com/sar-gupta/convisualize_nb/blob/master/cnn-visualize.ipynb" rel="nofollow noopener">this</a> implementation of Smoothgrad to operate on batches in order to speed things up. I changed the part of the function smooth_grad where it computes the gradients to:<br>
<code><br>
global model, transform<br>
tensor_input = torch.from_numpy(np.array(input)).type(torch.cuda.FloatTensor) # input is now of shape (w,h,c)<br>
# x is the sample size<br>
final_grad=[]<br>
calc_inputs=[]<br>
for i in range(x):<br>
<span class="hashtag">#print</span>(‘Sample:’, i+1)<br>
temp_input = tensor_input<br>
# According to the paper, noise level corrresponds to stddev/(xmax-xmin). Hence stddev = noise_percentage * (max-min) /100<br>
noise = torch.from_numpy(np.random.normal(loc=0, scale=(percent_noise/100) * (tensor_input.max() - tensor_input.min()).cpu(), size=temp_input.shape)).type(torch.cuda.FloatTensor)<br>
temp_input = (temp_input + noise).cpu().numpy()<br>
temp_input = Image.fromarray(temp_input.astype(np.uint8))<br>
temp_input = Variable(transform(temp_input).unsqueeze(0).cuda(), requires_grad=True)<br>
calc_inputs.append(temp_input.squeeze())<br>
tensor_calc=Variable(torch.stack([i for i in calc_inputs]),requires_grad=True)<br>
tensor_y=Variable(torch.tensor([label for i in range(x)]))<br>
dataset = torch.utils.data.TensorDataset(tensor_calc,tensor_y)<br>
loader = torch.utils.data.DataLoader(dataset, batch_size=50)<br>
for imgs, labels in loader:<br>
model.train()<br>
output = model(imgs)<br>
output.backward(torch.ones(50,1221).cuda())<br>
print(imgs.grad)<br>
<span class="hashtag">#final_grad</span>.append(imgs.grad))<br>
final_grad=sum(final_grad)<br>
</code><br>
But imgs.grad prints None. What am I doing wrong?</p>
How to fill an empty sparse tensor?

username: pg2455
cooked: <p>I am initiating a sparse tensor using <code>x = torch.sparse.FloatTensor(100,100,3)</code>. I want to fill it up later at different times. How can I do so? I am using this as a representation of a growing graph.</p>
<p>Doing <code>x[1,1,:] = torch.Tensor([1,2,3])</code> throws <code>RuntimeError: sparse tensors do not have strides</code>.<br>
Or just doing <code>x[1]</code> returns the same error.</p>

username: zhl515
cooked: <p>A sparse tensor in pytorch is represented as a pair of dense tensors: a tensor of values and a 2D tensor of indices. sparse tensor don’t have strides,  the directed indexing operation such as x[1] is not allow. torch.sparse module is  currently experimental,  maybe using dense tensor  is preferred</p>
<pre><code class="lang-auto">torch.randn(2, 3).to_sparse()
"""
tensor(indices=tensor([[0, 0, 0, 1, 1, 1],
                       [0, 1, 2, 0, 1, 2]]),
       values=tensor([ 0.5527, -0.7956,  0.7942,  0.6294,  0.9592,  0.3199]),
       size=(2, 3), nnz=6, layout=torch.sparse_coo)
"""
</code></pre>
how to get NLLLoss grad?

username: laizewei
cooked: <p>import torch<br>
import torch.nn as nn</p>
<h2>m = nn.LogSoftmax(dim=1)</h2>
<p>loss = nn.NLLLoss()<br>
a=[[2., 0.],<br>
[1., 1.]]<br>
input = torch.tensor(a, requires_grad=True)<br>
target = torch.tensor([1, 1])<br>
output = loss(m(input), target)<br>
output.backward()<br>
print(input.grad)</p>
<h2>tensor([[ 0.4404, -0.4404],</h2>
<p>[ 0.2500, -0.2500]])</p>
<p>How to get input.grad?What’s the formula？</p>

username: spanev
cooked: <p>Hi <a class="mention" href="/u/laizewei">@laizewei</a></p>
<p>I am not sure if you are asking what  input.grad represents or the full formula (which would require to write the full derivation of the network w/ the chain rule).</p>
<p><code>input.grad</code> gives you the gradients of <code>output</code> w.r.t. <code>input</code> (since you ran <code>output.backward()</code>).</p>
<p>Maybe <a href="https://pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html#gradients" rel="nofollow noopener">this tutorial</a> could help you gain some intuition about what is happening behind.</p>
<p>Hope this answers your question</p>

username: laizewei
cooked: <p>Thanks. Now I understand it.</p>
Unexpected output size when executing torch.squeeze() with multiple GPUs

username: Phoenix1327
cooked: <p>The example code is:</p>
<pre><code class="lang-auto"># input_size: N * 1024 * 1
output = torch.squeeze(input)
# output_size: N * 1024
</code></pre>
<p>where N is batch_size.</p>
<ul>
<li>When the code is running with single GPU mode, the output size is correct, i.e., N * 1024.</li>
<li>However, when using multiple GPUs (for example, 4 GPUs), the output size is wired:
<ol>
<li>When N is large, e.g. 64, the output size is correct.</li>
<li>When N is small, e.g. N=2, the output size will be 2048…</li>
</ol>
</li>
</ul>
<p>PS, when using <code>output = input.squeeze(-1)</code>, the output is always correct.</p>

username: ptrblck
cooked: <p><code>torch.squeeze</code> will squeeze all dimensions with size 1:</p>
<pre><code class="lang-python">x = torch.randn(1, 2, 1, 2, 1, 2, 1)
print(x.squeeze().shape)
&gt; torch.Size([2, 2, 2])
</code></pre>
<p>In the case of a small batch size, each device might get a chunk where <code>N==1</code>, which will also squeeze the batch dimension.<br>
To avoid such errors, I would recommend to use your second approach and to specify which dimension should be squeezed.</p>
Loaded images do have 3 channels despite using Grayscale transformation

username: talhak
cooked: <p>As the title clearly describes, the images in the dataset I use do have 3 color channels despite they are grayscale. So, I used <code>transforms.Grayscale(num_output_channels=1)</code> transformation in order to reduce the number of color channels to 1, but still the loaded images do have 3 channels.</p>
<p>Here is my implementation:</p>
<pre><code class="lang-auto">data_transforms = transforms.Compose([
    transforms.Grayscale(num_output_channels=1),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406],
                         std=[0.229, 0.224, 0.225])
])
face_train_dataset = datasets.ImageFolder(root=DEST_PATH_TRAIN, transform=data_transforms)

train_loader = DataLoader(face_train_dataset,
                          batch_size=train_batch_size, shuffle=True, num_workers=4)
</code></pre>

username: ptrblck
cooked: <p>If I run your code, I get the valid error:</p>
<pre><code class="lang-python">RuntimeError: output with shape [1, 334, 690] doesn't match the broadcast shape [3, 334, 690]
</code></pre>
<p>which is thrown, since <code>Grayscale</code> returns a single-channel image, while <code>Normalize</code> uses three values.<br>
Fixing the <code>mean</code> and <code>std</code> to a single value, yields valid output tensors of shape <code>[batch_size, 1, 334, 690]</code>.</p>

username: talhak
cooked: <p>Not that I was aware of, thanks <a class="mention" href="/u/ptrblck">@ptrblck</a>. Thought it could still work as it was expected.</p>

username: talhak
cooked: <p>Between, after decreasing the number of channels of the images, which were already grayscale (8 bits depth images), the accuracy of the network has decreased dramatically. The only change was that. Are there any workarounds or point that I miss? <a class="mention" href="/u/ptrblck">@ptrblck</a></p>

username: ptrblck
cooked: <p>Are you sure the depth images are not stored in 16bit?</p>
<p>Anyway, are you training your model from scratch or finetuning a pretrained model?<br>
Was the accuracy better when you loaded the depth images as 3-channel images?</p>

username: talhak
cooked: <p>Yes, here is the detail of an image (which is a sample image from the <a href="https://www.kaggle.com/c/challenges-in-representation-learning-facial-expression-recognition-challenge/data" rel="noopener nofollow ugc">FER2013 dataset</a>):</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/2X/6/6ecdcd804d7b50ba439b96a03c3efd71fb5802fd.jpeg" alt="depth" data-base62-sha1="fOds1Oj9uzdX43y0tXgvogKLknP" width="379" height="185"></p>
<p>I have constructed my model from scratch, now trying to improve its accuracy. Yes, the accuracy was way better when I loaded the images as 3-channels as the accuracy has dropped to about 25% from 50%.</p>
Purpose of specifying the devices in model_state

username: Valiox
cooked: <p>My model has its tensors on cuda, and its state is saved as such. When I load the model state for inference with <code>checkpoint = torch.load(weights_fpath)</code>, the model state keeps the same device configuration. This adds overhead to torch.load (in my case 1-2 seconds). Calling <code>checkpoint = torch.load(weights_fpath, "cpu")</code> takes around 1 millisecond however. I don’t really understand the point to all of this, because when I’ll call <code>model.load_state_dict(checkpoint["model_state"])</code> after, all my tensors are going to end up on cpu, regardless of the device specified in the model state.</p>
<ol>
<li>
<p>What’s the point of setting the device of the tensors in the model state, what’s the use of the map_location argument of torch.load?</p>
</li>
<li>
<p>Is there a way to directly load cuda tensors when doing <code>model.load_state_dict()</code> that would be faster than loading cpu tensors then calling <code>model.cuda()</code>?</p>
</li>
</ol>

username: ptrblck
cooked: <ol>
<li>The map_location argument remaps the tensors to the specified device. This is useful, if your current machine does not have a GPU and you are trying to load a tensor, which was saved as a CUDATensor.</li>
</ol>
<p>From the docs:</p>
<blockquote>
<p><a href="https://pytorch.org/docs/stable/torch.html?highlight=torch%20load#torch.load"> <code>torch.load()</code> </a> uses Python’s unpickling facilities but treats storages, which underlie tensors, specially. They are first deserialized on the CPU and are then moved to the device they were saved from. If this fails (e.g. because the run time system doesn’t have certain devices), an exception is raised. However, storages can be dynamically remapped to an alternative set of devices using the <code>map_location</code> argument.</p>
</blockquote>
<ol start="2">
<li>You could push the model to the GPU before loading the <code>state_dict</code>. However, the tensors have to be pushed somewhere to the device, so I guess you won’t save any time.</li>
</ol>
Accuracy of the precision and recall

username: Aiman_Mutasem-bellh
cooked: <p>The values of the accuracy, precision, and recall are close and I’m a little worried about these values. My question is these values are reliable? like the below:</p>
<p>Test Loss: 0.389 | Test accuracy: 0.919 | Test precision: 0.741 | Train recall: 0.742<br>
Test F1: 0.741</p>
<p>I calculated these values as :</p>
<p>def train(model, iterator, optimizer, criterion, clip):</p>
<pre><code>model.train()

epoch_loss = 0
epoch_acc = 0 
epoch_precision = 0 
epoch_recall = 0 

for i, batch in enumerate(iterator):
    
    src = batch.src
    trg = batch.trg
    
    optimizer.zero_grad()
    
    output, attention = model(src, trg[:,:-1])

    
    output = output.contiguous().view(-1, output.shape[-1])
    trg = trg[:,1:].contiguous().view(-1)       

    loss = criterion(output, trg)
    aa = acc(output, trg)
    precision = get_precision(output, trg)
    recall = get_recall(output, trg)
    
    loss.backward()
    
    torch.nn.utils.clip_grad_norm_(model.parameters(), clip)
    
    optimizer.step()
    
    epoch_loss += loss.item()
    epoch_acc +=  aa  
    epoch_precision += precision
    epoch_recall += recall 
    
return epoch_loss / len(iterator), epoch_acc / len(iterator), epoch_precision / len(iterator), epoch_recall / len(iterator)
</code></pre>
<p>N_EPOCHS = 100<br>
CLIP = 1</p>
<p>best_valid_loss = float(‘inf’)</p>
<p>for epoch in range(N_EPOCHS):</p>
<pre><code>start_time = time.time()

train_loss, train_acc, train_prce, train_recall  = train(model, train_iterator, optimizer, criterion, CLIP)
trainf1score = train_prce * train_recall * 2 / (train_prce + train_recall + 1e-20)
  
end_time = time.time()

epoch_mins, epoch_secs = epoch_time(start_time, end_time)

if valid_loss &lt; best_valid_loss:
    best_valid_loss = valid_loss
    torch.save(model.state_dict(), 'tut5-model.pt')

print(f'Epoch: {epoch+1:02} | Time: {epoch_mins}m {epoch_secs}s')
print(f'\tTrain Loss: {train_loss:.3f} | Train accuracy: {train_acc:.3f} | Train precision: {train_prce:.3f} | Train recall: {train_prce:.3f}')
print(f'\tTrain F1: {trainf1score:.3f}')</code></pre>
Graphs not showing in tensorboard Pytorch 1.2

username: John_Deterious
cooked: <p>I’m literally copying this example from documentations. Everything works (scalars, images etc), aside from graph tab, which shows nothing. Pytorch 1.2. Tensorboard 1.14<br>
Works like a charm in Pytorch 1.1</p>
<pre><code class="lang-auto">import torch
import torchvision
from torch.utils.tensorboard import SummaryWriter
from torchvision import datasets, transforms

# Writer will output to ./runs/ directory by default
writer = SummaryWriter()

transform = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.5,), (0.5,))])
trainset = datasets.MNIST('mnist_train', train=True, download=True, transform=transform)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=64, shuffle=True)
model = torchvision.models.resnet50(False)
# Have ResNet model take in grayscale rather than RGB
model.conv1 = torch.nn.Conv2d(1, 64, kernel_size=7, stride=2, padding=3, bias=False)
images, labels = next(iter(trainloader))

grid = torchvision.utils.make_grid(images)
writer.add_image('images', grid, 0)
writer.add_graph(model, images)
writer.close()
</code></pre>
Why do I get Resize is not callable?

username: Shisho_Sama
cooked: <p>I’m trying to create my own funcor for use in transformations, but it keeps failing with the error :<br>
<code>**TypeError** : 'Resize' object is not callable</code><br>
This my example :</p>
<pre><code class="lang-auto">import torchvision.transforms.functional as F
class Resize(object):
    def __init__(self, size, interpolation):
        self.size = size
        self.interpolation = interpolation

    def __cal__(self, input):
        return F.resize(input, self.size, self.interpolation)

to_tensor_f = ToTensor()
import PIL.Image as Image
resize_f = Resize(3, Image.BILINEAR)
numpy_tensor = np.random.rand(2, 3)
result_tensor = to_tensor_f(numpy_tensor)
print(result_tensor)
#resize this tensor!
x=resize_f(result_tensor)
print(x)
</code></pre>
<p>Thank you all in advance</p>

username: Shisho_Sama
cooked: <p>Got it. silly me mis typed <code>__call__</code> as <code>__cal__</code><br>
fixed it and all is good</p>
MPI Allreduce Queue

username: hpc-unex
cooked: <p>Hello,</p>
<p>What is the function/reason to <em>enqueue(std::move(entry));</em> in ProcessGroupMPI Allreduce? I don’t understand this part of the code</p>

username: pietern
cooked: <p>This is where the operation is queued to be executed by a background thread. All collective calls execute on a separate thread so you can block on their completion only when you need the result. This allows for overlapping of gradient reduction with gradient computation.</p>

username: hpc-unex
cooked: <p>When you talk about “overlapping of gradient reduction with gradient computation” you mean the overlapping between forward and backward propagation?</p>

username: pietern
cooked: <p>No, it is overlapped with gradient computation (backward propagation) only. As more and more gradients have been computed, they are ready to be reduced. There is no need to wait with reducing them until you have computed all of them.</p>

username: qiuyang
cooked: <p><a class="mention" href="/u/pietern">@pietern</a> Hi,could you tell ne why ProcessGroupNCCL not use enqueue(xxx) function？Does NCCL implement multiple nodes gradients calculation synchronization?</p>
Can I also get data indices using dataloader?

username: 11130
cooked: <p>Hello, everyone! In my project, I need to know the indices of the sampled data points in the training set. But I don’t know how to do this.</p>
<p>I tried an alternative below by setting the shuffle to False, in which I don’t need to know the data indices because the sampled data will simply has the same order as in the training set. But it seems the dataloader still reorders the samples.</p>
<pre><code class="lang-auto">sampler = torch.utils.data.DataLoader(
    trainset, batch_size = args.train_batch, shuffle = False, num_workers = args.workers
)
sampler = iter(sampler)
for i in range(len(trainloader)):
    inputs, targets =next(sampler)
    print(torch.sum(torch.abs(
        inputs[0]-trainset[i * args.train_batch][0]
    )))
</code></pre>
<p>If the dataloader has returned data exactly as the data’s order in the training set, this snippet should print 0s. But it prints non-zero values</p>

username: Oli
cooked: <p>Helooo <img src="https://discuss.pytorch.org/images/emoji/apple/sunny.png?v=9" title=":sunny:" class="emoji" alt=":sunny:"></p>
<p>The best way would be to create your own dataset class. In it you could return your images (or whatever data you have) <strong>plus</strong> your index for that data item. Try this and report back if you have a hard time creating your dataset <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: 11130
cooked: <p>Thank you for your help. This solves my problem <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=9" title=":smile:" class="emoji" alt=":smile:"></p>
Vl_nnconv Equivalent

username: DDong
cooked: <p>In Matlab there is a function called vl_nnconv, which returns either the results of a convolution for a forwards pass or the gradients with respect to the input, filters, and biases depending on the arguments. What is the closest equivalent in PyTorch, especially for the backwards version? I’m extending the Conv2d module in a way that isn’t differentiable (that is, it requires approximations to calculate practical gradients as it uses things like argmax).</p>

username: albanD
cooked: <p>Hi,</p>
<p>I’m not sure what you’re looking for?<br>
A way to implement a new operation with a custom backward?<br>
Or a way to access the backward of the regular convolution directly?</p>

username: DDong
cooked: <p>The end goal is to implement a new operation with a custom backward, but its new backward utilizes the backward of the regular convolution.</p>

username: albanD
cooked: <p>Hi,</p>
<p>You can find <a href="https://pytorch.org/docs/stable/notes/extending.html#extending-torch-autograd" rel="nofollow noopener">here</a> the instructions on how to build a new operation.<br>
I don’t think we provide direct access to the backward of a given convolution. But based on the parameters, you can simply use the corresponding <code>ConvTranspose</code> (with same parameters as long as you don’t do fancy stuff with dilated convolutions).</p>

username: DDong
cooked: <p>Thanks. Is it feasible to use forward and backward hooks to modify the forward and backward passes; that is, register a forward hook on a module that changes the output it sends to the next module, and register a backwards hook on the same module that changes the gradient it sends to the next module (the one preceding it in the forwards pass)? Or is creating a new module the way to do that?<br>
Reason for asking being that I am trying to do so by creating a new module and applying a function that applies autograd.Function as in the tutorial from the site you mentioned, but the grad_output it receives in the backward function is full of either zeros or infinities, depending on how I implement it.</p>
<p>More specifically, I just added some backwards hooks to check whether requires_grad was true or not, and for almost all modules it was False. The model is set to train(). I am adjusting the VGG16 model. Looking at the implementation, when the hook was set to self.features it returned a reasonable tensor as grad_output, but when I added it to a convolutional layer it returned all 0s with requires_grad equal to False.</p>