akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Build PyTorch android binaries with -O0?

username: solvingPuzzles
cooked: <p>I am trying to track down a bug that happens inside a pytorch android binary (speed_benchmark_torch) for one of my models.</p>
<p>When I look at the backtrace on android <code>logcat</code>, it looks like debug symbols are disabled, so I don’t know what line of code is failing. Also, I tried building with <code>-DANDROID_DEBUG_SYMBOLS=1</code>, and I still couldn’t get a stacktrace with debug symbols.</p>
<p>I suspect that I will get a better stacktrace if I can compile with <code>-O0</code> instead of the current <code>-O2</code>. Towards this end, I changed one of the lines in <code>pytorch/CMakeLists.txt</code> from this:<br>
<code>set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -O2 -fPIC")</code><br>
to this:<br>
<code>set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -O0 -fPIC")</code></p>
<p>I also changed the <code>O2</code> to <code>O0</code> in the QNNPACK CMakeLists.</p>
<p>And, after making that change, I recompiled pytorch. When I look at <code>pytorch/build/compile_commands.json</code>, I see we use -O0 in the pytorch build. Great.</p>
<p>Now, I run <code>scripts/build_android.sh</code>, and to my dismay I see that each command in <code>build_android/compile_commands.json</code> has both -O0 and -O2. That’s not good!</p>
<p>Does anyone know…<br>
(a) Where might this stray -O2 be coming from, and<br>
(b) How do I do an “only O0” build of the android binaries, particularly speed_benchmark_torch?</p>
<p>One idea I had is to just manually compile speed_benchmark_torch, since I have its compile command in <code>build_android/compile_commands.json</code>. However, I don’t know the link command, and I imagine there’s a long list of flags on the link command.</p>

username: solvingPuzzles
cooked: <p>Figured it out. The solution is to use <code>-DCMAKE_BUILD_TYPE=Debug</code> in build_android.sh. Otherwise, it defaults to <code>Release</code>, which seems to add the <code>-O2</code> flag.</p>
Backward hook sometimes not called

username: cookiereg
cooked: <p>Hi,</p>
<p>I want to use forward and backward hooks on a single layer in the VQA Ban model. The forward hook works as intended. The backward hook however is simply not called at some layers when experimenting with different layers.<br>
VQA Ban consists of a BAN network with a resnet152 and detectron backend.<br>
With some layers of BAN it works, with some not. At resnet152 no layer works.<br>
The backward hook is simple never called…<br>
I tried the same with Pythia vqa which consists of the Pythia network with a resnet152 and detectron backend. Here every backward hook works as intended without a problem.</p>
<p>Any idea why this could be the case?</p>
<p>Here is the code I use:<br>
(“name” is the name of the layer I want to hook)</p>
<pre><code>    def forward_hook(key):
        def forward_hook_(module, input, output):
            # Save featuremaps
            output = detach_output(output, 0)
            if not isinstance(output, dict) or not isinstance(output, BoxList):
                self.fmap_pool[key] = output

        return forward_hook_

    def backward_hook(key):
        def backward_hook_(module, grad_in, grad_out):
            # Save the gradients correspond to the featuremaps
            self.grad_pool[key] = grad_out[0].detach()

        return backward_hook_

    # If any candidates are not specified, the hook is registered to all the layers.
    for name, module in self.model.named_modules():
        if self.candidate_layers is None or name in self.candidate_layers:
            self.handlers.append(module.register_forward_hook(forward_hook(name)))
            self.handlers.append(module.register_backward_hook(backward_hook(name)))</code></pre>

username: albanD
cooked: <p>Hi,</p>
<p>I would guess that this most likely happens because of interactions between views and inplace operations.<br>
In general I would discourage using module backward hook. You can see in the doc a warning that their current behavior is not always as expected.</p>
<p>In your case, it would be more reliable to use <code>t.register_hook()</code> directly on the Tensors of interest. Note that you can do this inside your forward hook.</p>
RuntimeError: /pytorch/torch/csrc/jit/fuser/cuda/fused_kernel.cpp:202: a PTX JIT compilation failed

username: sigma_x
cooked: <p>This only happens when training Faster R-CNN. The only solution I found was to reinstall pytorch and torchvision. I can’t do it for various reasons. Is anything else possible?</p>

username: ptrblck
cooked: <p>Do you see this issue using a specific PyTorch version and an update fixed it or did a reinstallation of the same version fix it?</p>

username: sigma_x
cooked: <p>I wasn’t able to fix it yet. I can’t reinstall torch and torchvision. The versions are 1.1.0 and 0.3.0. Anything else I could do?</p>

username: ptrblck
cooked: <p>I might have misunderstood the solution from the first post.</p>
<aside class="quote no-group" data-username="sigma_x" data-post="1" data-topic="68360">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sigma_x/48/29584_2.png" class="avatar"> sigma_x:</div>
<blockquote>
<p>The only solution I found was to reinstall pytorch and torchvision</p>
</blockquote>
</aside>
<p>Did the reinstall with <code>1.1.0</code> fixed the issue or a bump to a newer version?</p>
Reinforcement learning with Transformer for NLP

username: Andrei_Ungureanu
cooked: <p>Hello everyone.</p>
<p>My question is related to implementing reinforcement learning [Gradient Policy] with a Transformer sequence to sequence model.</p>
<p>To detail, I am using a Transformer seq2seq model for abstractive summarization. In order to do this, I need to sample a token from the Transformer during inference [no golden target] and to get the probability of that sample.</p>
<p>The way I am trying to do this is:</p>
<pre><code class="lang-auto">def get_distribution(model, batch):

    src, (shift_tgt, lbl_tgt), segs, clss, mask_src, mask_tgt, mask_cls = batch

    # the mock tgt are just torch.zeros tensors that have the same shape as the tgt
    mock_tgt = get_mock_tgt(shift_tgt)
    mock_return = get_mock_tgt(shift_tgt)

    max_length = shift_tgt.shape[1]
    
    log_probs = []
    
    for i in range(0, max_length-1):
        prediction = model(src, mock_tgt, segs, clss, mask_src, mask_tgt, mask_cls)
        prediction = F.softmax(prediction, dim=2)
        
        multi_dist = Categorical(prediction[:, i])
        x_t = multi_dist.sample()
        mock_tgt[:, i+1] = x_t
        mock_return[:, i] = x_t
        log_prob = multi_dist.log_prob(x_t)
        log_probs.append(log_prob)
        
    return mock_return, log_probs
</code></pre>
<p>However, I doubt this is the correct way. More exactly, the Transformer outputs an entire tensor at a time, instead of just one token. In order to get the Distribution of just one token then I use <strong>multi_dist = Categorical(prediction[:, i])</strong> and store the token given by the sample function for the next inference step.</p>
<p>How am I supposed to do this the right way? I have tried searching for it everywhere but cannot seem to find an answer for this.</p>
<p>Help <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"></p>
Batch size of dataparallel

username: jiang_ix
cooked: <p>Hi, assume that I’ve choose the batch size = 32 in a single gpu to outperforms other methods. Now I want use dataparallet to split the training data. I have 4 gpus. To get the same results, should I use batch size = 8 for each gpu or batch size = 32 for each gpu?<br>
Thanks a lot.</p>
<p><span class="mention">@ptrblck_de</span><br>
Thanks a lot</p>

username: ptrblck
cooked: <p>I’m not sure which setup would work the best, but I would try to set <code>batch_size=4*32</code>, such that each GPU gets a batch of 32 samples.<br>
Your training will most likely differ a bit as described <a href="https://discuss.pytorch.org/t/dataparallel-results-in-a-different-network-compared-to-a-single-gpu-run/28635?u=ptrblck">here</a>.</p>
<p>PS: I’m not a huge fan of tagging people, since this might demotivate others to answer in this thread. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=6" title=":wink:" class="emoji" alt=":wink:"></p>

username: jiang_ix
cooked: <p>Hi, But my single gpu cannot use batch size=32 which is out of memory. The single gpu limit the batch size &lt;= 8. So I want to know when I use 4 gpus and each of which use batch size = 8 to train the model. is it right? Thanks a lot.</p>

username: ptrblck
cooked: <p>I misunderstood your question and thought <code>batch_size=32</code> was chosen, because this setup outperforms other models.<br>
Sure, you could try to split the 32 samples among the GPUs. If you encounter any problems, e.g. in the <code>BatchNorm</code> layers (since the sample size is now smaller), you could try to use NVIDIA’s <a href="https://github.com/NVIDIA/apex/tree/master/apex/parallel#synchronized-batch-normalization" rel="nofollow noopener">apex SyncBatchNorm</a>. Note that the default device (usually <code>cuda:0</code>) might use some more memory if you use <code>nn.DataParallel</code>.</p>

username: jiang_ix
cooked: <p>Wow, Thanks a lot. I got it.</p>
End to End trainable SVM in pytorch

username: Nikita_Mehrotra
cooked: <p>Hi,</p>
<p>I am using graphical neural networks to extract feature representation of the input data and then passing it to the binary classifier. Currently, I am using an FC layer with BCE Loss, but I want to replace it with SVM. So I would like to know is there any way to implement SVM that is an end to end trainable in PyTorch.</p>
Inconsistency with train and valiadtaion loss

username: kranti
cooked: <p>I defined a linear layer after the network and this layer is neither part of the network nor the weights are updated during training. It is defined just before the start of the first epoch as mentioned below.</p>
<pre><code class="lang-auto">mod1_encoder = linear([1024, 256, 256])
mod2_encoder = linear([1024, 512, 256])
mod1_classifier = linear([256, 4, 23])
mod2_classifier = linear([256, 4, 23])
net = encoder_individual_classifier(mod1_encoder, mod2_encoder, mod1_classifier, mod2_classifier)
net.cuda()

dset_train = data_loader(root_data_path=params['root_data'], labels_txt=all_labels, mode='trn', transform=None, randomize=True)
train_loader = DataLoader(dset_train, batch_size=params['batch_size'], num_workers=10, sampler=samplr)

dset_val = data_loader(root_data_path=params['root_data'], labels_txt=all_labels, mode='val', transform=None, randomize=False)
val_loader = DataLoader(dset_val, batch_size=params['batch_size'], shuffle=False, num_workers=10)

optimizer = torch.optim.Adam(net.parameters(), lr=params['lr'])
scheduler = torch.optim.lr_scheduler.MultiStepLR(optimizer, milestones=params['decay'], gamma=0.1)

tmp = linear([2,2])
for epoch in ....
</code></pre>
<p>I noticed that gradients backpropagated are different after adding the ‘tmp’ layer even after using the same seed for both the cases. What could be the possible explanation for this?</p>

username: ptrblck
cooked: <p>If you create a new layer, the parameters will be initialized, which will most likely call into a random initialization method and thus the pseudo-random number generator.<br>
If you reset the seed after the initialization of <code>tmp</code>, you might get the same values.<br>
Note that some methods are not deterministic as explained <a href="https://pytorch.org/docs/stable/notes/randomness.html">here</a>.</p>
Torch.backends.cudnn.version() - what does the output mean?

username: CDahmsCellarEye
cooked: <p>With a pip3 install of <code>torch 1.4.0</code> and <code>torchvision 0.5.0</code>, the python statement <code>torch.backends.cudnn.version()</code> shows a result of <code>7603</code>. What does this mean? <code>7.6.3.x</code> ?? or <code>7.6.0.3x</code> ?? or something different ??</p>

username: ptrblck
cooked: <p>The output of <code>7603</code> corresponds to <code>cudnn7.6.3.x</code>.</p>

username: CDahmsCellarEye
cooked: <p>Thanks for the clarification</p>
Different test accuracy

username: oasjd7
cooked: <ol>
<li>In train phase, I got 99.41% accuracy on test loader and  saved this model.</li>
<li>After train process,  I load above model and test -&gt;  accuracy: 99.26%</li>
</ol>
<p>For both train and test phase, I use this test() code.</p>
<p>In test phase, When I remove <code>model.eval()</code> I can get 99.41%.<br>
How can I understand this situation?<br>
In train phase, <code>model.eval()</code> is also applied…then why test_accuracy is different?</p>
<pre><code class="lang-auto">def test(model, dataloader):
    model.eval()
    s_correct = 0
    s_total = 0
    for batch_idx, (img, label) in enumerate(dataloader):
        img, label = img.cuda(), label.cuda()
        with torch.no_grad():
            outputs = model(img)
        _, predicted = outputs.max(1)
        s_total += label.size(0)
        s_correct += predicted.eq(label).sum().item()

    model.train()
    acc = s_correct / s_total
    return acc
</code></pre>
How to use RandomSampler

username: klory
cooked: <p>Ubuntu 18.04 or Mac OS Catalina, Python 3.7, PyTorch 1.4.0</p>
<p>I try to sample from a dataset using predefined indices, <code>SubsetRandomSampler</code> works as expected, <code>RandomSampler</code> does not work as expected, I check the <a href="https://github.com/pytorch/pytorch/blob/b564eaf7a803ae7019ca0523d4ba8c060c30b0cb/torch/utils/data/sampler.py#L107" rel="noopener nofollow ugc">source code</a>, and it seems RandomSampler is just using the length of the <code>data_source</code> argument and the samples has nothing to do with <code>data_source</code>, can anyone help me understand what is the meaning of the argument <code>data_source</code>?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/1/d192729b84c48bea72cbe0692ec340eb60f50cb0.png" data-download-href="https://discuss.pytorch.org/uploads/default/d192729b84c48bea72cbe0692ec340eb60f50cb0" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/1/d192729b84c48bea72cbe0692ec340eb60f50cb0_2_690x403.png" alt="image" data-base62-sha1="tTXylJY0YxZTFwiOoKUrfg38UeY" width="690" height="403" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/1/d192729b84c48bea72cbe0692ec340eb60f50cb0_2_690x403.png, https://discuss.pytorch.org/uploads/default/optimized/3X/d/1/d192729b84c48bea72cbe0692ec340eb60f50cb0_2_1035x604.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/d/1/d192729b84c48bea72cbe0692ec340eb60f50cb0_2_1380x806.png 2x" data-dominant-color="F5F6F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1832×1072 129 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: G.M
cooked: <p>Judging from the source code of <code>RandomSampler</code>, <code>data_source</code> is only there to provide the number of elements <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:">. If u need to sample from some specific indexes, I guess writing your own Sampler would work, it’s not very hard.</p>
Undo/Revert Resize and CenterCrop transformations

username: mlearner
cooked: <p>Hi,<br>
after I use the transformations <code>Resize</code> and <code>CenterCrop</code> I would like to go back to the original image, with the integer RGB values (in [0,255]) for the channels. How can I do this?</p>
<p>I need it because I have to open it with <code>opencv</code>, that only accepts this format to correctly open the image.</p>
<p>Thanks</p>

username: ptrblck
cooked: <p><code>Resize</code> and <code>CenterCrop</code> do not normalize the image, so the pixel values should still be in the same range.<br>
You can hardly undo a cropping operation, as you lose information during the crop.</p>
<p>If you want to undo the normalization, you could multiply with the same <code>stddev</code> and add the <code>mean</code> to the tensor.</p>
Dimensionality of batches

username: al314
cooked: <p>When creating batches</p>
<pre><code class="lang-auto">train_iter = Iterator(trn, batch_size=4)
</code></pre>
<p>I get the dimensionality of the batch for the target is <code>torch.Size([4, NumOfClasses])</code> while dimensionality of batch for predictors is <code>torch.Size([NumOfWordsInSentence, 4])</code>.</p>
<p>This makes a problem in my code to use <code>CrossEntropyLoss()</code> which requires that the first dimension to be the batch size.</p>
<p>Where am I wrong?<br>
I expected the first dimension both for target and predictors to be 4.</p>
<p>E.g. in my code:</p>
<pre><code class="lang-auto">for i, batch in enumerate(train_iter):
    if i == 5:
        print(batch.text.size(), batch.category.size())

# Output
Generated string:  ['moon', 'earth', 'world', 'world', 'hello']
Length =  5
torch.Size([6, 4]) torch.Size([4, 3])
torch.Size([9, 4]) torch.Size([4, 3])
torch.Size([7, 4]) torch.Size([4, 3])
torch.Size([5, 4]) torch.Size([4, 3])
torch.Size([9, 4]) torch.Size([4, 3])
</code></pre>
<p>Full code: <a href="https://gist.github.com/av-maslov/1802eacdf8ceb4704de4deee77fae8e0" rel="nofollow noopener">https://gist.github.com/av-maslov/1802eacdf8ceb4704de4deee77fae8e0</a></p>
Equal/scaled weight to multiple loss functions

username: alexander1
cooked: <p>Loss function = Loss1 + Loss2 + Loss3 +…</p>
<p>What’s the best way to make sure that equal weight (relative to the gradient vector) is given to each LossX no matter the magnitude of each? Should I do Loss1 * Loss2 * Loss3 or call backward multiple times to accumulate the gradient?</p>
<p>I may also wish to scale each loss function (equal to giving each a different learning rate). Merely adding them and scaling does not seem like a good solution since higher magnitude losses will be given more gradient influence even though they are all measuring different things. Hyperparameter search for scaling parameters does not appeal either, there should be a simple analytical solution to this.</p>
Simple for loop on GPU

username: obi_wan7
cooked: <p>Hello,<br>
I would like to ask about parallelization the simple for loop or any loop to be executed on GPU. Currently i have the following code but works very slow on CPU. I cannot find the answer for get this work on GPU. Is it even possible? I search the web without any luck. Any help will be appreciated.<br>
Thank you in advance.</p>
<pre><code class="lang-auto">img_height_vec = [0] * detected_bright_regions_image.shape[0]
img_width_vec = [0] * detected_bright_regions_image.shape[1]
    for i in range(0, detected_bright_regions_image.shape[0]):
        for j in range(0, detected_bright_regions_image.shape[1]):
            if(detected_bright_regions_image[i][j] &gt; 180):
                img_height_vec[i] = 1
                img_width_vec[j] = 1
</code></pre>

username: Shisho_Sama
cooked: <p>I guess, one way would be to create a mask and then ditch the for loop and use the vectorized implementation to achieve what you want.</p>

username: G.M
cooked: <p>Python loops are slow. If u can’t find a way to use some Pytorch methods to replace these loops, u can write them in C++ or use TouchScript. I would suggest first try to replace these loops with other methods. I think the following code should achieve ur goal</p>
<pre><code class="lang-auto">h_max, w_max = image.max(0)[0], image.max(1)[0]
h_vec, w_vec=h_max&gt;180, w_max&gt;180
# these 2 tensors are bool tensors, u can cast them to other types
</code></pre>

username: obi_wan7
cooked: <p>Thank you very much.<br>
I have found the following solution:</p>
<pre><code class="lang-auto">res = (image &gt;= 180)
res = res.type(torch.int)
hv = (torch.sum(res, dim=0) &gt;=1)
wv = (torch.sum(res, dim=1) &gt;=1)
</code></pre>
<p>Pytorch is great. Much more friendly then tensorflow. It requires different mindset, but the results are great.</p>
<p>Thanks, for help.</p>
Pipeline sklearn

username: Natasha_667
cooked: <p>Hi guys,</p>
<p>How can I build a sklearn pipeline to do the following?</p>
<p>What I have:</p>
<blockquote>
<p>A, B = getAB(X_train)<br>
X_train = transform(X_train)<br>
model(A, B, X_train)</p>
</blockquote>
<p>What I want:</p>
<blockquote>
<p>pipe = Pipeline([<br>
(‘ab’, getAB),<br>
(‘tranf’, transform),<br>
(‘net’, net)<br>
]</p>
<p>pipe.fit(X_train, y_train)</p>
</blockquote>
<p>Please help! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

username: ptrblck
cooked: <p><a href="https://github.com/skorch-dev/skorch">skorch</a> provides a scikit-learn compatible wrapper to use with your PyTorch model.<br>
Would that work for you?</p>
How to convert jit::Value into Tensor?

username: JohnZ
cooked: <p>Currently, I load the torch script model as jit module.<br>
I want to print all those parameters. So I want to convert jit::Value data strcture to Tensor.</p>
<pre><code class="lang-auto">if(value.type()-&gt;expect&lt;torch::jit::TensorType&gt;()) {
      auto tensor = value.toTensor();
}
</code></pre>
<p>I try to use same function of IValue but it does not work. Any idea?<br>
Thanks</p>

username: driazati
cooked: <p><code>IValue</code> doesn’t have a <code>.type()</code> method since it’s a generic container for values (similar to Python’s <code>PyObject</code>). You should be able to just call <code>.toTensor()</code> to convert it to a tensor (make sure to guard it with a call to <code>.isTensor()</code> to make sure the <code>IValue</code> is what you expect).</p>
<p>You can find more details <a href="https://pytorch.org/cppdocs/api/structc10_1_1_i_value.html#exhale-struct-structc10-1-1-i-value">here</a>. We are actively working on improving the JIT’s C++ API so issues like this will be more clear in the future.</p>

username: eellison
cooked: <p><a class="mention" href="/u/driazati">@driazati</a> I think it does have a <code>type</code> method ? <a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/core/ivalue.cpp#L26" rel="nofollow noopener">Here</a></p>

username: JohnZ
cooked: <p>Thanks a lot, <a class="mention" href="/u/driazati">@driazati</a>.<br>
I am using torch::jit::Value, not IValue .<br>
I know how to convert IValue to tensor by toTensor.  How about for for torch::jit::Value or torch::jit::Node?</p>

username: driazati
cooked: <p>Sorry for the confusion, you can think of <code>torch::jit::Value</code> as a variable in the graph (i.e. it has no concrete value until the graph is run), so there’s really no way to extract its value since it doesn’t have one. You can read more about it <a href="https://github.com/pytorch/pytorch/blob/master/torch/csrc/jit/docs/OVERVIEW.md#value">here</a>. If you are trying to extract intermediate values from the graph, there’s no way to do that currently but you can follow along at <a href="https://github.com/pytorch/pytorch/issues/21064">this issue</a>.</p>

username: JohnZ
cooked: <p>I got it. So actually jit::Value is just symbol variable.<br>
Thanks a lot, <a class="mention" href="/u/driazati">@driazati</a></p>
MSE loss GPU warning

username: dem123456789
cooked: <p>I get following warning when running pytorch 0.4.1 with python 3.6 on two K80 GPU<br>
/python3.6/site-packages/torch/nn/parallel/_functions.py:58: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.</p>
<p>I have a vector of loss returned. 2 GPU returns 2 losses. For now, I just use torch.mean().<br>
Is this the right thing to do?</p>

username: InnovArul
cooked: <p>Ideally that should be fine.<br>
You seem to be using dataparallel and MSELoss inside your model definition. Is that right or Am I missing something?</p>
<p>What if you use MSE Loss once after you collect the output from the forward pass?</p>

username: dem123456789
cooked: <p>You are correct. I acutally need the loss inside my model, since it seems like I cannot compute the gradient after nn.MSELoss.<br>
The result is similar, so I think the warning is specifically due to my achitecture.</p>

username: InnovArul
cooked: <p>Yes. It is due to your architecture. If you really want to avoid the warning, you should keep the loss function outside the model.</p>

username: Magolor
cooked: <p>Hi! I am having the same problem, but I am using a PyTorch built-in model and thus unable to change its architecture. I’m curious about what would happen if I ignore this warning, which is how will the squeeze operation influence the loss, the training process, and the final precision. Or, what should I do to fix this? Maybe running multiple models? I would really appreciate it if you could reply!</p>

username: dem123456789
cooked: <p>To my best knowledge, it wont affect the result as long as the batch size distributed for each GPU is the same. Otherwise, you need to weighted average the loss. The precision should not be a big issue.</p>
Network not converging

username: FATTOMCAT
cooked: <p>I used various structerees for my network but none of it converged better than 40% so I decided to check if it converges to a constant, it does.<br>
Then I checked if it converges in a “do nothing” case, the net takes 27 inputs and returns a single float, I set all input values to the same as the expected output value and even in this case nothing converges.</p>
<p>a view tested structures :</p>
<pre><code class="lang-auto">self.conv = nn.Sequential(
                nn.Conv1d(1, 1, kernel_size=3, padding=1),
                nn.AvgPool1d(3, padding=1),
                nn.Linear(9, 9),
                nn.Conv1d(1, 1, kernel_size=3, padding=1),
                nn.AvgPool1d(3, padding=1),
                nn.Linear(3, 3),
                nn.Conv1d(1, 1, kernel_size=3, padding=1),
                nn.AvgPool1d(3, padding=1)
            )
</code></pre>
<pre><code class="lang-auto">self.dumb = nn.Sequential(
                nn.Linear(27, 27),
                nn.Linear(27, 1),
            )
</code></pre>
<pre><code class="lang-auto">self.dumb = nn.Sequential(
                nn.Linear(27, 1),
            )
</code></pre>
<p>I used optim.Adam with learning rates from 0.1 to 0.001 and a batch size of 16 and 64 sets in total to cause overfitting<br>
What could solve this issue ?</p>

username: shagunsodhani
cooked: <p>Could you please share a small script with your dummy data and the model that does not converge?</p>

username: SimonW
cooked: <p>you don’t have nonlinearities in your network. in fact all three versions are equivalent.</p>
Training multiple models End-to-End

username: sadegh
cooked: <p>Hi,</p>
<p>My question is regarding training multiple models end-to-end. Consider I have three models that the first model gets input and produce some intermediate representation. The second model modifies the intermediate representation and the first model continues processing the modified representation. Finally, the third model works on the output of the first model.</p>
<p>My question is how to train these networks. There are two options, but I’m not sure which one is practically correct (or if there is a third option).</p>
<p>First option: Each model has its own loss, backward, and optimizer step.<br>
Second option: There is only one loss, which is a (weighted) sum of all models’ losses, one single backward of the whole loss, and optimizer step of all models’ optimizers.</p>
<p>Which one is correct?</p>
<p>Please note, the goal is the output of the first model. The second model is added as guidance, and the third model provides additional feedback.</p>
<p>Thanks,<br>
Sadegh</p>

username: shagunsodhani
cooked: <p>I think both of the options are fine and I would probably prefer the second option.</p>
Ways to save pytorch Field

username: khe
cooked: <p>Right now I am trying to save my pytorch field for NLP related thing, and I am trying to pass the bandit security check. I tried with dill, and it is saying that it has security issue. I was wondering is there any other ways to do it?</p>

username: shagunsodhani
cooked: <p>Hey <a class="mention" href="/u/khe">@khe</a> could you explain what is the pytorch Field that you are trying to save ?</p>
Standardized way to initialize layers

username: partially_observed
cooked: <p>Hi,<br>
I was wondering if there was a standard way to initialize various layers in PyTorch.<br>
For example, I have used something like this for <code>linear</code> and <code>conv</code> layers (please correct me if this is incorrect) :</p>
<pre><code class="lang-auto">        def _weights_init(m):
            if isinstance(m, (nn.Conv2d, nn.Linear)):
                init.xavier_normal_(m.weight)
                # m.bias.data.zero_()
            elif isinstance(m, (nn.BatchNorm2d, nn.BatchNorm1d)):
                m.weight.data.fill_(1)
                m.bias.data.zero_()
</code></pre>
<p>Similarly, what is the best way to initialize recurrent layers (such as LSTMs and GRUs)?</p>
<p>Thanks!</p>

username: shagunsodhani
cooked: <p>Your use of <a href="https://pytorch.org/docs/stable/nn.init.html" rel="nofollow noopener"><code>torch.nn.init</code></a> seems to be the standard way.  For initializing the weights of LSTSMs/GRUs, use the <code>init</code> module on the weights/bias matrics.</p>
Multidimentional tensor max

username: Sam3
cooked: <p>Hi,<br>
I have a multi-dimensional tensor, in wich  way can i get the maximum over all dimensions and the multi-dimensional index for the maximum?</p>

username: ptrblck
cooked: <p>This code should work:</p>
<pre><code class="lang-python">x = torch.randn(100, 100, 100)
x_max = torch.max(x)
idx_max = (x == x_max).nonzero()
</code></pre>

username: Sam3
cooked: <p>Thank you so much  <img src="https://discuss.pytorch.org/images/emoji/apple/blush.png?v=9" title=":blush:" class="emoji" alt=":blush:"></p>
Creat a tensor with random number of 1 and -1

username: mingren200323
cooked: <p>In order to add noise to the XNOR-Net, I need to modify the trained weights which contains only 1 and -1. So I think the problem is how to  generate a tensor with random number of 1 and -1, and then multiply this tensor with the trained weights. The solution of mine is following:</p>
<pre><code class="lang-auto"> def add_noise_to_weights(m):
            s = m.data.size()
            n = m.data.nelement()
            r = round(n*0.01) #0.01 is the noise ratio
            mask = -torch.ones([1, r]).type_as(m.data)
            neg_mask = torch.ones([1, n-r]).type_as(m.data)
            t = torch.cat((mask, neg_mask),1).reshape(s)
            idx = torch.randperm(t.nelement())
            t = t.view(-1)[idx].view(t.size())
            m.data = m.data.mul(t)
           return m
</code></pre>
<p>Despite it works, the code is too complicated, so could you guys have some simply solutions?</p>

username: tux
cooked: <p>If I understand correctly, you just want to generate a tensor which contains -1 and 1 values ?</p>
<p>Would something like this work for you ?</p>
<pre><code class="lang-auto">a = torch.Tensor([-1, 1])
idx = np.random.randint(2, size=your_shape)
noise = a[idx]
</code></pre>
<p>Edit :<br>
Furthermore, if you want to have some control on the number of 1 or -1 in your tensor, I would suggest:</p>
<pre><code class="lang-auto">a = torch.Tensor([-1, 1])
idx = np.random.choice(2, size=your_shape, p=[r, 1-r])
noise = a[idx]
</code></pre>

username: mingren200323
cooked: <aside class="quote no-group" data-username="tux" data-post="3" data-topic="68333">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tux/48/2364_2.png" class="avatar"> tux:</div>
<blockquote>
<p>idx = np.random.choice(2, size=your_shape, p=[r, 1-r])</p>
</blockquote>
</aside>
<p>Thank you very much for your reply! Based on your reply,I think this may be the right  answer to my question.</p>
<pre><code class="lang-auto">s = x.data.size()
t = torch.from_numpy(numpy.random.choice([-1, 1], size=s, p=[0.5, 0.5])).type_as(x)
x = x.data.mul(t)
</code></pre>
Do i need to set same batch size when testing?

username: davidlee
cooked: <p>Hi, Here is simple question,</p>
<p>I trained DNN containing batchnorm1D. (batch size is 64)<br>
<strong>Do i need to set same batch size(64) when testing with test loader?</strong></p>

username: tux
cooked: <p>Hi,</p>
<p>If you are testing your model, you should set <code>model.eval()</code> mode. Then this would deactivate batchnorm and dropout layers. Then, you don’t need to have the same batch size between the test and train loader.</p>
Loading multiple images from same class

username: harshj94
cooked: <p>I am having a dataset with multiple classes with each class having multiple images.<br>
While running the loop over data loader I want to get two images from each class in every iteration.<br>
I would like to have an iterator like</p>
<p><code>for image1, image2, label in dataloader:</code></p>
<p>Please let me know how can I achieve this.</p>

username: Nathan_Drake
cooked: <blockquote>
<p>Blockquote<br>
<a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html%5C" rel="noopener nofollow ugc">https://pytorch.org/tutorials/beginner/data_loading_tutorial.html\</a></p>
</blockquote>
<p>Look this official tutorial. You should rewrite the <strong>getitem</strong> method for your custom dataset which inherite from torch.utils.data.Dataset module.</p>

username: harshj94
cooked: <p><a class="mention" href="/u/nathan_drake">@Nathan_Drake</a> Thanks. Its done.</p>

username: lephesien
cooked: <p>Hello,</p>
<p>The link seems to be broken.<br>
<a class="mention" href="/u/harshj94">@harshj94</a> Can you please share how you did that ?<br>
Were u able to use torchvision.datasets.ImageFolder in some way  ?</p>

username: harshj94
cooked: <p>The updated link is: <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html" rel="nofollow noopener">Data Loading Tutorial</a></p>
<p>You need to override the _<em>len_</em> and _<em>getitem_</em> function.<br>
You can do one-time processing needed in the init function.</p>

username: lephesien
cooked: <p>Thanks a lot !<br>
I got it and finally manage to build a multi-inputs network (2 images and 1 vector of custom features).</p>
What does the 'score' in mrcnn consist of?

username: Michael-J98
cooked: <p>I’m following the tutorial of object detection using mrcnn on the website. And I noticed that there is a ‘score’ as one of the outputs of model. So I wonder how does this ‘score’ being calculated? Is it concerned about one or more of object, classification, mask or anything else?</p>

username: ekilic
cooked: <p>It is classification score of objects.</p>

username: Michael-J98
cooked: <p>Thank you very much!</p>

username: Michael-J98
cooked: <p>excuse me…?<br>
I fed this model with my own dataset and here is what I got:<br>
‘labels’: tensor([1, 1, 1, 1, 1, 1], device=‘cuda:0’),<br>
‘scores’: tensor([0.9990, 0.9987, 0.9987, 0.9889, 0.2024, 0.0583]], device=‘cuda:0’).<br>
Why the labels of the last 2 are positive while their scores are so low? If the the score comes from the output of softmax classification, shouldn’t it be at least over 0.5 when only 2 classes, target and background?</p>
Channel Convolution

username: Hoang_Minh_Q
cooked: <p>In Depth-wise convolution layer of MobileNet-v1, each channel of the input is convolved with a kernel with channel length = 1, then we use groups = in_channels in Conv2d to produce 1-channel kernels (in_channels = out_channels in depth-wise convolution). In the paper MoBiNet (<a href="https://arxiv.org/abs/1907.12629" class="inline-onebox" rel="noopener nofollow ugc">[1907.12629] MoBiNet: A Mobile Binary Network for Image Classification</a>), the author define the term K-dependency<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/5/1/51cded9451f1ca82f2abe955a0d02d43441fa8db.png" alt="image" data-base62-sha1="bFFSbiskUXnR9StZzCFninmIQkj" width="485" height="224"><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/a/9/a94a6195dd5b1dbd1aea8023412bf5be25bfaf57.png" alt="image" data-base62-sha1="o9C3jz39KJs29B7kJcIPdDhsMDl" width="231" height="106"><br>
From what I understand, this means that each output is now the activation of the sum of the convolution of each corresponding 1-channel kernel with channels in the same group of the inputs.<br>
For example, we have Input with 4 channels C1, C2, C3, C4, then we seperate them into 2 groups, group 1 has C1, C2 and group 2 contains C3, C4. After that we have 4 1-channel kernels K1, K2, K3, K4. Then the output will be<br>
O1 = Activation(C1<em>K1 + C2</em>K1),<br>
O2 = Activation(C1<em>K2 + C2</em>K2),<br>
O3 = Activation(C3<em>K3 + C4</em>K3),<br>
O4 = Activation(C3<em>K4 + C4</em>K4)<br>
(* is the convolution operation)<br>
I have tried several ways to implement this on pytorch but they turned out to be failures. Any suggestion on how to implement this?<br>
Thank you in advance!</p>
Saving all checkpointable objects for the best model

username: kbtorcher
cooked: <p>Although this issue has been raised before, I am new to PyTorch and am having some difficulty understanding model checkpointing.</p>
<ol>
<li>
<p>How can we save all checkpointable objects for the best model with the least validation loss? I don’t want to save multiple checkpoints corresponding to different iteration/epoch.</p>
</li>
<li>
<p>What’s the difference between deep cloning checkpointable objects vs using model.state_dict()?</p>
</li>
</ol>

username: ptrblck
cooked: <ol>
<li>
<p>You could use the code from the <a href="https://github.com/pytorch/examples/blob/0c1654d6913f77f09c0505fb284d977d89c17c1a/imagenet/main.py#L254">ImageNet example</a> to store only the best checkpoint.</p>
</li>
<li>
<p>If you write them to a file, there won’t be any difference. However, if you want to reload a (temporary) checkpoint after some iterations, you could need to perform a deepcopy, as the <code>state_dict</code> stores references to the parameters.</p>
</li>
</ol>
Tensor product problem

username: yuejiang_li
cooked: <p>Hi,</p>
<p>I’m new to pytorch, and I have a problem here:</p>
<p>Suppose I have a tensor <code>a</code> with shape <code>(seq_len, batch_size, feature_size)</code>, and another tensor <code>b</code> with shape <code>(batch_size, feature_size)</code>. I want to product tensor <code>a</code> and <code>b</code>, so that I can obtain a tensor <code>c</code> with shape <code>(seq_len, batch_size)</code>. The entry <code>(i,j)</code> to tensor <code>c</code> is the inner product of <code>a[i, j, :]</code> and <code>b[j, :]</code>.</p>
<p>Is there any convenient way to calculate this?</p>
<p>Thanks</p>

username: ptrblck
cooked: <p>These approaches should work:</p>
<pre><code class="lang-python"># Setup
seq, batch, feature = 2, 3, 4
a = torch.randn(seq, batch, feature)
b = torch.randn(batch, feature)

# Einsum
b_ = b.permute(1, 0).unsqueeze(0).expand(2, -1, -1)
out1 = torch.einsum('sbf, sfb-&gt;sb', a, b_)

# Mul + sum
out2 = (a * b).sum(-1)

# Manual
out3 = torch.zeros(seq, batch)
for i in range(seq):
    for j in range(batch):
        a_ = a[i, j]
        b_ = b[j]
        out3[i, j] = torch.dot(a_, b_)

# Check
print(torch.allclose(out1, out2) and torch.allclose(out1, out3))
&gt; True
</code></pre>
<p>Depending on the workload one might be faster.</p>

username: yuejiang_li
cooked: <p>Seems like <code>torch.einsum</code> can do a lot of this kind of tensor operation.<br>
Many thanks!</p>
Loss_rpn_box_reg is nan [torchvision 0.3]

username: kl_divergence
cooked: <p>I’m using torchvision 0.3.<br>
Here’s an example from both datasets I’m using:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/d/d241018e107ba735acafb9fca512ce8af44cb29e.png" data-download-href="https://discuss.pytorch.org/uploads/default/d241018e107ba735acafb9fca512ce8af44cb29e" title="Screenshot%20from%202019-07-07%2013-32-15"><img src="https://discuss.pytorch.org/uploads/default/original/2X/d/d241018e107ba735acafb9fca512ce8af44cb29e.png" alt="Screenshot%20from%202019-07-07%2013-32-15" data-base62-sha1="tZZxHQHJLcZlvKKEB64i9cCWbJY" width="690" height="294" data-dominant-color="F4F4F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot%20from%202019-07-07%2013-32-15</span><span class="informations">989×422 32.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/a/a68bf05368f3b4903b28dd698b0ede37b3b7a333.png" data-download-href="https://discuss.pytorch.org/uploads/default/a68bf05368f3b4903b28dd698b0ede37b3b7a333" title="Screenshot%20from%202019-07-07%2013-26-48"><img src="https://discuss.pytorch.org/uploads/default/original/2X/a/a68bf05368f3b4903b28dd698b0ede37b3b7a333.png" alt="Screenshot%20from%202019-07-07%2013-26-48" data-base62-sha1="nLl4O28z3vz87pwOZvPZ7IeY85t" width="690" height="339" data-dominant-color="EFEFEF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot%20from%202019-07-07%2013-26-48</span><span class="informations">989×487 46.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Here’s my model :</p>
<pre><code class="lang-auto">def get_model(num_classes):
    model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True)
    in_features = model.roi_heads.box_predictor.cls_score.in_features
    model.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_classes) # replace the pre-trained head with a new one
    return model.cuda()
</code></pre>
<p>I’ve made both the dataset classes identical as per torchvision docs. I’ve seen the same behaviour when I train on high res images for target_dataset also after some epochs. <strong>What can done to deal with this</strong> ? <strong>What can be the reason for getting ‘nan’ for the first dataset in first run</strong>?<br>
I’ve tried lowering lr, normalizing images but still the same issue persists.</p>

username: Magolor
cooked: <p>fasterrcnn_resnet50_fpn takes bbox [x1,y1,x2,y2] as input but the data set is using [x1,y1,width,height] it seems.</p>
Loss does not decrease

username: satinder147
cooked: <p>I have been trying modified U-Net on a custom dataset. I have attached the image and its corresponding mask. After some time the loss stops decreasing. It remains at around 1. The results produced are not satisfactory. The architecture I am using is :</p>
<pre><code class="lang-auto">class Unet(nn.Module):
    '''U-Net Architecture'''

    def __init__(self,inp,out):
        super(Unet,self).__init__()
        self.c1=self.contracting_block(inp,64)
        self.c2=self.contracting_block(64,128)
        self.c3=self.contracting_block(128,256)
        self.maxpool=nn.MaxPool2d(2)
        self.upsample=nn.Upsample(scale_factor=2,mode="bilinear",align_corners=True)
        self.c4=self.contracting_block(128+256,128)
        self.c5=self.contracting_block(64+128,64)
        self.c6=nn.Conv2d(64,out,1)

    def contracting_block(self,inp,out,k=3):
        block =nn.Sequential(
            nn.Conv2d(inp,out,k,padding=1),
            nn.ReLU(),
            nn.BatchNorm2d(out),
            nn.Conv2d(out,out,k,padding=1),
            nn.ReLU(),
            nn.BatchNorm2d(out)
        )
        return block


    def forward(self,x):
        #280,280,1
        conv1=self.c1(x) 
        #280,280,128
        x=self.maxpool(conv1)
        #140,140,128
        conv2=self.c2(x)
        #140,140,256
        x=self.maxpool(conv2)
        #70,70,256
        conv3=self.c3(x)
        #70,70,512
        x=self.upsample(conv3)
        #140,140,512
        x=torch.cat([conv2,x],axis=1)
        #140,140,768
        x=self.c4(x)
        #140,140,256
        x=self.upsample(x)
        #280,280,256
        x=torch.cat([conv1,x],axis=1)
        #280,280,384
        x=self.c5(x)
        #280,280,128
        x=self.c6(x)
        #280,280,3
        return x
</code></pre>
<p>I am using Adam as my optimizer and BCEWithLogitsLoss as my loss function. Any help would be highly appreciated.<br>
Thank you.</p>

username: ptrblck
cooked: <p>I would try to scale down the problem a bit and try to overfit a small data sample first.<br>
Once your model successfully overfits the small data sample, you could try to scale it up again.</p>
<p>Based on your loss function, it seems you are dealing with a multi-label segmentation use case, i.e. each pixel might belong to zero, one, or more classes.<br>
If your use case is a multi-class segmentation, i.e. each pixel belongs to once class only, you would have to use <code>nn.CrossEntropyLoss</code> instead.</p>