akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Add custom Layer in the middle of torchvision model

Shantanu_Ghosh: <p>Hi,<br>
I have created a resnet model from torchvision like this:<br>
model = torchvision.models.resnet101(pretrained=pre_trained)<br>
Now i want to create a custom layer say conv layer, just after  max_pool as indicated in the picture</p>
<p>To be precise, i want to replace the batch norm layer with a custom metadatanorm as indicated in the paper: “<a href="https://arxiv.org/pdf/2104.09052.pdf" rel="noopener nofollow ugc">https://arxiv.org/pdf/2104.09052.pdf</a>”</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/8/a8b1f2480a24d868b642b4a186c27fa5e5eda28d.png" data-download-href="https://discuss.pytorch.org/uploads/default/a8b1f2480a24d868b642b4a186c27fa5e5eda28d" title="Screen Shot 2022-08-19 at 4.52.37 AM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/8/a8b1f2480a24d868b642b4a186c27fa5e5eda28d_2_690x239.png" alt="Screen Shot 2022-08-19 at 4.52.37 AM" data-base62-sha1="o4lsJyw41TUHD1WAhP7NKSYZoxn" width="690" height="239" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/8/a8b1f2480a24d868b642b4a186c27fa5e5eda28d_2_690x239.png, https://discuss.pytorch.org/uploads/default/original/3X/a/8/a8b1f2480a24d868b642b4a186c27fa5e5eda28d.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/a/8/a8b1f2480a24d868b642b4a186c27fa5e5eda28d.png 2x" data-dominant-color="EDEDED"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2022-08-19 at 4.52.37 AM</span><span class="informations">1016×353 74 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>You could replace the layer with an <code>nn.Sequential</code> block containing the original layer as well as your new layer as described <a href="https://discuss.pytorch.org/t/adding-intermediary-layers-to-a-pre-trained-bert-model/158825/2">here</a>.</p>
Feature dimension in Convolution

timosy: <p>If we use RGB and set 32 as out_channels, feature of R is extracted with 12 filters?<br>
nn.Conv2d(in_channels=RGB, out_channels=32, kernel_size=3, padding=1),</p>

ptrblck: <p>No, each if the <code>32</code> filters in the conv layer will use <em>all</em> input channels to create a single output activation map. These outputs will then be stacked to the <code>32</code> output channels.</p>

timosy: <p>Thanks for your explanation.</p>
<p>nn.Conv2d(in_channels=RGB, out=32),<br>
and<br>
nn.Conv2d(in_channels=R, out=32),<br>
nn.Conv2d(in_channels=G, out=32),<br>
nn.Conv2d(in_channels=B, out=32),</p>
<p>Calculation time between upper and lower is same, but, the result is sometimes very different?</p>

ptrblck: <p>Sorry, I don’t fully understand the use case. If you are re-initializing new conv layers, the results would always differ unless you are properly seeding the code, setting the filter weights and bias manually to defined values, or are loading a <code>state_dict</code>. If you want to split the filters to work on each input channel separately, you could use the <code>groups</code> argument.</p>

timosy: <p>For instance, among RGB R does not have features, GB have some features. In this case, when RGB is input, convoluted by a filter-A, a generated new feature map-A after the convolution might have worse features compared with a new feature map-A that is generated/added with each feature map convoluted via individual input R, G, and B. just thinking…</p>
<blockquote>
<p>you could use the <code>groups</code> argument.</p>
</blockquote>
<p>I did not know it.<br>
I will check it, thnak you for the information.</p>
PyTorch profiler CUDA results not shown

Pramesh_Gautam1: <p>I’ve been using PyTorch profiler and the results are attached here. The chart only shows DataLoader, CPU Exec and Other. Tensorboard chart is not showing GPU time. I’ve used <code>activities=[torch.profiler.ProfilerActivity.CUDA, torch.profiler.ProfilerActivity.CPU], </code> in profiling code and the GPU is being utilized as well. How can this be fixed so that GPU timings are also shown?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/5/65c24d5d6324e6c6c35ee79912d976b5da5f65e6.png" data-download-href="https://discuss.pytorch.org/uploads/default/65c24d5d6324e6c6c35ee79912d976b5da5f65e6" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/5/65c24d5d6324e6c6c35ee79912d976b5da5f65e6_2_690x385.png" alt="image" data-base62-sha1="ewcvVW987zTLbAs4HevII7c8axw" width="690" height="385" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/5/65c24d5d6324e6c6c35ee79912d976b5da5f65e6_2_690x385.png, https://discuss.pytorch.org/uploads/default/optimized/3X/6/5/65c24d5d6324e6c6c35ee79912d976b5da5f65e6_2_1035x577.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/6/5/65c24d5d6324e6c6c35ee79912d976b5da5f65e6_2_1380x770.png 2x" data-dominant-color="F1E9EA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1523×850 58.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

beschtaPy: <p>Hi!<br>
I have the same issue. Did you find a solution? Are you also on Windows and using PyTorch 1.11?</p>
<p>Printing the results in the console works however.<br>
print(prof.key_averages().table(sort_by=“cuda_time_total”, row_limit=20))</p>
"Couldn't lower all tuples" when export model with onnx

oPluss: <p>When i export the onnx model, i got the error:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "/home/opluss/Documents/deploy/FSD_deploy/fsd/deploy/onnx.py", line 60, in &lt;module&gt;
    verbose=True
  File "/home/opluss/anaconda3/envs/py37torch17/lib/python3.7/site-packages/torch/onnx/__init__.py", line 230, in export
    custom_opsets, enable_onnx_checker, use_external_data_format)
  File "/home/opluss/anaconda3/envs/py37torch17/lib/python3.7/site-packages/torch/onnx/utils.py", line 91, in export
    use_external_data_format=use_external_data_format)
  File "/home/opluss/anaconda3/envs/py37torch17/lib/python3.7/site-packages/torch/onnx/utils.py", line 639, in _export
    dynamic_axes=dynamic_axes)
  File "/home/opluss/anaconda3/envs/py37torch17/lib/python3.7/site-packages/torch/onnx/utils.py", line 421, in _model_to_graph
    dynamic_axes=dynamic_axes, input_names=input_names)
  File "/home/opluss/anaconda3/envs/py37torch17/lib/python3.7/site-packages/torch/onnx/utils.py", line 159, in _optimize_graph
    torch._C._jit_pass_lower_all_tuples(graph)
RuntimeError: Couldn't lower all tuples.
</code></pre>
<p>What does the  <code>tuples</code>  mean? What are the possible reasons?</p>

marksaroufim: <aside class="quote no-group" data-username="oPluss" data-post="1" data-topic="125901">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/opluss/48/39953_2.png" class="avatar"> oPluss:</div>
<blockquote>
<p><code>all tuples.</code></p>
</blockquote>
</aside>
<p>Sorry for the late response but could you share a repro? Hard to debug without more context</p>

zaxk: <p>Have you solved this problem? I have the same issue.</p>
Extracting metadata from PDF

MagaMakhauri: <p>Hi!<br>
I have used PyPDF2 with regex to extract metadata from different invoice PDFs my company recieves. Then I tried tesseract-ocr and thought some manipulation improved the ‘PDF reading capabilities’<br>
But its still not enough. I was wondering if its at all possible to train a ML model to do that. Like write out the metadata (adress, total amount etc) by hand and ‘show’ it as a desired result.</p>
<p>My question is: Is it possible? Is it something pytorch can actually do?</p>
Will torch.cat create new space for data?

ZiQing_Li: <p>I use <code>embedded_cat = torch.cat([ embedded, embedded_dirs], -1)</code>, but I found the GPU memory increased 3G. It seems that <code>torch.cat()</code> will create new space for data. Is there any way to take advantage of the original data space (i.e. embedded and embedded_dirs)?</p>

ptrblck: <p>No, I don’t think this is possible as both tensors could (and most likely are) allocated in different memory regions.</p>
Which is the best practice in coding PANET style networks?

alessandro_mondin: <p>I am currently implementing from scratch YOLOV5 and I was wondering if there are any suggestion on how to write down “elegantly” such networks.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/b/8b2470970a8b961ba5e64ef36a98182fac66887d.png" data-download-href="https://discuss.pytorch.org/uploads/default/8b2470970a8b961ba5e64ef36a98182fac66887d" title="Screenshot 2022-08-18 at 19.01.27"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/b/8b2470970a8b961ba5e64ef36a98182fac66887d_2_620x500.png" alt="Screenshot 2022-08-18 at 19.01.27" data-base62-sha1="jQUriz8FzTAS6cgGfM2q40bjRDv" width="620" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/b/8b2470970a8b961ba5e64ef36a98182fac66887d_2_620x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/8/b/8b2470970a8b961ba5e64ef36a98182fac66887d_2_930x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/8/b/8b2470970a8b961ba5e64ef36a98182fac66887d_2_1240x1000.png 2x" data-dominant-color="D8C9C4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2022-08-18 at 19.01.27</span><span class="informations">1624×1308 360 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>In my case the only strategy that comes to my mind is to for code separately each top-down/buttom-up routes (the arrows in the picture) to store the desired routes_connections tensors to be used later on.<br>
More precisely am creating separated nn.Sequential() or nn.ModuleList() for each one of those blocks.</p>
<p>However I was wondering if any of you know if it is possible to link the parts <strong>2)</strong> and <strong>3)</strong> together inside a unique nn.Sequential or nn.ModuleList()</p>
<p>Hope I was clear,<br>
Alessandro</p>
Build and Run QNNPACK on X86

Darshan: <p>Hi Pytorch team,</p>
<p>I want to build and run QNNPACK on X86 CPU and there are not many references to do the same. It would be great if I can get the references to build and execute QNNPACK on the X86 platform.</p>
<p>Thank you!</p>

Darshan: <p>Look forward to your reply.</p>
<p>Thank you</p>

Vasiliy_Kuznetsov: <p>Hi <a class="mention" href="/u/darshan">@Darshan</a> , qnnpack should be included by default with x86 builds of PyTorch as long as your hardware supports it.  Can you try the following in your setup:</p>
<pre><code class="lang-auto">import torch
print(torch.backends.quantized.supported_engines)
&gt; ['qnnpack', 'none', 'onednn', 'fbgemm']
</code></pre>
Reordering colum breaks gradient

KaisersHolm: <p>Hello</p>
<p>I created a function in a multimodal model, which is supposed to reorder columns based on their correlation<br>
X1 and X2 are the embeddings of the two views.</p>
<p>Below is the function:</p>
<pre><code class="lang-auto">        def get_corr_idx(self,X1, X2, n_rows):
        corr_matrix = torch.corrcoef(torch.concat([X1.t(), X2.t()]))
        # take lower matrix without diagonal
        corr_matrix = corr_matrix[-n_rows:, :n_rows]
        # make matrix absolute cause we dont care about negative or positive correlation
        corr_matrix = torch.abs(corr_matrix)
        view1_idx = []
        view2_idx = []
        for i in range(0, n_rows):
            arg_max = torch.argmax(corr_matrix)
            row = arg_max.item() / n_rows
            row = int(row)
            column = arg_max.item() % n_rows
            view1_idx.append(column)
            view2_idx.append(row)
            corr_matrix[:, column] = 0
            corr_matrix[row, :] = 0
        a = torch.LongTensor(view1_idx).to(torch.device("cuda:0"))
        b = torch.LongTensor(view2_idx).to(torch.device("cuda:0"))
        X1_trans = torch.index_select(X1, 1, a)
        X2_trans = torch.index_select(X2, 1, b)
        return X1_trans, X2_trans
</code></pre>
<p>I think the problem here is that grad_fn of “a” and “b” is None, which breaks the gradient. Because of that, my model stops learning and the losses stagnate.</p>
<p>Does anyone have an suggestion on how to do this?</p>
<p>Thank you.</p>
<p>Hilmar</p>

JuanFMontesinos: <p>it’s not about reordering.<br>
When you call</p>
<pre><code class="lang-auto">torch.LongTensor
</code></pre>
<p>you are reinstantiating the tensor and breaking the attached graph.<br>
you can cast to long via <code>tensor.int()</code> maybe <code>tensor.long()</code></p>

srishti-git1110: <p>In addition to what <a class="mention" href="/u/juanfmontesinos">@JuanFMontesinos</a> correctly pointed out, I’m also thinking about <code>int(row)</code> which makes <code>row</code> an<code> int</code> object.</p>
<p>That to say even if you use what Juan has suggested, (so that the data isn’t copied reinstantiating the tensor) that <code>int</code> object (<code>row</code>) shall anyway have no graph attached to it -  graphs are for tensor objects only.<br>
Makes sense?</p>
<p>Also, I do not fully know or understand your use-case, but here’s something -</p>
<p>you are constructing <code>a</code> and <code>b</code> as <code>long</code> type and also wanting them to have their <code>requires_grad=True</code> or something, essentially wanting them to be somewhere in the gradient calculation thing. But, from what I know only tensors of data type = float or some complex type can have their <code>requires_grad=True</code>.</p>
<p>Not very sure if this would cause any errors for your use case though.</p>
<p>Hoping it helps,<br>
Srishti</p>
NT-Xent loss vs. other infoNCE variants

joohyunglee: <p>It may be an unfitting question to this thread.<br>
It looks like NT-Xent loss from <a href="https://arxiv.org/abs/2002.05709?context=stat.ML" rel="noopener nofollow ugc">SimCLR</a> is different from other papers, e.g. <a href="https://arxiv.org/abs/1911.05722" rel="noopener nofollow ugc">MoCo</a>, <a href="https://arxiv.org/abs/1807.03748" rel="noopener nofollow ugc">CPC</a>, <a href="https://arxiv.org/abs/1805.01978" rel="noopener nofollow ugc">instance discrimination paper</a>, in that NT-Xent loss exclude positive sample in its denominator. So, NT-Xent loss has 2N-2 terms in its denominator whereas the other infoNCE variants have 2N-1 terms (Is this correct?) Why does NT-Xent loss exclude the positive sample at its denominator? Thank you ahead.</p>

tom: <p>Thank you for mentioning this.</p>
<p>Note that in SimCLR the positive sample (i, j) is not excluded in the denominator but only the (i, i) pairing. So effectively, the sampling model is to draw the second index from all indices but i.</p>
<p>My reading of the loss in the CPC paper (and it is referenced by MoCo) is that they do not consider the index i at all and then draw the index j implicitly excluding i (because they just say “one positive sample”).</p>
<p>So for all I know, from the formulas it might be just a different presentation of the same loss.<br>
With these details, it is not uncommon to see subtle differences between the paper and the implementation, so it might be good to look at those, too.</p>
<p>Best regards</p>
<p>Thomas</p>

mxahan: <p>We can provide a simplified table for clarification. We are assuming s1 as sample  1 and t(s1) as its’ augmented format.</p>
<div class="md-table">
<table>
<thead>
<tr>
<th>sample</th>
<th>Augmentation</th>
</tr>
</thead>
<tbody>
<tr>
<td>s1</td>
<td>t(s1)</td>
</tr>
<tr>
<td>s2</td>
<td>t(s2)</td>
</tr>
<tr>
<td>s3</td>
<td>t(s3)</td>
</tr>
<tr>
<td>s4</td>
<td>t(s4)</td>
</tr>
</tbody>
</table>
</div><p>The InfoNCE considers <em>s1</em> and <em>t(s1)</em> as positives (numerator) and uses all 7 of them <em>except s1</em> itself in the denominator.</p>
<p>The SimCLR considers all such combinations s2, t(s2), and… in a single loss calculation (we see the sum of them).</p>
<p>Hope that clarifies a bit.</p>
Backprop for hierarchical multilabel (~6k)

VictorCallejas: <p>Hi there,</p>
<p>I am working on a <strong>multilabel classification</strong> problem where <strong>from process parameters</strong> I want to <strong>predict defects</strong> to later make optimization and root cause analysis with SHAP.</p>
<p>The <strong>number of labels is huge</strong> although they are <strong>structured in a hierarchical way</strong>:<br>
For each view (Left, Right, Top… (6)) I have 10x10 positions(x,y coords) where there can be up to 10 defect types.</p>
<p>I am <strong>modeling this like a tree structure from the latent space</strong>.<br>
latent space → views → positions → defect type</p>
<p>My question is, which is the <strong>best approach for the backpropagation</strong>?</p>
<ul>
<li>BCE from the 6k labels</li>
<li>BCE and backprop for each end logit (defect type)</li>
<li>Another approach? <img src="https://discuss.pytorch.org/images/emoji/apple/slightly_smiling_face.png?v=12" title=":slightly_smiling_face:" class="emoji" alt=":slightly_smiling_face:" loading="lazy" width="20" height="20">
</li>
</ul>
<p>I have found very scarce information about this topic, so I would love any input or reference materials</p>
<p>Thanks! <img src="https://discuss.pytorch.org/images/emoji/apple/ninja.png?v=12" title=":ninja:" class="emoji" alt=":ninja:" loading="lazy" width="20" height="20"></p>
Warning when evaluating a model for the second time

jmaltar: <p>Hello everyone,</p>
<p>first of all - this is my first post on PyTorch forum - so I would like to thank for every kind of support I’ve ever found here since 2018. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>Now, back to my problem. I’m using a <em>torchscript-ed</em> model from Python’s PyTorch here in C++ code. Experiments conducted by far show how values are identical both in Python and C++. Therefore, I think things are exported/imported successfully.</p>
<p><strong>But</strong>, there is a strange warning I receive. In this minimal example</p>
<pre><code class="lang-auto">int main() {
    torch::NoGradGuard guard;

    torch::Device device = torch::kCPU;
    // same warning when: torch::Device device = torch::kCUDA;

    torch::Tensor t1 = torch::rand({1, 3, 224, 224}, torch::kFloat).to(device);
    torch::Tensor t2 = torch::rand({1, 3, 224, 224}, torch::kFloat).to(device);
    torch::Tensor t3 = torch::rand({1, 3, 224, 224}, torch::kFloat).to(device);
        
    torch::jit::script::Module model = torch::jit::load("../cache/model.pt", device);
    
    std::cout &lt;&lt; "Feedforwarding 1st time\n";
    std::vector&lt;torch::jit::IValue&gt; inputs{};
    inputs.push_back(t1);
    model.forward(inputs).toTensor();

    std::cout &lt;&lt; "Feedforwarding 2nd time\n";
    inputs.clear();
    inputs.push_back(t2);
    model.forward(inputs).toTensor();

    std::cout &lt;&lt; "Feedforwarding 3rd time\n";
    inputs.clear();
    inputs.push_back(t3);
    model.forward(inputs).toTensor();

    return 0;
}
</code></pre>
<p>when I feedforward  <code>t1</code>, everything seems fine, then when feedforward <code>t2</code> the warning appears, and afterwards everything is fine once again - the output is:</p>
<pre><code class="lang-auto">Feedforwarding 1st time
Feedforwarding 2nd time
[W graph_fuser.cpp:108] Warning: operator() profile_node %353 : int = prim::profile_ivalue(%out_dtype.1)
 does not have profile information (function operator())
Feedforwarding 3rd time
</code></pre>
<p>Also, during this second evaluation I noticed abnormal running time (approx. 5 seconds for 2nd evaluation, and approx. 20ms for 1st, 3rd, 4th,… evaluations on my modest notebook GPU).</p>
<p>Any help is appreciated.</p>

tom: <p>This ist the JIT looking for optimization potential.<br>
These three (somewhat undocumented?) API calls from graph_executor.h / namespace <code>torch::jit</code> might help:</p>
<pre><code class="lang-auto">TORCH_API std::atomic&lt;bool&gt;&amp; getProfilingMode();
TORCH_API std::atomic&lt;bool&gt;&amp; getExecutorMode();
TORCH_API std::atomic&lt;size_t&gt;&amp; getNumProfiledRuns();
</code></pre>
<p>Use <code>getNumProfiledRuns</code> to choose which run will be slow or change one of the two the above to disable things.<br>
Of course, only do this if you don’t need the fuser.</p>
<p>Best regards</p>
<p>Thomas</p>
How to upload a custom dataset to a contrasitive learning SIMCLR implementation

AP_M: <p>Link: <a href="https://github.com/larsh0103/simclr/blob/main/main.py" class="inline-onebox" rel="noopener nofollow ugc">simclr/main.py at main · larsh0103/simclr · GitHub</a></p>
<p>Hello, I implemented the github code (given in Link) and worked well. I want to implement the same code using my timeseries dataset present on my desktop. I am unable to stack the samples and read through dataloader . Please help</p>

AP_M: <p>Can anyone please help in this.</p>

ptrblck: <p>You would need to explain the issue a bit more, what you’ve tried so far, where you are currently stuck etc. Right now the question sounds more like a request for someone to implement a fully working solution for you.</p>

AP_M: <p>Looking at the link, I have given in my question, {simclr/main.py at main · larsh0103/simclr · GitHub} I have tried to upload my dataset by stacking 2 tensors in the “get_item” method as is done in the link(main.py) for cifar10 dataset. But when iterating through dataloader it gives me the error that i have described in “<a href="https://discuss.pytorch.org/t/valueerror-not-enough-values-to-unpack-expected-2-got-1/158701" class="inline-onebox">ValueError: not enough values to unpack (expected 2, got 1)</a>”. Now able to understand how should i return my 2 stacked samples (with data augmentations done) from get_item method so that it is readable by training dataloader.</p>

ptrblck: <p>Are you still running into the same issue from the other topic?<br>
I’ve followed up with a code snippet, which works fine and which you could use to compare your code to mine to narrow down where the difference (and thus error) is coming from.</p>

AP_M: <p>Corrected. Thanks a lot for the help.</p>
Combining optimally weighted multiple losses

EmilPeters: <p>Hi guys,</p>
<p>I’m currently designing a network that combines multiple losses.</p>
<p>My output consists of 4 channels, where the first channel’s loss should be calculated using the <code>BCEWithLogitsLoss</code> and the other three channels should be calculated using MAE (<code>L1Loss</code>).</p>
<p>Currently, my implementation looks as follows:</p>
<pre><code class="lang-python">bceLoss = self.BCE.forward(x[0, 0, :], y[0, 0, :])
mae = self.MAE.forward(x[0, 1:, :], y[0, 1:, :])
return bceLoss + mae
</code></pre>
<p>The issue I’m getting is that I feel that the MAE loss outweighs the BCE loss by a lot, and the network is thus focussing more on reducing the MAE loss.<br>
I know I can do something in the likes of: <code>return bceLoss * alpha + mae * beta</code></p>
<p>The question I know have is how to know what <code>alpha</code> and <code>beta</code> should be? To weigh them properly.</p>
<p>Kind regards,<br>
Emil</p>

tom: <p>Hi Emil,</p>
<p>so one observation is that doubling alpha and beta just scales your loss, so maybe just setting one, say alpha, of them to 1 would be OK.<br>
Of course, one way to be sure is to have an additional hold-out set (or do cross validation or so) and then try a few values beta and taking the best-performing one is “the gold standard”. If you really need to do it well and are prepared to expend effort for it, this is the thing to do.</p>
<p>As that can be a bit tedious, though, my intuitive approach would be to move the two loss components to be similar in size (doesn’t need to be exactly the same - won’t last anyway - but say within a factor of 2 or 3). If you don’t do this, the gradient descent will essentially optimize the larger loss and the smaller only has an influence once the larger does not change much. (This is a bit handwavy and I don’t mention gradient magnitude and all, but maybe it can be useful).</p>
<p>Best regards</p>
<p>Thomas</p>
Exact same codes, same random seed, huge different outcomes

LXR: <p>Hi, I am doing a Resnet50 transfer learning with no pre-trained weights.</p>
<p>For the same codes with the same random state, my friend has both  accuracy constantly increasing to 90%, However, my train acc stuck at 40%, valid acc stuck at 33%<br>
We have the exact same codes. The only difference is that he ran the codes using A6000 GPU, mine is the 2060s.</p>
<p>What could be possible reasons for these outcomes?</p>
<p>ps. his num_workers is 2 ,mine is 0 because of windows system, using other number will raise error.</p>

tom: <p>I’d probably check the optimizer and the data-loading setup in more detail.<br>
My usual approach is to have a very detailed log of what is fed, length of the dataloader etc.</p>
<p>Best regards</p>
<p>Thomas</p>
As I downloaded libtorch, I have two choices: Pre-cxx11 ABI and cxx11 ABI. The question is that what is the difference between them？

Vista: <p><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/3/2339e014eed1e1f4949c03f60c79512d87c3c73b.png" alt="image" data-base62-sha1="51CHWuHTOn8EMq37VNZvTs3EddN" width="606" height="339"></p>

tom: <p>In itself, the choice would not matter too much (I’d probably take the cxx11 ABI).<br>
The tricky part is when combining it with other libraries:</p>
<ul>
<li>Typically C++ libraries are linked with the cxx11 ABI by default.</li>
<li>The official PyTorch Python module are linked with the old pre-cxx11 ABI, so if you want to use them, you need this.</li>
<li>If you have a mix of these, you are in a real pickle. <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12" title=":frowning:" class="emoji" alt=":frowning:" loading="lazy" width="20" height="20">
</li>
</ul>
<p>Best regards</p>
<p>Thomas</p>
How to retry a randomly crashed training task in bash script?

JimW: <p>Hello,  I am new to this field. I am launching some pytorch training tasks to server but it crashed randomly at various epochs.  The crash might be related to CUDA memory and num_worker&gt;0, and some setup incompatibilities, which makes it difficult to debug.  But I was wondering if it is possible to get around by retrying the training in bash script.</p>
<pre><code class="lang-auto">My bash script currently looks like this:
apt-get install XXX
cd my folder/
python my_train.py. --save_ckpth_path my_checkpoint_folder
</code></pre>
<p>The my_train.py command will save checkpoint files generated from every epoch.<br>
What I hope to do is:  Every time  the training command crashed,  we rerun the same training command but make it start from the latest checkpoint saved as initialization.</p>
<p>Do you think the above idea makes sense?  If so, could someone provide some guidance on how to implement it in the script?  Thanks.</p>
Does each worker collects consecutive items?

malfonsoarquimea: <p>if I have multiple workers, would each worker collect consecutive indexes?<br>
For example, if I have num_workers set to 2 , prefetch_factor set to 2 and batch_size set to 10 then, would one worker collect the first 2 batches (idx 0 to 6) and the second worker the following 2 batches (idx 6 to 12)?</p>

nivek: <p>Each worker reads all the samples unless you specified differently in <code>worker_init_fn</code>.</p>
<p>See <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.get_worker_info" rel="noopener nofollow ugc">this note</a>:<br>
When used in a <code>worker_init_fn</code> passed over to <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader" rel="noopener nofollow ugc"><code>DataLoader</code></a>, this method can be useful to set up each worker process differently, for instance, using <code>worker_id</code> to configure the <code>dataset</code> object to only read a specific fraction of a sharded dataset, or use <code>seed</code> to seed other libraries used in dataset code.</p>

ptrblck: <p>Yes, your description is correct but each worker would collect 10 samples as seen here:</p>
<pre><code class="lang-python">class MyDataset(Dataset):
    def __init__(self):
        self.data = torch.arange(20).view(-1, 1)
        
    def __getitem__(self, idx):
        worker = torch.utils.data.get_worker_info()
        if worker:
            print("worker id {}, idx {}".format(worker.id, idx))
        x = self.data[idx]
        return x
    
    def __len__(self):
        return len(self.data)

dataset = MyDataset()
batch_size = 10
loader = DataLoader(dataset, batch_size, num_workers=2, prefetch_factor=2)

for data in loader:
    print(data.shape)

# worker id 0, idx 0
# worker id 0, idx 1
# worker id 0, idx 2
# worker id 0, idx 3
# worker id 0, idx 4
# worker id 0, idx 5
# worker id 0, idx 6
# worker id 0, idx 7
# worker id 0, idx 8
# worker id 0, idx 9
# worker id 1, idx 10
# worker id 1, idx 11
# worker id 1, idx 12
# worker id 1, idx 13
# worker id 1, idx 14
# worker id 1, idx 15
# worker id 1, idx 16
# worker id 1, idx 17
# worker id 1, idx 18
# worker id 1, idx 19
# torch.Size([10, 1])
# torch.Size([10, 1])
</code></pre>
Look up Table in Pytorch

gerstla: <p>Hi</p>
<p>I am new to Pytorch and I was wondering if the following can be done.<br>
If I have a 2D tensor of indices for example:</p>
<p><em>tensor([[1, 2, 4],</em><br>
<em>[4, 2, 0],</em><br>
<em>[0, 1, 3]])</em></p>
<p>and a 1D tensor of values</p>
<p><em>tensor([6, 8, 9, 11, 23])</em></p>
<p>I would like to get an output of</p>
<p><em>tensor([[8, 9, 23],</em><br>
<em>[23, 9, 6],</em><br>
<em>[6, 8, 11]])</em></p>
<p>Thanks</p>

ptrblck: <p>Yes, you can directly index the tensor:</p>
<pre><code class="lang-python">x = torch.tensor([6, 8, 9, 11, 23])
idx = torch.tensor([[1, 2, 4],
                    [4, 2, 0],
                    [0, 1, 3]])


ret = x[idx]
print(ret)
# tensor([[ 8,  9, 23],
#         [23,  9,  6],
#         [ 6,  8, 11]])
</code></pre>
GRU output and h_n relationship

Peixiang_Zhong: <p>As shown in the documentation: <a href="http://pytorch.org/docs/0.2.0/nn.html#torch.nn.GRU" rel="nofollow noopener">http://pytorch.org/docs/0.2.0/nn.html#torch.nn.GRU</a></p>
<p>The GRU outputs <strong><em>output (seq_len, batch, hidden_size * num_directions): tensor containing the output features h_t from the last layer of the RNN, for each t.</em></strong></p>
<p>and</p>
<p><em><strong>h_n (num_layers * num_directions, batch, hidden_size): tensor containing the hidden state for t=seq_len</strong></em></p>
<p>For GRU with <code>n_layers = 2</code> and <code>bi_directional = True</code>,</p>
<p>Then I assume that the backward hidden state at the last position <code>output[-1,:,hidden_size:]</code> is equal to <code>h_n[3,:,:],</code> but in my code that is not true.</p>
<p>I have tried <code>h_n[0,:,:]</code>, <code>h_n[1,:,:]</code> and <code>h_n[2,:,:]</code> as well, but none is equal to <code>output[-1,:,hidden_size:]</code></p>
<p>Why? (I need forward hidden state at last position to initialize decoder hidden state in my Seq2Seq model)</p>
<p>Thanks in advance.</p>

DrShushen: <p>As it turns out, this is answered here:</p><aside class="quote quote-modified" data-post="2" data-topic="3561">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/spro/48/146_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/bilstm-hidden-states-and-output-dont-match/3561/2">BiLSTM hidden states and output don't match</a> 
  </div>
  <blockquote>
    You’re actually comparing the first output of the reverse direction. I think you want out[0, :, 500:] 
 <a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/1X/b01a9e75313b8c651c7e3cb22c2161abbaa4751c.png" data-download-href="https://discuss.pytorch.org/uploads/default/b01a9e75313b8c651c7e3cb22c2161abbaa4751c" title="Screen Shot 2017-05-31 at 12.56.45 PM.png">[image]</a>
  </blockquote>
</aside>
R-CNN time scales linearly with batch size because of RPN?

Michael_Crouch: <p>I have a system which performs inference using fasterrcnn_resnet50_fpn on single images, and recently I’ve been testing the performance when I batch examples together. However, the time taken by model(images) seems to scale almost exactly linearly (+/- 5%) with the number of images in a batch – it’s taking the same amount of time per image. I’m not seeing any speedup from sending multiple images to the GPU at once.</p>
<p>Looking through the code, I’ve noticed that fasterrcnn_resnet50_fpn requires a <em>list</em> of images, rather than the Tensor I’d expect. Why?</p>
<p>There’s a lot of code called from fasterrcnn_resnet50_fpn, but so far I’ve found things that don’t seem to parallelize in:<br>
models/detection/anchor_utils.py::AnchorGenerator.forward()<br>
models/detection/transform.py::GeneralizedRCNNTransform.forward()<br>
models/detection/rpn.py::RPNHead.forward()</p>
<p>I’m not good enough with profiling pytorch code yet to figure out what the real problems are, though. I’ve done enough profiling to verify that 95%+ of my code’s time is being used in fasterrcnn_resnet50_fpn.forward(), so I know it’s not a problem with data loaders or any of my surrounding code.</p>
<p>Are the RPN algorithms that pytorch uses fundamentally not able to parallelize across a batch? Is this an implementation problem? Have I misunderstood something?</p>
How should I use "@" operator when I want to multipy two tensors?

111466: <p>How should I use “@” operator when I want to multipy two tensors?</p>

AlphaBetaGamma96: <p>Hi <a class="mention" href="/u/111466">@111466</a>,</p>
<p>The <code>@</code> operator is just shorthand for matrix multiplication, if you want to do element-wise multiplication you’ll want to use the <code>*</code> operator instead.</p>
Please help me explain the mechanism of this case of torch.matmul

111466: <p>Please help me explain the mechanism of this case of torch.matmul.<br>
I can not understand why this case is this result.<br>
Thank you very much!</p>
<pre><code class="lang-auto">import torch
# first argument 2D and second argument 1D
mat2_1 = torch.tensor([[1, 2, 3],
                       [4, 3, 8],
                       [1, 7, 2]])

mat2_2 = torch.tensor([3, 6, 2])

# assigning to output tensor
out_2 = torch.matmul(mat2_1, mat2_2)

print("\n2D-1D matmul :\n", out_2)
</code></pre>
<p>The output of above case is as follows</p>
<pre><code class="lang-auto">2D-1D matmul :
 tensor([21, 46, 49])
</code></pre>

AlphaBetaGamma96: <p>Hi <a class="mention" href="/u/111466">@111466</a>,</p>
<p>This is just standard matrix multiplication, in your case you have a <code>3x3</code> matrix that multiples with a <code>3x1</code> vector (both are represented as <code>Tensors</code>). When multiplying a <code>3x3</code> matrix with a <code>3x1</code> vector you get a <code>3x1</code> vector as the output.</p>
<p>The output of each element will be defined by a dot product of a row of your matrix and your vector. So, the first element of <code>out_2</code> is the dot product of the 1st row of the matrix with your vector, the second element will be the dot product of the 2nd row and your vector and so on. This is just the definition of matrix multiplication, you can read more about it on Wikipedia <a href="https://en.wikipedia.org/wiki/Matrix_multiplication" rel="noopener nofollow ugc">here</a></p>
How to interpolate items of a tensor?

malfonsoarquimea: <p>Hi! I have a tensor of shape (300x1024) containing a 1024 length codification of 300 frames<br>
Some keyframes, one every 30 frames (frame 0,30,60…) is a keyframe.<br>
I need to, for each code that doesn’t belong to a keyframe, to interpolate between the previous and next keyframe and set the result as the code for that frame<br>
Any elegant solution to do this?<br>
thanks!</p>

ptrblck: <p>I guess <code>interpolate</code> would work if you select every 30th data point:</p>
<pre><code class="lang-python">x = torch.randn(300, 1024) 

x = x[None, None]
out = torch.nn.functional.interpolate(x[:, ::30], size=(300, 1024))
out = out.squeeze()
</code></pre>

malfonsoarquimea: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="159020">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>you</p>
</blockquote>
</aside>
<p>thanks! that is what I was looking for!</p>
Gradient Accumulation in Detectron2

Chung-Hao_Ku: <p>I was wondering whether calling optimizer.zero_grad() after after optimizer.step() has the same effect as the usual order within a single iteration? The reason for this is because I am trying to use gradient accumulation in Detectron2 for my model as memory size is limited. However, in Detectron2 every iteration step is defined as a function, including zeroing out gradients, backpropogation and weight update. Therefore, if I put optimizer.zero_grad at first, as step() is called for a new iteration, it will just zero out all gradients instead of accumulating it. If I were to accumulate gradients for a specified number of iterations, I was thinking to put optimizer.zero_grad() after the optimizer step() like the following:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/5/f588ee00e7f38526b6923be6ec4138fd53d6408d.png" alt="擷取" data-base62-sha1="z26lKrzTrbpKFtFZQiSxSG4G737" width="450" height="113"></p>
<p>In this case, I will only zero out graidents at a certain number of iterations. I am wondering if my thought process is correct? Thanks (model is trained with DDP).</p>

Ahmad_Zobair_Surosh: <p>Hi<br>
Have you managed to solve this issue <a class="mention" href="/u/chung-hao_ku">@Chung-Hao_Ku</a> ?<br>
If not, <a class="mention" href="/u/ptrblck">@ptrblck</a> , any insight ?</p>
<p>Thanks in advance</p>

ptrblck: <p>Yes, you should not zero out the gradients without executing the <code>optimizer.step()</code> method as you would lose this backward pass.<br>
<a href="https://discuss.pytorch.org/t/why-do-we-need-to-set-the-gradients-manually-to-zero-in-pytorch/4903/20">This post</a> explains different approaches for gradient accumulation.</p>
No module named 'torchvideotransforms'

szahan: <p>I  am trying to run a github repo that has the following import<br>
<code>from torchvideotransforms import video_transforms, volume_transforms</code></p>
<p>I installed pytorchvideo using but it’s not working</p>
<pre><code class="lang-auto">pip install pytorchvideo
</code></pre>
<p>I might be wrong about the library but I could not find anything suitable. I know about image transformation but am interested to know about this library. Please help. Thanks.</p>

ptrblck: <p>I’m not sure where <code>torchvideotransforms</code> comes from but it seems <code>pytorchvideo</code> uses <code>pytorchvideo.transforms</code> as the namespace for the transformations as seen <a href="https://pytorchvideo.readthedocs.io/en/latest/api/transforms/transforms.html">here</a>.</p>

szahan: <p>Found it. Itis another github repo</p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/hassony2/torch_videovision/blob/master/torchvideotransforms/video_transforms.py">
  <header class="source">

      <a href="https://github.com/hassony2/torch_videovision/blob/master/torchvideotransforms/video_transforms.py" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/hassony2/torch_videovision/blob/master/torchvideotransforms/video_transforms.py" target="_blank" rel="noopener nofollow ugc">hassony2/torch_videovision/blob/master/torchvideotransforms/video_transforms.py</a></h4>


      <pre><code class="lang-py">import numbers
import random
import numpy as np
import PIL
import skimage.transform
import torchvision
import math
import torch

from . import functional as F


class Compose(object):
    """Composes several transforms

    Args:
    transforms (list of ``Transform`` objects): list of transforms
    to compose
    """

</code></pre>



  This file has been truncated. <a href="https://github.com/hassony2/torch_videovision/blob/master/torchvideotransforms/video_transforms.py" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
What should you store or append a batch of tensors to in C++ when using LibTorch?

Alio: <p>In C++, when using LibTorch (The C++ version of PyTorch), what should you store a batch of tensors in? I’m running into the problem of not being able to reset the batch on the next step because C++ doesn’t allow storing a new variable over an existing variable.</p>
<p>In my attempt my batch of tensors is one single 385x42 tensor. The batch size is 385. In a for loop I use <code>torch::cat</code> to concatenate 385 smaller 1D tensors, which are 42 numbers long. (Maybe ‘stack’ or ‘append’ are better terms for what I’m doing since the are stacked together picket fence style more than ‘concatenated’, but that’s what I’m using.) Anyways, there is not problem with this shape. It seems to work fine for one forward and backward pass but then the tensor becomes 770x42 on the next pass instead of a 385x42 tensor of the next 385, 42 long arrays. I hope I am painting a picture and not being too verbose.</p>
<p>The code.<br>
Near the bottom I have the line <code>all_step_obs = torch::tensor({});</code> to try to wipe out the contents of the tensor, AKA, the batch, but this gives me a <code>Segmentation fault (core dumped)</code>. I guess for trying to access the tensor outside of the loop(?)<br>
If I don’t have this line I get a 770x42 tensor after the next <code>step</code>.</p>
<pre><code>int max_steps = 385;
int steps = 2000;
auto l1_loss = torch::smooth_l1_loss;
auto optimizer = torch::optim::Adam(actor.parameters(),  3e-4);

torch::Tensor train() {
    torch::Tensor all_step_obs;

    for (int i = 0; i&lt;steps; ++i)
    {
        for (int i = 0; i&lt;max_steps; ++i)
        {
            all_step_obs = torch::cat({Gym().step().unsqueeze(0), all_step_obs});
        }
        auto mean = actor.forward(all_step_obs);

        auto loss = l1_loss(mean, torch::rand({385, 42}), 1, 0);

        optimizer.zero_grad();
        loss.backward();
        optimizer.step();

        all_step_obs = torch::tensor({});

        if (steps == 1999) {
            return loss;
        }
    }
};
</code></pre>

Matej_Kompanek: <p>Not sure what you’re trying to achieve, can’t you do it like this (torch::cat is giving me exception with empty ‘all_step_obs’)</p>
<pre><code class="lang-auto">std::vector&lt;torch::Tensor&gt; step_obs;
for (int j = 0; j &lt; max_steps; ++j)
{
   step_obs.push_back(torch::rand({ 42 }));
}
auto all_step_obs = torch::stack(step_obs);
</code></pre>
<p>PS: nested loop with same iteration variable ‘i’ doesnt seem right</p>
Why would we need to differentiate through data augmentations?

Vedant_Roy: <p>The Kornia library (<a href="https://github.com/kornia/kornia" class="inline-onebox" rel="noopener nofollow ugc">GitHub - kornia/kornia: Open Source Differentiable Computer Vision Library</a>) allows us (amongst other features) to differentiate/backprop through data augmentations.</p>
<p>I’ve been trying to figure out <em>why</em> someone would find this useful.<br>
Generally backprop will stop at the 1st layer of the neural net, since there’s no reason to update the input data itself.</p>
<p>So why would someone want to use Kornia augmentations (besides speed)?</p>
Advanced indexing with torch.topk

AreTor: <p>Suppose I have a 3d Tensor <code>x</code>, and I run <code>itorch.topk(x, k=2, dim=0)[1]</code> to retrieve the indices of the first two max values over the 0th dimension.</p>
<p>Then I want to use those indices to index the tensor to assign a value, however I am not able to define the code to perform the correct advanced indexing, The only thing I was able to do is:</p>
<pre><code>    _, H, W = x.shape
    inds = torch.topk(x, k=self.k, dim=1)[1]

    h = torch.arange(H).long().repeat(W, 1).transpose(0, 1).contiguous().view(-1)
    w = torch.arange(W).long().repeat(H)
    for i in range(self.k):
        x[list(inds[i, :, :].view(-1)), list(h), list(w)] = val
</code></pre>
<p>however this is very inefficient. Does someone know some more efficient version?</p>

ptrblck: <p>Would this work:</p>
<pre><code class="lang-auto">x = torch.randn(3, 10, 10)
idx = torch.topk(x, k=2, dim=0)[1]
x.scatter_(0, idx, 100)
print(x)
</code></pre>

Zoran_Peric: <p>It, works now. Thanks very much.</p>
<p>predicted_k_indexes = torch.topk(predictions_2[0, -1, :],k=3)<br>
prk_0 = predicted_k_indexes[0]<br>
prk_1 = predicted_k_indexes[1]<br>
for item11 in prk_1:<br>
print (item11.item())</p>
<p>output:<br>
484<br>
523<br>
35075</p>

pinakinathc-me: <p>can you please explain the purpose of the <strong>third</strong> argument in the function?</p>
<pre><code class="lang-auto">x.scatter_(0, idx, &gt;&gt;100&lt;&lt;)
</code></pre>

ptrblck: <p>The third argument is the <code>src</code> and represents the value which will be scattered into <code>x</code> in the <code>0</code> dimension defined by the indices in <code>idx</code>.<br>
In my code snippet the value <code>100</code> will be written to <code>x</code>.</p>
Multi head attention gives wrong attention weights when model is set to .eval()

Gul_Zain: <p>Hello,<br>
I am using nn.MultiHeadAttention in my transformer layer and I want to visualize the attention on my tokens. However, whenever, i set model to model.eval(), multihead attention outputs a matrix of constant values. However, whenever, i set model to model.train(), it outputs something different that is not constant. I am not sure why this weird behaviour is showing up. I have tested it with pytorch 1.7 and pytorch 1.11. Looking forward to an answer.</p>

eqy: <p>I’m not sure I can reproduce the issue from the current description:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch
&gt;&gt;&gt; model = torch.nn.MultiheadAttention(256, 8)
&gt;&gt;&gt; model = model.cuda()
&gt;&gt;&gt; inp = torch.randn(128, 4, 256, device='cuda')
&gt;&gt;&gt; out = model(inp, inp, inp)
&gt;&gt;&gt; model = model.eval()
&gt;&gt;&gt; out2 = model(inp, inp, inp)
&gt;&gt;&gt; torch.allclose(out[0], out2[0])
True
&gt;&gt;&gt; torch.allclose(out[1], out2[1])
True
&gt;&gt;&gt; model = model.train()
&gt;&gt;&gt; out3 = model(inp, inp, inp)
&gt;&gt;&gt; torch.allclose(out[0], out3[0])
True
&gt;&gt;&gt; torch.allclose(out[1], out3[1])
True
&gt;&gt;&gt;
</code></pre>
<p>Do you have a minimal code snippet that reproduces the issue?</p>