akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to solve GTest and LibTorch linkage conflict

tot0k: <p>This question follows <a href="https://stackoverflow.com/questions/62654297/how-to-use-gtest-with-cmake-linkage-problem-when-following-googles-guide/" rel="nofollow noopener">my precedent one on StackOverflow</a>.</p>
<p>I’m writing a C++ program with OpenCV, Torch and NumCpp. The program compiles and works fine for now, but I need to write unit tests.</p>
<p>I’ve followed <a href="https://github.com/google/googletest/tree/master/googletest#incorporating-into-an-existing-cmake-project" rel="nofollow noopener">google’s tutorial</a> to build GTest and GMock inside my project, but it fails. When I don’t link Torch libraries, that works.</p>
<p><strong>Error when linking GTest + Torch :</strong></p>
<pre><code class="lang-auto">/usr/bin/ld: CMakeFiles/TryGTest_test.dir/test/boxTest.cpp.o: in function `testing::AssertionResult testing::internal::CmpHelperEQFailure&lt;int, int&gt;(char const*, char const*, int const&amp;, int const&amp;)':
/tmp/tmp.Z1zXnMtLsD/cmake-build-debug-ubuntu_2/googletest-src/googletest/include/gtest/gtest.h:1511: undefined reference to `testing::internal::EqFailure(char const*, char const*, std::string const&amp;, std::string const&amp;, bool)'
collect2: error: ld returned 1 exit status
</code></pre>
<p>I suppose it comes from Libtorch defining macros with the same name as Gtest, and Google proposes a workaround at the end of the linked tutorial to fix this type of problem, but I need to find which macro is failing.</p>
<p>Hope someone can help me finding libtorch macro names <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
How can I create custom dataloader with Multi30k dataset?

I_H_Yoo: <p>I’m trying to train the Attention model of the <a href="https://pytorch.org/tutorials/beginner/torchtext_translation_tutorial.html" rel="nofollow noopener">PyTorch tutorial</a> via TPU in colab Env. So I need a DataLoader for <code>torch_xla.distributed.parallel_loader.ParallelLoader()</code> to train multi core TPU. But the process of converting Multi30k dataset to a DataLoader is getting trouble. What am I missing?</p>
<pre><code class="lang-python">train_data, valid_data, test_data = Multi30k.splits(exts =('.de', '.en'), fields=(SRC, TRG))

SRC.build_vocab(train_data, min_freq=2)
TRG.build_vocab(train_data, min_freq=2)

# for item in torch.utils.data.DataLoader(train_data, batch_size=128): # Error occurs
#     print(item)
</code></pre>
<p>The error message is TypeError: default_collate: batch must contain tensors, numpy arrays, numbers, dicts or lists; found &lt;class ‘torchtext.data.example.Example’&gt;.<br>
Or if there’s another way to create ParallelLoader() of xla by using 'Bucketiterator`, please give me an advice.</p>
Channel-wise proportional threshold

FruitVinegar: <p>Hello all.</p>
<p>I’d like to implement a function:</p>
<ul>
<li>
<p>argument:<br>
– x (torch.Tensor): x.shape == (B, C, H, W)<br>
– recipe (maybe python:dict): ex) {1: (0, 50), 2: (50, 75), 3: (75, 100)}</p>
</li>
<li>
<p>returns:<br>
– mask (torch.Tensor): mask.shape == (B, C, H, W)</p>
</li>
<li>
<p>how it works:<br>
In below figure, white matrix represents only one channel of given x.<br>
(i.e., x[sample_batch_idx][sample_channel_idx])<br>
also arbitrary recipe is given by proportional.(not threshold)<br>
the function returns mask represented by colored matrix.<br>
I’d like to do this process in channel-wisely.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/2/02297a911a9521ef2fc6dae7ba6e550e1c170919.png" data-download-href="https://discuss.pytorch.org/uploads/default/02297a911a9521ef2fc6dae7ba6e550e1c170919" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/2/02297a911a9521ef2fc6dae7ba6e550e1c170919.png" alt="image" data-base62-sha1="j7P0SDXSekP8wEE1HkJRVVCtf3" width="244" height="500" data-dominant-color="E0F6D7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">369×755 4.62 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
</li>
</ul>
<p>Currently, I implemented a test function using for loop, but It shows horrible performance.<br>
Is there any efficient way to implement this function?<br>
I think the (automatic) broadcasting will be the key idea instead of for loop, but I don’t know how to apply this.</p>
<p>Any suggestions will be welcome for me.</p>
Cannot get the value of torch::TensorList in vector

Keunhoi_An: <pre><code class="lang-auto"> for (i=0; i&lt;score_.sizes()[0]; i++){
    if (score_[i].item&lt;double&gt;() &gt; cfg_visualization_eval_score_threshold){
                        
        result_vector.push_back(torch::TensorList({clipped_boxes[i],score_[i],labels_[i]}));

        for(torch::TensorList result_vec: result_vector) {
                  std::cout&lt;&lt; result_vec.size() &lt;&lt; "result_vec.size()" &lt;&lt; std::endl;

                  std::cout&lt;&lt; result_vec &lt;&lt; endl &lt;&lt; "result_vec" &lt;&lt; std::endl;
                  system("pause");
               }
           }
    }
</code></pre>
<p>I think push_back works well, but when I print out result_vec, it shows correct output of result_vec.size() as 3 but the value is weird as shown below.</p>
<pre><code class="lang-auto">[[ Tensor (undefined) ], [ Tensor (undefined) ], [ Tensor (undefined) ]]
result_vec
</code></pre>
<p>Is there anyone who knows the reason of this weird result?</p>
How can I implement a mask tensor in libtorch?

Keunhoi_An: <p>As the title, how can I implement a mask tensor with libtorch written in python such as:</p>
<pre><code class="lang-auto">mask_ = (box[:, 3] &lt;= box[:, 1]) | (box[:, 2] &lt;= box[:, 0])
</code></pre>
<p>I’ve done with indexing but I’ve struggled with “|” operator.</p>
<pre><code class="lang-auto">mask_ = box.select(1, 3) &lt;= box.select(1, 1);
</code></pre>

tom: <p>Just using <code>|</code> would seem to work in C++:</p>
<pre><code class="lang-python">csrc = """
torch::Tensor fn(const torch::Tensor&amp; a, const torch::Tensor&amp; b) {
  return a | b;
}

PYBIND11_MODULE(TORCH_EXTENSION_NAME, m) {
  m.def("fn", &amp;fn, "test function");
}
"""

ext = torch.utils.cpp_extension.load_inline("test_ext", csrc)
a = torch.tensor([False, True])
print(ext.fn(a, a[:, None]))
</code></pre>
<p>Best regards</p>
<p>Thomas</p>

Keunhoi_An: <p>I’ve checked before I post this question, and I’ve found that <code>at::Tensor | at::Tensor</code> is not allowed…<br>
<code>mask_ = ( box.select(1, 3) &lt;= box.select(1, 1) ) | ( box.select(1, 2) &lt;= box.select(1, 0) );</code></p>

tom: <p>You probably want to upgrade your PyTorch version, I did check that the snippet above works with <em>recent</em> PyTorch.<br>
Boolean tensors have been introduced in PyTorch 1.2 or so and PyTorch 1.3 and following refined the semantics.</p>
<p>Best regards</p>
<p>Thomas</p>
<p>P.S.: We don’t use at::Tensor but torch::Tensor these days <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:">.</p>

Keunhoi_An: <p>I’ve found that<br>
<code>mask_ = (box[:, 3] &lt;= box[:, 1]) | (box[:, 2] &lt;= box[:, 0])</code><br>
works but<br>
<code>mask_ = ( box.select(1, 3) &lt;= box.select(1, 1) ) | ( box.select(1, 2) &lt;= box.select(1, 0) );</code><br>
does not work…which is quite not understandable for me…<br>
In addition, does libtorch depend on pytorch version? I don’t understand clearly whether libtorch matters with pytorch version because I think libtorch is not connected with pytorch directly.</p>
<p>(Oh, I declare most of variables as torch::Tensor, but when I check the code with vscode, it shows that those are at::Tensor,  which is unexpected for me. Anyway it seems no problem.)</p>

tom: <p>libtorch is roughly the C++ part of PyTorch plus a C++ API for the bits of PyTorch implemented in Python. In particular, they share the versions. I usually just use the libtorch that comes with PyTorch to build things, but that could be me.<br>
Do you use a recent libtorch?</p>

Keunhoi_An: <p>Yes and I think version of both libtorch and pytorch are satisfied.</p>
Using a trained model from PyTorch in MATLAB/Simulink

abodh_ltd: <p>I have a 1-D CNN model trained in PyTorch and now I want to use the trained model in MATLAB/Simulink. Can anybody provide me an insight on this?</p>

rayanelhelou: <p>Any luck yet?<br>
I’m also trying to run a model in Simulink.</p>

abodh_ltd: <p>Unfortunately no. Sadly, I think they have not worked on making Pytorch compatible for Simulink</p>
Should multi-task losses be normalized?

Epoching: <p>Let’s say there were two losses, and some of the losses share some parts of the architecture (i.e. a feature extractor for images).</p>
<ul>
<li>Cross Entropy</li>
<li>MSE</li>
</ul>
<p>The floating value for the Cross Entropy loss would be something low, say 0.25<br>
The floating value for the MSE loss would be something larger, say 20.4</p>
<pre><code class="lang-auto"># Let the below variables be tensors holding the graph for calculating the loss
# cross_entropy_loss = 0.25
# mse_loss = 20.4
</code></pre>
<p>I’m currently doing this:</p>
<pre><code class="lang-auto">tot_loss = cross_entropy_loss + mse_loss 
tot_loss.backward()
</code></pre>
<p>Is this ok? Or is it recommended to normalize the floating values of the losses to ensure both losses have equal backprop gradient importance/values?</p>
How can I change a 32-bit operating system from CPU and GPU to a 4-bit or 8-bit operating system?

learningsteady0J0: <p>I created a new quantum technique using pytorch!</p>
<p>So I’m trying to see how well this technique works on my server computer, Raspberry Pie B3+ and the Jetson TX2.</p>
<p>But there’s a problem.</p>
<p>I don’t know how to change 32bit operation to 4bit operation.</p>
<p>Now the value itself is quantified through clipping, but it still operates at 32bit in GPU or CPU.</p>
<p>I’ve been looking for ways to use OpenCL or CUDA, but I haven’t found a clue.</p>
<p>I really don’t know where to approach this to solve.</p>
<p>How can I change a 32-bit operating system from CPU and GPU to a 4-bit or 8-bit operating system?</p>
<p>I’d really appreciate it if you let me know.</p>
Pytorch shared memory behavior in multi processing

rui_zhang_331: <p>Hi , I am using multiprocessing framework to speed up the CPU processing data into tensor following the best practice on<br>
<a href="https://pytorch.org/docs/stable/notes/multiprocessing.html" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/docs/stable/notes/multiprocessing.html</a></p>
<p>suppose I have list  <code>[1,2,...32]</code>  dataset, I have 2 outer process, each process with a 4 worker data loader, so</p>
<ul>
<li>process 1 will load</li>
</ul>
<pre><code class="lang-auto">[1,2,3,4]
...
[13,14,15,16]
</code></pre>
<ul>
<li>process 2 will load</li>
</ul>
<pre><code class="lang-auto">[17,18,19,20]
...
[29,30,31,32]
</code></pre>
<p>so in each process, I have 4 worker data loader, so let’s see one data loader in process1 will only load<br>
<code>[1,2,3,4]</code>.</p>
<p>all these process and multi worker dataloader is to speed up the data loading and data cooking, however, I only have one GPU and one model, so in this case, should I use <code>share_memory()</code> ? so each process will just process partial dataset, but their tensor will be shared?</p>
Can't use from_blob to construct tensor on GPU in C++

farmersrice: <p>I’m attempting to construct a tensor directly on the GPU from a float array. When I generate a random tensor and pass it the same <code>TensorOptions</code> item, it successfully generates on the GPU, but when I do the same thing with <code>from_blob</code>, it gives me an error.</p>
<p>Minimum working example:</p>
<pre><code class="lang-auto">#include &lt;bits/stdc++.h&gt;
#include &lt;torch/torch.h&gt;

int main() {
	auto options = torch::TensorOptions().device(torch::kCUDA);

	float temp[12];
	for (int i = 0; i &lt; 12; i++) {
		temp[i] = i;
	}

	torch::Tensor cudaTest = torch::rand({12}, options); //works 
	std::cout &lt;&lt; cudaTest &lt;&lt; std::endl;

	cudaTest = torch::from_blob(temp, {12}, options); //doesn't work
	std::cout &lt;&lt; cudaTest &lt;&lt; std::endl;  
}
</code></pre>
<p>Output:</p>
<pre><code class="lang-auto"> 0.9597
 0.6517
 0.5960
 0.4641
 0.9111
 0.4068
 0.5783
 0.0447
 0.2747
 0.7274
 0.0227
 0.7498
[ CUDAFloatType{12} ]
terminate called after throwing an instance of 'c10::Error'
  what():  CUDA error: invalid argument (getDeviceFromPtr at ../aten/src/ATen/cuda/CUDADevice.h:13)
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt; const&amp;) + 0x6a (0x7fcdfa928aaa in /usr/local/libtorch/lib/libc10.so)
frame #1: &lt;unknown function&gt; + 0x390babf (0x7fcdad161abf in /usr/local/libtorch/lib/libtorch_cuda.so)
frame #2: at::from_blob(void*, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, c10::TensorOptions const&amp;) + 0x823 (0x5580868fcd03 in ./a)
frame #3: torch::from_blob(void*, c10::ArrayRef&lt;long&gt;, c10::TensorOptions const&amp;) + 0x8c (0x5580868fdaac in ./a)
frame #4: main + 0x189 (0x5580868f2e69 in ./a)
frame #5: __libc_start_main + 0xf3 (0x7fcda8e040b3 in /lib/x86_64-linux-gnu/libc.so.6)
frame #6: _start + 0x2e (0x5580868f398e in ./a)

Aborted (core dumped)
</code></pre>
<p>Any idea what the problem could be? I don’t want to construct in regular memory and use .to(device) since it seems to be a bit slower.</p>

glaringlee: <p><a class="mention" href="/u/farmersrice">@farmersrice</a><br>
check this issue.<br>
‘<a href="https://github.com/pytorch/pytorch/issues/15426" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/15426</a>’, I think our document need update.<br>
You can not copy memory from CPU to GPU directly. Your temp[] is not on GPU.<br>
I think you have to use .to(device) at this point.</p>
CPU_CAPABILITY_AVX2 definition

rohithkrn: <p>Where is <code>CPU_CAPABILITY_AVX2</code> defined? It is used in <code>vec256*</code> headers. (e.g <a href="https://github.com/pytorch/pytorch/blob/d1352192e21001521035dacc0eda574b58224198/aten/src/ATen/cpu/vec256/vec256.h#L53" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/d1352192e21001521035dacc0eda574b58224198/aten/src/ATen/cpu/vec256/vec256.h#L53</a>). Is it defined in system headers? I am trying understand if  my ops are making use of AVX instructions?</p>

glaringlee: <p><span class="mention">@rohithken</span><br>
These flags are generated when compiling the source files within aten/src/ATen/native/cpu.<br>
The code is here,<br>
‘<a href="https://github.com/pytorch/pytorch/blob/eebd492dcfb8d4152a5ba4240486f3df5aaf112d/cmake/Codegen.cmake#L105" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/eebd492dcfb8d4152a5ba4240486f3df5aaf112d/cmake/Codegen.cmake#L105</a>’<br>
cpu folder will be compiled with all these flags, pytorch automatically detect whether your CPU support those instruction sets.</p>
<p>I think if you want to see whether your op used the AVX, you don’t need to check these flags since as long as your cpu support AVX/2, pytorch will compile  with <code>CPU_CAPABILITY_AVX2</code>/2 for sure. What you can do is debug your op using eg. dbx, gdb, setup some breakpoint and print out the<br>
Assembly code, see whether your op really call the AVX/2 instructions.</p>
Hadamard Product Libtorch

Hiperdyne19012: <p>How to get Hadamard product aka element-wise mul of two Libtorch torch tensors in C++?</p>

glaringlee: <p><a class="mention" href="/u/hiperdyne19012">@Hiperdyne19012</a><br>
You can just use operator *<br>
eg. torch::Tensor a = torch::tensor(2, 3, 4)<br>
torch::Tensor b = torch::tensor(2, 3, 4)<br>
torch c = a * b;<br>
c will be [4, 9, 16]</p>
Libtorch Pytorch Interpolate Differences

Hiperdyne19012: <p>I am getting considerable differences in heatmaps generated by Libtorch <code>torch::nn::functional::</code> <code>interpolate</code> vs Python torch.nn.functional.interpolate. On C++ I am using kBilinear setting and on Python I using mode=‘bilinear’.<br>
Align corners is set to false for both as well.</p>

glaringlee: <p><a class="mention" href="/u/hiperdyne19012">@Hiperdyne19012</a><br>
Can you post more details? Post a sample code and result will be very helpful, thanks.</p>

Hiperdyne19012: <p>I have just learned that it is non-deterministic.</p>
Hot to train neck and head only in warmup

zyc: <p>How to config the optimizer so that it only trains neck and head during warm up, and with backbone in training.</p>

ptrblck: <p>You could freeze the backbone parameters by setting their <code>.requires_grad</code> attributes to <code>False</code> during the warmup iterations, and then to <code>True</code> during the complete training.<br>
This would make sure that these parameters won’t get any gradients and the optimizer won’t update them in its <code>step()</code> operation.</p>

zyc: <p>Thank you for your answer. What about add the param group of backbone to optimiser after warm up is finished? Which one is a better practice?</p>

ptrblck: <p>Both should work equally well and it might just depend on your personal coding style / preference.</p>

flyingmrwang: <p>I have the same problem, which is unsolved unfortunately <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"><br>
I have modelA as backbone and modelB as aux output, which means my whole model has two outputs. What I wanna do is freeze pre-trained backbone and train modelB only. But either way that you suggested would touch the weights in backbone. Any suggestion?</p>

ptrblck: <p>If <code>modelB</code> is “on top” of <code>modelA</code>, you still could freeze the parameters of <code>modelA</code> and just train <code>modelB</code>.<br>
Why would the <code>modelA</code> weights be touched?</p>

flyingmrwang: <p>Thanks for reply. Yes, it is exactly as you said. This is a piece of code, which may help to locate the problem.<br>
(BTW, pytorch version is 1.4.0)</p>
<h1>create graph and load weigths for modelA</h1>
<p>checkpoint = torch.load(model_path, map_location=device)<br>
modelA = BackboneNet().to(device)<br>
modelA.load_state_dict(checkpoint[‘backbone’])<br>
modelB = AuxNet().to(device)</p>
<h1>freeze modelA</h1>
<p>for param in modelA.parameters():<br>
param.requires_grad = False</p>
<h1>asign modelB to train</h1>
<p>optimizer = torch.optim.Adam(<br>
[{<br>
‘params’: modelB.parameters()<br>
}],<br>
lr=base_lr,<br>
weight_decay=weight_decay)</p>

flyingmrwang: <p>It seems like add model.eval() after set requires_grad works! Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a> for help~</p>
Why different coding style obtain different results in Pytorch?

wayne1226: <aside class="onebox stackexchange">
  <header class="source">
      <a href="https://stackoverflow.com/questions/62420826/why-different-coding-style-obtain-different-results-in-pytorch" target="_blank" rel="nofollow noopener">stackoverflow.com</a>
  </header>
  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/11789721/wayne" target="_blank" rel="nofollow noopener">
    <img alt="Wayne" src="https://lh5.googleusercontent.com/-dXCr1AC52dc/AAAAAAAAAAI/AAAAAAAAAAA/ACHi3rdiYpNK5qWI5MLnjOelBQAU2t9gyA/photo.jpg?sz=128" class="thumbnail onebox-avatar" width="128" height="128">
  </a>
<h4>
  <a href="https://stackoverflow.com/questions/62420826/why-different-coding-style-obtain-different-results-in-pytorch" target="_blank" rel="nofollow noopener">Why different coding style obtain different results in Pytorch?</a>
</h4>

<div class="tags">
  <strong>python, deep-learning, neural-network, pytorch</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stackoverflow.com/users/11789721/wayne" target="_blank" rel="nofollow noopener">
    Wayne
  </a>
  on <a href="https://stackoverflow.com/questions/62420826/why-different-coding-style-obtain-different-results-in-pytorch" target="_blank" rel="nofollow noopener">03:07AM - 17 Jun 20 UTC</a>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


wayne1226: <p>I finally find that it is because I haven’t shuffle the datasets.</p>
How to train a patch based net

Harold_Ginger: <p>Hi, I am a beginner in deep learning. For the past week I have been wandering how to train a patch-based neural network, since my images are quite big, my basic idea to divide it into patches to utilize the high resolution. My current thought is to design an end to end net, take the patches from one image, go through a feature extractor part then feed into a classifier part, finally output the label of this whole image without patch label.<br>
What I am asking is that how do I design the net? Say I cut one image into 35 patches, should I just concatenate the features from feature extractor part every 35 times and then feed into the classifier part? Can you provide some examples? I can’t find any on google. Thanks a lot in advance.</p>
TorchScript compatibility between pytorch versions

antoche: <p>Hi,</p>
<p>I am trying to use libtorch in a C++ application that’s restricted to the gcc-4.8 ABI, which forces me to link against libtorch-1.2.0 (I was told newer version of libtorch aren’t compatible with that ABI).</p>
<p>This application needs to load a pre-trained model. That model was trained and saved from a python script that needs to use pytorch-1.3.0 or newer.</p>
<p>Trying to load the serialized TorchScript model aborts the program with this error:</p>
<pre><code>terminate called after throwing an instance of 'c10::Error'
  what():  [enforce fail at inline_container.cc:137] . PytorchStreamReader failed closing reader: file not found 
</code></pre>
<p>This happens both from the C++ and the python API. Is this a compatibility issue?</p>
<p>This is easily reproducible: from an environment with either pytorch-1.3.1 or 1.5.0, run</p>
<pre><code class="lang-auto">import torch

class TestModule(torch.nn.Module):
    def __init__(self, N, M):
        super().__init__()
        self.weight = torch.nn.Parameter(torch.rand(N, M))

    def forward(self, x):
        return self.weight.mv(x)

my_module = TestModule(10,20)
test_module = torch.jit.script(my_module)

test_module.save("test_module.pt")
</code></pre>
<p>Then from an environment with pytorch-1.2.0:</p>
<pre><code class="lang-auto">import torch
m = torch.jit.load('test_module.pt')
</code></pre>
<p>If the file was written from 1.5.0, I get</p>
<pre><code>RuntimeError: version_number &lt;= kMaxSupportedFileFormatVersion INTERNAL ASSERT FAILED at ../caffe2/serialize/inline_container.cc:131, please report a bug to PyTorch. Attempted to read a PyTorch file with version 2, but the maximum supported version for reading is 1. Your PyTorch installation may be too old.
</code></pre>
<p>which is clearly a compatibility issue. But if the file was written from 1.3.1, I get the “file not found” error above (note this is a process abort, not an exception, from both C++ and python API).</p>
<p>I could not find documentation on TorchScript file compatibility across versions. Is that information available somewhere?</p>
<p>Thanks,<br>
A.</p>

eellison: <p>Hi, we maintain backwards compatibility but not forwards compatiblity. New versions will work with older models, but older versions will not work with newer models.</p>
K-means plotting torch tensor

alex_gilabert: <p>Hello</p>
<p>This is a home-made implementation of a K-means Algorith for Pytorch.<br>
I have a tensor of dimensions [80, 1000] that represents the centroids of the cluster that go changing until they are fixed values.</p>
<p>Also there are the labels of the features that are considered the “centers” in the variable called  “indices_”.</p>
<p>I am having some issues when i want to represent the tensor. I am just considering 2 main classes.</p>
<p>I have also tried the scatter but have not very much info on how i could do a propper plot.</p>
<p>Here is the code below:</p>
<pre><code class="lang-auto">def k_means_torch(dictionary, model):
    centroids = torch.randn(len(dictionary), 1000).cuda()
    dist_centroids = torch.cdist(dictionary,centroids, p=2.0)
    (values, indices) = torch.min(dist_centroids, dim=1)
    centroids_new = dictionary[indices]
    
    while True:
        dist_centroids_loop = torch.cdist(dictionary,centroids_new, p=2.0)
        (values_, indices_) = torch.min(dist_centroids_loop, dim=1)
        new_centers = dictionary[indices_]
        torch.allclose(torch.tensor([[1., 2.], [3., 4.]]), torch.tensor([[1., 2.000000001], [3., 4.]]))


        a = torch.all(torch.lt(torch.abs(torch.add(centroids_new, -new_centers)), 1e-5))
        #print(a)
        if (a == True):
          break
        rng = np.random.RandomState(0)
        centroids_new = (new_centers + centroids_new)/2
        #plt.scatter(centroids_new[0].cpu().numpy(), dictionary[0].cpu().numpy(),   alpha=0.5)
        #plt.show()

    return centroids_new
</code></pre>
<p>¿Any ideas on how i could plot this? Maybe use another function on the matplotlib?</p>

ptrblck: <p>It seems you are working with a 1000-dimensional feature space, which would be hard to plot.<br>
You could start by plotting only two dimensions and use <code>plt.scatter</code> and check the results first.<br>
To reduce your feature space you could use e.g. PCA, but I’m not sure how this would affect the kmeans centroids and data. Might still be worth a try.</p>

alex_gilabert: <p>Yeah… What i wanted to implement unfortunately is not possible. I have had a look to PCA but its not exactly what i was looking for. Anyways thanks for the idea <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
How to train only about a specific indices in DistributedSampler?

jwchoi384: <p>Hi all,<br>
I am trying DistributedSampler and Dataloader for my custom dataset and the model.<br>
But I’m in trouble using specific indice in Dataloader.<br>
I want to train my model by random shuffling only for a specific indices from the whole data.</p>
<p>For example, Here is my example code:</p>
<blockquote>
<p>from torch.utils.data.distributed import DistributedSampler as DistributedSampler</p>
</blockquote>
<blockquote>
<p>train_dataset = MyDataset(args.data_dir, split=‘xxx’, input_size=input_size)<br>
train_sampler = DistributedSampler(train_dataset, num_replicas=args.gpus, rank=gpu)<br>
train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=int(args.batch_size/args.gpus), shuffle=False, <em><strong>sampler=train_sampler</strong></em>, collate_fn=train_dataset.collate_fn, num_workers=int(args.workers/args.gpus), pin_memory=True)</p>
</blockquote>
<p>I think I need to modify part <em><strong>sampler</strong></em>.<br>
But I have no idea how to make this function to use specific indices.<br>
Is there a way to implement the function I want there?</p>
<p>P.S.<br>
When I used Dataparallel, I can use specific indices like below:</p>
<blockquote>
<p>specific_indices = [10, 99, 5, 8, …]<br>
train_dataset = MyDataset(root=xxx, transform=xxx)<br>
train_loader = DataLoader(train_dataset, batch_size=args.batch_size, num_workers=args.num_workers,  <em><strong>sampler=SubsetRandomSampler(specific_indices)</strong></em>, collate_fn=detection_collate, pin_memory=True)</p>
</blockquote>
<p>Thank you in advance.</p>

Ajay_Babu: <p>I also had the same issue with Weighted Random Sampler and I found <a href="https://discuss.pytorch.org/t/how-to-use-my-own-sampler-when-i-already-use-distributedsampler/62143/8">This reply</a> from <a class="mention" href="/u/ptrblck">@ptrblck</a> to be the best one<br>
So based on the source code it would change like this</p>
<pre><code class="lang-auto">class DistributedSubset(Sampler):
    """
    https://discuss.pytorch.org/t/how-to-use-my-own-sampler-when-i-already-use-distributedsampler/62143/8
    
    """
    #It’s common to call the total number of processes the world size

    def __init__(self,indices):
        self.indices = indices
    
    def __iter__(self):
        # deterministically shuffle based on epoch
        x=[self.indices[i] for i in torch.randperm(len(self.indices))]
        return iter(x)
    def __len__(self):
        return len(self.indices)

    def set_epoch(self, epoch):
        self.epoch = epoch
</code></pre>
<p>Just make sure you use set_epoch() for every epoch to shuffle the data<br>
Let me know if it was helpful !</p>

jwchoi384: <p>Thank you for your answer!<br>
I think it’s a little different, but I’ll try it.</p>
Can the type of a gradient only be a floating type in new versions of Pytorch?

rahul003: <p>When I return a Long gradient, Pytorch 1.5 complains with this error that it should be a floating type. <a href="https://github.com/pytorch/pytorch/blob/54c05fa34e1fbbb5096746a8ae92e27a08116de4/torch/csrc/autograd/engine.cpp#L610" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/54c05fa34e1fbbb5096746a8ae92e27a08116de4/torch/csrc/autograd/engine.cpp#L610</a></p>
<p>However when I cast this to float gradient in Pytorch 1.2, it complains that expected Long gradient but received float.</p>
<p>Has behavior changed over versions?</p>

albanD: <p>Hi,</p>
<p>Gradients for Long Tensors were never really supported. And were raising errors if used at random places.<br>
We now unified this and properly raise an error when they are created (as you cannot use them anyway) so that the user can easily pinpoint the issue.</p>
Adding layers to nn.RNN

Chiara: <p>This is my code:</p>
<pre><code class="lang-auto">class gwRNN(nn.Module):
  #declare a few variables with initialisation function:
    def __init__(self, batch_size, n_steps, n_inputs, n_neurons, n_real_outputs, num_layers):
        super(gwRNN, self).__init__()
        
        self.batch_size = batch_size
        self.n_steps = n_steps
        self.n_inputs = n_inputs
        self.n_neurons = n_neurons
        self.n_real_outputs = n_real_outputs
        self.num_layers = num_layers

        # declare basic RNN layer
        self.basic_rnn = nn.RNN(self.n_inputs, self.n_neurons, num_layers, nonlinearity = 'relu', dropout=0)
        # declare fully-connected layer 
        self.FC = nn.Linear(self.n_neurons, self.n_real_outputs, bias= False) #nn.linear Applies a linear transformation to the incoming data
       
    def init_hidden(self,): # returns hidden state with zero values
        # (num_layers, batch_size, n_neurons)
        return (torch.zeros(self.num_layers, self.batch_size, self.n_neurons))
        
    def forward(self, X):
        # transforms X to dimensions: n_steps X batch_size X n_inputs
        X = X.permute(1, 0, 2) 

        # The data flows through the RNN layer and then through the fully-connected layer.        
        self.batch_size = X.size(1) # size of 2nd element of X
        self.hidden = self.init_hidden()
        
        lstm_out, self.hidden = self.basic_rnn(X, self.hidden)   #SEE ABOVE for self.basic_rnn and nn.RNN !!    
        out = self.FC(self.hidden) # output represent the log probabilities of the model.  
        
        
        return out.view(-1, self.n_real_outputs) # batch_size X n_output
</code></pre>
<p>I’d like to add stack rnns by changing num_layers, but when I do this my code doesn’t work as it returns out with shape (n_layers*batch size) X n_output instead of batch_size X n_output</p>

harsha_g: <aside class="quote no-group" data-username="Chiara" data-post="1" data-topic="87621">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/chiara/48/25931_2.png" class="avatar"> Chiara:</div>
<blockquote>
<pre><code class="lang-auto">        # transforms X to dimensions: n_steps X batch_size X n_inputs
        X = X.permute(1, 0, 2) 

        # The data flows through the RNN layer and then through the fully-connected layer.        
        self.batch_size = X.size(1) # size of 2nd element of X
</code></pre>
</blockquote>
</aside>
<p>You permuted <code>X</code> to make batch the second dimension. Going by that, the <code>self.hidden</code> has to be permuted back to make batch the first dimension before feeding to <code>self.FC</code>.</p>

googlebot: <p>just slice it to the last layer: self.hidden[-1]</p>
Training Domain Adaptation Algorithm for Activity Recognition task

qasimgilani: <p>Hi,<br>
I am trying to implement domain adaptation for activity recognition in PyTorch. For this task, I am using PAMAP2 &amp; Opportunity data set. I loaded the both datasets in PyTorch by reading “h5py” files. In domain adaptation we need both data set during training process.<br>
I do not understand how I can use both datasets at the same time in PyTorch during training…</p>
<p>Regards</p>
Use of .clamp(min=0) in BCEwithlogitsloss

singhvishal0209: <p>Why is .clamp(min=0) used to find the max_val in the implementation of binary cross-entropy with logits loss, and why not use torch.max() function instead.<br>
Also, why do they have used negative sign in front of (-input).clamp(min=0)</p>
<p>I have not clearly understood the first line of the code.</p>
<p>in PyTorch as defined <a href="https://github.com/pytorch/pytorch/blob/d5a7e304fa858136eeffa1ebe35e22ed886bf256/torch/nn/functional.py#L859" rel="nofollow noopener">here </a></p>
<pre><code class="lang-auto">max_val = (-input).clamp(min=0)
loss = input - input * target + max_val + ((-max_val).exp() + (-input - max_val).exp())
</code></pre>
<p>below I have shown how torch.max() should be used instead of .clamp()</p>
<pre><code class="lang-auto">input = torch.rand([2, 5])
max_val1 = (-input).clamp(min=0)
max_val2 = torch.max(input,1)
print("input:",input)
print("max_val1:", max_val1,"\nmax_val2:", max_val2)
</code></pre>
<pre><code class="lang-auto">input: tensor([[0.8552, 0.3244, 0.2635, 0.1885, 0.1494],
        [0.2624, 0.8370, 0.9788, 0.8573, 0.8751]])
max_val1: tensor([[0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0.]]) 
max_val2: torch.return_types.max(
values=tensor([0.8552, 0.9788]),
indices=tensor([0, 2]))
</code></pre>
<p>here .clamp() doesn’t really finds the max value but instead makes all the negative value to positive and positive values to 0, that’s it.</p>
How handle memory allocation error

ahmadreza9: <p>Hello, what are your suggestions for handling blow error?<br>
X_train = (X_train - X_mean) / X_std<br>
MemoryError: Unable to allocate 3.56 GiB for an array with shape (106640, 1, 20, 224) and data type float64</p>
<pre><code class="lang-auto">    X_mean = np.average(X[:num_train + num_test], axis=0)
    X_std = np.std(X[:num_train + num_test], axis=0)

    X_train = (X_train - X_mean) / X_std
</code></pre>

albanD: <p>Hi,</p>
<p>This is the numpy OOM error right?<br>
Well in general, there isn’t much you can do but change your program to use less memory or buy new ram <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"></p>

ahmadreza9: <p>yes sir, Can I batch input and calculate its average to reduce memory?</p>

albanD: <p>It seems like the error is with the temporary values.<br>
If you switch your arrays to torch Tensor, you can do <code>X_train.sub_(X_mean).div_(X_std)</code>.<br>
Not sure if these exist for numpy though.</p>
Problem in cross entropy loss

Mayuresh_Anand: <p>import torch<br>
import torch.nn as nn<br>
loss = nn.CrossEntropyLoss()<br>
inpt = torch.randn(10,3,requires_grad=True)<br>
target = torch.empty(10, dtype=torch.long).random_(5)<br>
output = loss(inpt, target)<br>
print(inpt)<br>
print(target)<br>
print(output)</p>
<h2>I get following error don’t know why?</h2>
<p>IndexError                                Traceback (most recent call last)<br>
 in ()<br>
4 inpt = torch.randn(10,3,requires_grad=True)<br>
5 target = torch.empty(10, dtype=torch.long).random_(5)<br>
----&gt; 6 output = loss(inpt, target)<br>
7 print(inpt)<br>
8 print(target)</p>
<p>3 frames<br>
/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py in nll_loss(input, target, weight, size_average, ignore_index, reduce, reduction)<br>
2113                          .format(input.size(0), target.size(0)))<br>
2114     if dim == 2:<br>
-&gt; 2115         ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)<br>
2116     elif dim == 4:<br>
2117         ret = torch._C._nn.nll_loss2d(input, target, weight, _Reduction.get_enum(reduction), ignore_index)</p>
<p>IndexError: Target 3 is out of bounds.</p>

Nikronic: <p>Hi,</p>
<aside class="quote no-group" data-username="Mayuresh_Anand" data-post="1" data-topic="87600">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mayuresh_anand/48/14253_2.png" class="avatar"> Mayuresh_Anand:</div>
<blockquote>
<p>inpt = torch.randn(10,3,requires_grad=True)<br>
target = torch.empty(10, dtype=torch.long).random_(5)</p>
</blockquote>
</aside>
<p>Your input data has 3 classes <code>[10, 3]</code> but output has labels from 0 to 5 so you get error for labels bigger than 2. <code>.random_(3)</code> can solve your issue.</p>
<p>Bests</p>
Extract specific outputs through pre trained classifier

Frida: <p>Does someone know how to extract outputs at specific locations? For example, I would like to get all outputs corresponding to convolution layers, including the hidden one under sequential. Any idea?<br>
Thanks <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

ptrblck: <p>You could use forward hooks to get the intermediate activations from your model as described <a href="https://discuss.pytorch.org/t/how-can-l-load-my-best-model-as-a-feature-extractor-evaluator/17254/6">here</a>. Let me know, if this would work for you or if I misunderstood the question. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

Frida: <p>Thanks a lot<br>
What if part of the layers are built in sequential? I am using the resnet18 pre-trained model of pytorch, how to call those layers?</p>

ptrblck: <p>You can index modules inside an <code>nn.Sequential</code> container via indexing:</p>
<pre><code class="lang-python">model = nn.Sequential(
    nn.Conv2d(3, 6, 3, 1, 1),
    nn.ReLU(),
    nn.Conv2d(6, 1, 3, 1, 1)
)

print(model[2]) # index second conv layer
</code></pre>
How to create multi process training with PyTorch without back-propagation and with no_grad()?

Ajay_Madhavan: <p>I am new to PyTorch and trying to understand how I should code multi-process with no_grad(). Below is the requirement.</p>
<p>I am working on converting a c++ program which was used in a paper to convert nodes to vectors, so it can be used for ML tasks. I basically initialize the weight vector of size V x d (V - number of nodes, d - embedding size) with values between -0.5 to 0.5. Then we sample nodes as u, v, and neg_v, where u &amp; v has an edge and u &amp; neg_v does not have an edge.</p>
<p>The calculation selects the weight(embedding) of respective nodes and uses the below logic for calculating scores.<br>
if we sample u, v:<br>
score = -bias<br>
score += weight[u] * weight[v]<br>
score = (1 - fast_Sigmoid(score)) * lr<br>
weight[u] += weight[v] * score<br>
weight[v] += weight[u] * score</p>
<p>if we sample u, neg_v:<br>
score = -bias<br>
score += weight[u] * weight[neg_v]<br>
score = - fast_Sigmoid(score) * lr<br>
weight[u] += weight[neg_v] * score<br>
weight[neg_v] += weight[u] * score</p>
<p>since all this are performed in a single go and we want to have faster calculation, we came across with torch.no_grad() (" Disabling gradient calculation is useful for inference when you are sure that you will not call backward(). it will reduce memory consumption") and we are using no_grad(). How to multiple-process this training along with memory sharing. Looking for some details about it!!.</p>

albanD: <p>Hi,</p>
<p>Is there a specific reason you want to go with multiprocessing and not multi threading?</p>
<p>If you really need multiprocessing, you don’t need to do anything special here to avoid gradients as nothing requires_grad anyways. So no graph will be created.</p>

Ajay_Madhavan: <p>Hi,</p>
<p>Thank you for the reply. I am trying to code for multi gpus or gpu, cpu mix. I am sharing memory with this multi cores, so I am looking to get best standards. The requirement is more like using multi process and multi threads. For instance, if the graph is big I want to run the calculation is multi gpus or gpu, cpu mix to calculate it faster with multi threads. I saw an example implementation <a href="https://github.com/dmlc/dgl/blob/master/examples/pytorch/deepwalk/deepwalk.py" rel="nofollow noopener">https://github.com/dmlc/dgl/blob/master/examples/pytorch/deepwalk/deepwalk.py</a><br>
I am trying to follow the same idea, but i just want to make sure i do it right!!</p>
Usage of register_buffer gives RuntimeError

Mohamed_Elashry: <p>Hello,<br>
I know that <code>register_buffer</code> method would create variable that won’t be updated during backprobagation. So It won’t be a problem if applied in-place operation on that variable. right?<br>
but when I did that it gives me the error of <code>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [256, 5]] is at version 3; expected version 2 instead.</code><br>
I was using custom layer.</p>
<pre><code class="lang-auto">class NoisyLinear(nn.Linear):

    def __init__(self, in_features, out_features, sigma_init=0.017, bias=True):
        super(NoisyLinear, self).__init__(in_features, out_features, bias=bias)
        self.sigma_weight = nn.Parameter(torch.full((out_features, in_features), sigma_init))
        self.register_buffer("epsilon_weight", torch.zeros(out_features, in_features))
        if bias:
            self.sigma_bias = nn.Parameter(torch.full((out_features,), sigma_init))
            self.register_buffer("epsilon_bias", torch.zeros(out_features))
        self.reset_parameters()

    def reset_parameters(self):
        std = math.sqrt(3 / self.in_features)
        self.weight.data.uniform_(-std, std)
        self.bias.data.uniform_(-std, std)

    def forward(self, input):
        self.epsilon_weight.normal_()
        bias = self.bias
        if bias is not None:
            self.epsilon_bias.normal_()
            bias = bias + self.sigma_bias * self.epsilon_bias
        return F.linear(input, self.weight + self.sigma_weight * self.epsilon_weight, bias)
</code></pre>
<p>so in the <code>forward</code> method I wanted to change <code>epsilon_weight</code> in-place using <code>noraml_</code> method but it gave me the mentioned error.<br>
But when I used <code>torch.normal</code> like this</p>
<pre><code class="lang-auto">self.epsilon_weight= torch.normal(self.epsilon_weight)
</code></pre>
<p>it worked fine. so why??</p>

albanD: <p>Hi,</p>
<p>When you do <code>self.sigma_weight * self.epsilon_weight</code>, then to compute the gradient of <code>self.sigma_weight</code> which is you Parameter, you need the value of <code>self.epsilon_weight</code>.<br>
So if you modify it inplace, after it was used in this computation, then you will see this error.</p>
<p>The out of place version creates a new Tensor containing the new value. So the original value that was used is still there and the backward can work fine.</p>
Types of loss functions

michelepy: <p>Hi guys I’m new to PyTorch and I was wondering what was the difference between:<br>
nn.BCEWithLogitsLoss and nn.functional.binary_cross_entropy_with_logits?<br>
Thanks</p>

lfolle: <p><code>nn.BCEWithLogitsLoss</code> is the class and <code>nn.functional.binary_cross_entropy_with_logits</code> is the function of the binary cross-entropy with logits loss.<br>
They both have the same results, but are used in a different way:</p>
<ol>
<li>
<code>criterion = torch.nn.BCEWithLogitsLoss(pos_weight=pos_weight)</code><br>
Then you can do <code>criterion(output, target)</code>
</li>
<li><code>loss = nn.functional.binary_cross_entropy_with_logits(output, target)</code></li>
</ol>
<p>Using <code>BCEWithLogitsLoss</code> you are able to define certain parameters when creating the loss (e.g. <code>weight </code>). <a href="https://pytorch.org/docs/master/generated/torch.nn.BCEWithLogitsLoss.html" rel="nofollow noopener">Please look at the documentation for details.</a></p>

michelepy: <p>Thank you so much, in fact I am working on a multi-class, multi-label, classification problem. In my situation, which one would you suggest me to use?<br>
Also I have a sever problem of unbalanced data, so I was thinking about using parameters such as weight.</p>

lfolle: <p>Then I would certainly advise you to use <code>criterion = torch.nn.BCEWithLogitsLoss(pos_weight=pos_weight)</code> as you can then select the weighting required for your specific dataset.</p>

michelepy: <p>Thank you so much!, One last thing, I have the weight of the classes, should I use them as pos_weight? Or it has to be computed differently, and in such case could you tell me how?</p>
How can I fetch pixel coordinates without a loop

JasonChenhx: <p><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/f/3f11149574ebf12209dfd34c3e6431f5677e2b7a.png" alt="2007_000032" data-base62-sha1="8ZUFRCcVJKUDimfwmNPfWocRIMi" width="500" height="281"><br>
I want to extract the coordinates of the colored pixels without a loop.</p>

lfolle: <p>You should be able to identify the colorized pixels by identifying their respective values.<br>
Say the dark red corresponds to a value of 1.0.</p>
<p>Then you can get the indices of red pixels this way:<br>
<code>indices = torch.nonzero(image==1.0)</code></p>

JasonChenhx: <p>Thanks!<br>
I can try your method.</p>
Intermediate tensors in NeMo

jhh3: <p>Hi,<br>
I’m looking at QuartzNet in NeMo and trying to probe some of the internal tensors. I see that I can use <code>evaluated_tensors = neural_factory.infer(tensors=[a,b,c])</code>  to run inference and return the evaulations of a,b,c, but I can’t figure out how to get a list of the intermediate activation tensors, or to get a pointer to one.  I’m looking for a method of either the neural_factory or the individual models (like jasper_encoder) that would return a list of tensors that I can pass to infer().  Any ideas?<br>
Thanks<br>
<em>Edit:</em>  I know infer() and the neural_factory object are NeMo, so maybe out of scope for this board, but the underlying model is based on a PyTorch module, so I’m hoping a general PyTorch method for getting internal activations will be useful here.  Here’s the inheritance tree for that encoder model.</p>
<pre><code class="lang-auto">[nemo.collections.asr.jasper.JasperEncoder,
 nemo.backends.pytorch.nm.TrainableNM,
 nemo.core.neural_modules.NeuralModule,
 abc.ABC,
 torch.nn.modules.module.Module,
 object]</code></pre>

ptrblck: <p>For a general PyTorch model you could use forward hooks to get the intermediate activations as described <a href="https://discuss.pytorch.org/t/how-can-l-load-my-best-model-as-a-feature-extractor-evaluator/17254/6">here</a>.  Since NeMo seems to be using a PyTorch model internally, you would have to access its layers to register the hooks.</p>

jhh3: <p>Thanks ptrblck!  That worked.  It takes a little poking around to figure out the class structure, but not too much.  For example, using your example, I was able to use your example and this line:<br>
<code>encoder.encoder[1].mconv[0].conv.register_forward_hook(get_activation('B1(a).mconv.1D'))</code><br>
to capture the output of the 1D convolution in the first block of a QuartzNet encoder model.<br>
Again, thanks for the help.</p>

jhh3: <p>After messing with this for a while, I wanted to add one caveat that I encountered.  Some layers objects, like ReLU, are re-used throughout the network.  I guess it’s any layer without parameters, but I’m not sure.  The result is that if you put a hook on a ReLU layer, like <code>encoder.encoder[1].mout[0].register_forward_hook(get_activation('B1.mout.relu'))</code> in QuartzNet, <em>it gets called for every ReLU in the whole model</em>, not just the one you wanted.  So the final result in the dictionary is actually the ReLU output for the final model output, not the ReLU output associated with the layer where you registered the hook.</p>