akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
ParameterList and ModuleList: numpy like and boolean indexing

trancelestial: <p>Hi all,</p>
<p>I was wondering if there is a way of grouping multiple Parameter class instances in a collection (i.e. ParameterList) s.t. they are registered well as Module parameters, and can be indexed with boolean mask, or an array/list of indexes (like a numpy array)? Nesting parameters in ParameterList registers them correctly, but only list-like indexing is supported.</p>
<p>If you need more specific use case info, I would be happy to provide it.</p>
<p>Thank you!</p>

ptrblck: <aside class="quote no-group" data-username="trancelestial" data-post="1" data-topic="117264">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/trancelestial/48/36784_2.png" class="avatar"> trancelestial:</div>
<blockquote>
<p>If you need more specific use case info, I would be happy to provide it.</p>
</blockquote>
</aside>
<p>I would be interested in hearing more about your use case.<br>
If I understand it correctly, you would like to be able to use something like:</p>
<pre data-code-wrap="python"><code class="lang-python">params = nn.ParamArray([[param1, param2], [param3, param4]])
p = params[0, 1]
</code></pre>

trancelestial: <p>I am sorry for this delayed reply.</p>
<p>My use case might be best described as multitask learning, where I have a network hard shared between tasks and a per-task 1-d tensor which linearly combines the shared network outputs. Let’s say I have n tasks in total, and in each batch, I have samples from k (k&lt;&lt;n) tasks so I want:</p>
<ol>
<li>
<p><em>to <strong>efficiently</strong> index the ‘collection’ of all per-task weights s.t. for an input of shape (b,d) where b is the batch size, I end up with a tensor of shape (b,t) where t is the size of per-shape tensor.</em><br>
This is where the implementation using Parameter/ModuleList, wrapping the individual Parameter(per_shape_tensor) fails, as I can’t index the ParamList with boolean mask or list of indices (as in torch or numpy arrays).</p>
</li>
<li>
<p><em>that the loss.backward() time doesn’t scale with the total number of tasks (backward computation time in a batch should only depend on the number of different tasks in the current batch i.e. gradients should not be calculated for the per-task tensors which are not used in the current batch forward computation)</em><br>
This is what prevents me from having a large Parameter wrapping a matrix with per-task weights as rows. In this way, I can efficiently index, however backward computation scales with the number of tasks.<br>
(this sems similar to the embedding layers - I tried implementing both with torch.nn.Embedding, and torch.nn.functional.embedding but both implementations seem to scale with number of tasks)</p>
</li>
</ol>
<p>I would be very thankful if you would provide me with any helpful insights for implementing this while achieving both 1) and 2).</p>
Inverse or torch.chunk (view version of torch.cat/stack)

trancelestial: <p>Hi all,</p>
<p>I was wondering if there exists an operation that given a list of (1-d) tensors, joins them in a single (2-d) tensor, s.t. the resulting tensor is a ‘view’ of the original tensors i.e. changing the values in either the original tensors or the joined one, would reflect in changes in the other?<br>
torch.cat and torch.stack seem to create a new tensor. The resulting vectors of torch.chunk are views of the original but the operation is inverse to what I need.</p>
<p>Thank you!</p>

tom: <p>No, there isn’t and it’s not supported. The JIT fusers can fuse a final torch.cat by allocating the entire tensor and then filling the parts.<br>
Your best option is to keep around the larger tensor / allocate a larger tensor in advance and then work on the parts.</p>
<p>Best regards</p>
<p>Thomas</p>

trancelestial: <p>Hi Thomas,</p>
<p>I see, thank you!</p>
<p>Working with preallocated tensor wouldn’t be feasible in my case as I would like the gradient to be calculated only for certain rows of that large tensor, which is also not possible in Pytorch if I understand well? It is possible that the gradient is set to 0 for the rows not used, but I need them to not be calculated - so that backward time doesn’t depend on the total number of rows in that large matrix, but only on the number of rows used in that operation (and their size ofc). Do you have an idea on how to achieve this?</p>
<p>I found this to be similar to torch.nn.functional.embedding, but there backward time (and also optimizer step time) depend on the total size of the values matrix.</p>
<p>Best regards,<br>
Bozidar</p>

tom: <p>I think modelling it similar to the “sparse” option for embedding is pretty much how you would manage it.<br>
I would probably try to use custom autograd.Functions with a “custom sparse representation” (i.e. you keep track of the indices of non-zero rows or somesuch and then only instantiate a tensor with these parts).</p>

trancelestial: <p>Thank you, I will try implementing it that way!</p>
AttributeError: 'torch._C.PyTorchFileReader' object has no attribute 'seek'

111220: <p>I tried to load the parameters of the model, but  reported this error</p>
<p>Code:</p>
<pre><code class="lang-auto">
from naie.datasets import get_data_reference
import torch
import torchvision.models as models
from utils import EarlyStopping
if __name__ == "__main__":
    model=models.resnet50()
    model=model.cuda()
    torch.save(model.state_dict(),'/cache/step1.pt')

    checkpoint=torch.load('/cache/step1.pt')
    model.load_state_dict(checkpoint)
    

</code></pre>
<p>Error:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "test_code.py", line 11, in &lt;module&gt;
    checkpoint=torch.load('/cache/step1.pt')
  File "/home/work/anaconda/lib/python3.6/site-packages/torch/serialization.py", line 584, in load
    return _load(opened_zipfile, map_location, pickle_module, **pickle_load_args)
  File "/home/work/anaconda/lib/python3.6/site-packages/moxing/framework/file/file_io_patch.py", line 200, in _load
    _check_seekable(f)
  File "/home/work/anaconda/lib/python3.6/site-packages/torch/serialization.py", line 309, in _check_seekable
    raise_err_msg(["seek", "tell"], e)
  File "/home/work/anaconda/lib/python3.6/site-packages/torch/serialization.py", line 302, in raise_err_msg
    raise type(e)(msg)
AttributeError: 'torch._C.PyTorchFileReader' object has no attribute 'seek'. You can only torch.load from a file that is seekable. Please pre-load the data into a buffer like io.BytesIO and try to load from it instead.
</code></pre>
<p>my pytorch version is 1.6</p>

ptrblck: <p>I cannot reproduce this issue using the provided code snippet and am not sure what might cause this error.<br>
Are you also using <code>torchvision==0.7</code>?</p>

EM_Seyfi: <p>Same here, my  pytorch is alos 1.6. I also used AMP library.<br>
I have been working with this code for so long. With pytorch 1.3 I never faced this problem.</p>
<p>torchvision.<strong>version</strong> = 0.7.0<br>
torch.<strong>version</strong>= 1.6.0<br>
cu 10.2</p>

Mutlu_Simsek: <p>I have the same problem. The same code works on local but does not work on company cloud. Did anybody find a solution?</p>
Error making a json file

Ragnor: <p>ValueError: not enough values to unpack (expected 3, got 2)</p>
<p>binary, contours, hierarchy = cv2.findContours(sub_label, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE)</p>

ptrblck: <p>I’m unsure where the json file is created, but based on the posted code it seems this issue is not PyTorch-related so e.g. StackOverflow might be a better place to post the question. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
RuntimeError: CUDA out of memory in train mode

Homa: <p>Hi<br>
I get ‘RuntimeError: CUDA out of memory’ in train mode. I can not reproduce the problem anymore,Tried all of these:<br>
. gc.collect()<br>
.torch.cuda.empty_cache()<br>
. set the batch size equal to 1<br>
What is strange, is that the EXACT same code ran fine before. When I tried to run the same code with slightly different hyperparams (doesn’t affect the model,like learning rate and decay) it breaks during the first iteration of the first epoch. Even when I try to run the same hyperparams as my first experiment, it failed. Can you hepl me?thanks.</p>

ptrblck: <p>Your GPU memory might be allocated by another process, so you could check it via <code>nvidia-smi</code> and make sure the GPU is empty before starting your training.</p>

Homa: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="123140">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>nvidia-smi</p>
</blockquote>
</aside>
<p>I’m testing this. thanks for reply</p>

saba: <p>Hi Ptrblck,</p>
<p>I hope you are well. Sorry my job is running and after 500 epoch it stops and gave me this error. would you please help me with that? I am running the code on HPC Bracewell.</p>
<pre><code class="lang-auto">RuntimeError: CUDA out of memory. Tried to allocate 1.23 GiB (GPU 0; 15.90 GiB total capacity; 14.78 GiB already allocated; 269.75 MiB free; 198.87 MiB cached)

</code></pre>

ptrblck: <p>This error indicates that your device is running out of memory and crashes.<br>
This could happen, e.g. in case you are storing unnecessary tensors and are thus increasing the memory usage throughout the training (check the memory usage during the training via <code>nvidia-smi</code> and make sure it’s constant after some iterations/epochs), or e.g. if you are working with variable input shapes and one particular batch contains especially large inputs (use a max. shape in that case), or e.g. if you are using a shared system and another user uses the same GPU and allocated memory, etc.</p>
How to assign values to nn.Linear object

Paarulakan: <p>This works but the weights and bias values are random. How can I assign values to them without training over samples?</p>
<pre><code># nn.Linear
x = Variable(torch.Tensor([[1], [4]]), requires_grad=True)
# creates a wx + b layer 
f = nn.Linear(1,1)
print(f.weight, f.bias)

y = f(x)
print(y)
</code></pre>
<p>I have tried the following to assign values to ‘weight’ and ‘bias’</p>
<pre><code>f.weight = 2.0
f.bias = 1.0


f.weight = torch.Tensor([2])
f.bias = torch.Tensor([1])


f.weight = nn.Parameter(torch.Tensor([2]))
f.bias = nn.Parameter(torch.Tensor([1]))
</code></pre>
<p>None seems to work.</p>

Tudor_Berariu: <p>If you want to operate directly on the tensors containing the parameters, you can access them using <code>f.weight.data</code> and <code>f.bias.data</code>.</p>
<p>I am not sure what is your context, but maybe <code>state_dict()</code> and <code>load_state_dict()</code> are the recommended methods to save and load values for some model’s parameters.</p>

MasterSkepticista: <p>Does this still apply now that <code>.data</code> is deprecated alongside <code>Variable</code> after PyTorch 0.4?</p>

ptrblck: <p>As you’ve mentioned, the <code>.data</code> attribute and <code>Variable</code>s are deprecated, so you could wrap the code in a <code>with torch.no_grad()</code> block and manipulate the parameters via e.g. <code>param.copy_(tensor)</code>.</p>
Tensor state in cuda kernel

gouthamvgk: <p>I am using a custom kernel where a new tensor is created in the host calling method and passed to the kernel. I am not using device synchronization at any point so the kernel is launched asynchronously and returns to python. Now if I access the elements of the created tensor, it seems the kernel is synchronised and computed values are returned. This is an expected behavior in pytorch. But how does pytorch do this?. Is there any global state maintained where every tensor’s pointer location is recorded and checked if they are in use by cuda kernel before accessing its elements? . Also if I pass a constant reference of a tensor to the cuda kernel which will be only read but not changed, will accessing its element also synchronize the kernel?</p>

ptrblck: <p>PyTorch would synchronize the code (for you), if a CPU operation is used. If I’m not mistaken a Python print would fit this definition. However, besides that you should still be careful about your custom CUDA code as you can create race conditions by missing necessary syncs.</p>
[TorchScript][@property] Transfer to ScriptedModel, can't find attribute in defined @property

RebirthT: <p>When turning model to scripted model,</p>
<blockquote>
<p>the error occur: has no attribute xxxxx</p>
</blockquote>
<p>Small Snippet:</p>
<pre><code class="lang-auto">class Base(nn.Module):
    def __init__(self):
        super(Base, self).__init__()
        self.__module_name: List[str] = []
    @property
    def module_name(self):
        return self.__module_name

model = Base()
ScriptModel = torch.jit.script(model, (input,))
.................................................
[Error]: Module 'Base' has no attribute '__module_name' : File "/xxx/xxx/xxx/Base.py"
    @property
    def module_name(self):
        return self.__module_name
               ~~~~~~~~~~~~~~~~~~~~~~~ &lt;--- HERE
</code></pre>
<p>Still can’t figure out why. Any solution will be great.<br>
Thanks!</p>

RebirthT: <p>Solved by bellow:<br>
<strong>self.__module_name → self._module_name</strong></p>
<pre><code class="lang-auto">class Base(nn.Module):
    def __init__(self):
        super(Base, self).__init__()
        self._module_name: List[str] = []
    @property
    def module_name(self):
        return self._module_name
</code></pre>
Will Momentum matrix in optimizer.state_dict() should be modified if we remove few filters from the architecure?

sarvani: <p>I’m removing filters from a convolutional neural network. So I modified the weight and bias matrices of all layers accordingly. As including ‘momentum’ in SGD optimizer will generate a momentum values matrix equivalent to each module’s (i.e Conv weight, Conv bias, BatchNorm weight etc.) shape. I modified the momentum matrix shape too. But it threw the following error</p>
<p>Traceback (most recent call last):<br>
File “/content/drive/MyDrive/lenet_prune/custom_loss2.py”, line 482, in <br>
optimizer.step()<br>
File “/usr/local/lib/python3.7/dist-packages/torch/optim/lr_scheduler.py”, line 65, in wrapper<br>
return wrapped(*args, **kwargs)<br>
File “/usr/local/lib/python3.7/dist-packages/torch/optim/optimizer.py”, line 89, in wrapper<br>
return func(*args, **kwargs)<br>
File “/usr/local/lib/python3.7/dist-packages/torch/autograd/grad_mode.py”, line 27, in decorate_context<br>
return func(*args, **kwargs)<br>
File “/usr/local/lib/python3.7/dist-packages/torch/optim/sgd.py”, line 117, in step<br>
nesterov)<br>
File "/usr/local/lib/python3.7/dist-packages/torch/optim/<em>functional.py", line 169, in sgd<br>
buf.mul</em>(momentum).add_(d_p, alpha=1 - dampening)<br>
RuntimeError: The size of tensor a (19) must match the size of tensor b (20) at non-singleton dimension 0</p>
Libtorch c++ is not compitable with opencv 4.5?

coincheung: <p>Hi,</p>
<p>My code is like this:</p>
<pre><code class="lang-auto">#include&lt;string&gt;
#include&lt;vector&gt;
#include&lt;iostream&gt;

#include&lt;torch/script.h&gt;
#include&lt;opencv2/opencv.hpp&gt;


using std::string;
using std::cout;
using std::endl;
using std::vector;
using cv::Mat;


int main() {
    // Mat im = cv::imread("/root/47036.tif");
    Mat im = cv::imread("./res.jpg");
    return 0;
}

</code></pre>
<p>And my CMakeLists.txt is like this:</p>
<pre><code class="lang-auto">
CMAKE_MINIMUM_REQUIRED(VERSION 3.8.2)

project(segmentation)

set(CMAKE_BUILD_TYPE Release)
set(CMAKE_CXX_FLAGS "-std=c++14 -O3")

set(CMAKE_PREFIX_PATH ~/build/libtorch)
set (OpenCV_DIR /opt/opencv/lib/cmake/opencv4/)
find_package(Torch REQUIRED)
find_package(OpenCV REQUIRED)


add_executable(run play.cpp)
target_link_libraries(run PUBLIC
     "${TORCH_LIBRARIES}"
    "${OpenCV_LIBRARIES}"
    )
set_property(TARGET run PROPERTY CXX_STANDARD 14)
</code></pre>
<p>libtorc is downloaded from official website, the file name is <code> libtorch-shared-with-deps-1.8.1+cu102.zip</code>, and opencv’s commit is <code>04d907fb976196df62f1259808fe98a44b874db1</code>.</p>
<p>The error message is:</p>
<pre><code class="lang-auto">
Scanning dependencies of target run
[ 50%] Building CXX object CMakeFiles/run.dir/play.cpp.o
[100%] Linking CXX executable run
CMakeFiles/run.dir/play.cpp.o: In function `main':
play.cpp:(.text.startup+0x43): undefined reference to `cv::imread(std::string const&amp;, int)'
collect2: error: ld returned 1 exit status
CMakeFiles/run.dir/build.make:133: recipe for target 'run' failed
make[2]: *** [run] Error 1
CMakeFiles/Makefile2:95: recipe for target 'CMakeFiles/run.dir/all' failed
make[1]: *** [CMakeFiles/run.dir/all] Error 2
Makefile:103: recipe for target 'all' failed
make: *** [all] Error 2
</code></pre>
<p>I see no opencv problem if I comment part about libtorch in the code and CMakeLists.txt, so opencv works well alone, but there would be problem if I use opencv together with libtorch. How to solve this problem please ?</p>

take-cheeze: <p>I don’t know what opencv you’re using but mostly they are built with c++11 ABI.<br>
Use <a href="https://download.pytorch.org/libtorch/cu102/libtorch-cxx11-abi-shared-with-deps-1.8.1%2Bcu102.zip" rel="noopener nofollow ugc">https://download.pytorch.org/libtorch/cu102/libtorch-cxx11-abi-shared-with-deps-1.8.1%2Bcu102.zip</a> instead.</p>
Add a capture hook for torch::autograd::grad()

swilson314: <p>I’m making a call to torch::autograd::grad(), and I want to store individual gradients rather than accumulating them.</p>
<p>I tried adding a hook to the model parameters via tensor::register_hook(), ala TEST(CustomAutogradTest, Hooks) which requires a call to backward() rather than autograd::grad(), but the hook only got called once, with the gradient accumulated values.</p>
<p>I’ve called through the code and it seems like what I need is to add a hook to capture.hooks_. (See code below.) To do so, I need access to the current GraphTask, which is created in Engine::execute(), via a call to run_backward(). See DistEngine::computeDependencies() for an example of adding to capture.hooks_.</p>
<p>Is there a way to add to capture.hooks_? If not, it seems I need to replace Engine::execute(), maybe subclass Engine?</p>
<p>I feel like if I could do this, this method could be employed by the users who want to create a jacobian from c++.</p>
<pre><code class="lang-auto">void Engine::evaluate_function(
    std::shared_ptr&lt;GraphTask&gt;&amp; graph_task,
    Node* func,
    InputBuffer&amp; inputs,
    const std::shared_ptr&lt;ReadyQueue&gt;&amp; cpu_ready_queue) {
  // If exec_info_ is not empty, we have to instrument the execution
  auto&amp; exec_info_ = graph_task-&gt;exec_info_;
  if (!exec_info_.empty()) {
    auto&amp; fn_info = exec_info_.at(func);
    if (auto* capture_vec = fn_info.captures_.get()) {
      // Lock mutex for writing to graph_task-&gt;captured_vars_.
      std::lock_guard&lt;std::mutex&gt; lock(graph_task-&gt;mutex_);
      for (const auto&amp; capture : *capture_vec) {
        auto&amp; captured_grad = graph_task-&gt;captured_vars_[capture.output_idx_];
        captured_grad = inputs[capture.input_idx_];
        for (auto&amp; hook : capture.hooks_) {
          captured_grad = (*hook)(captured_grad);
        }
      }
    }

</code></pre>

swilson314: <p>I’m now doubtful of this approach, I set a breakpoint on AccumulateGrad::apply() (used for grad_accumulator_) and it’s just getting called once (per tensor) to call to backward(), so it’s actually a copy rather than an accumulator.</p>
Loss function to prefer nearby classes

Koen: <p>Hey, I am looking for a loss function that prefers nearby classes, so for example I have 10 classes, and a sample has class 6, then class 5 and 7 should receive a relatively low loss, while further classes should receive an increasingly higher loss value. (bonus points if I can tweak the penalty based on how far away a class is or something)</p>
<p>(quick note, I am only just getting into all of this, so sorry if this is common knowledge, I did not manage to find it…)</p>

KFrank: <p>Hi Koen!</p>
<aside class="quote no-group" data-username="Koen" data-post="1" data-topic="123696" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/koen/48/39194_2.png" class="avatar"> Koen:</div>
<blockquote>
<p>I am looking for a loss function that prefers nearby classes, so for example I have 10 classes, and a sample has class 6, then class 5 and 7 should receive a relatively low loss, while further classes should receive an increasingly higher loss value.</p>
</blockquote>
</aside>
<p>Consider <a href="https://pytorch.org/docs/1.8.1/generated/torch.nn.MSELoss.html#torch.nn.MSELoss" rel="noopener nofollow ugc">MSELoss</a>.</p>
<aside class="quote no-group">
<blockquote>
<p>(bonus points if I can tweak the penalty based on how far away a class is or something)</p>
</blockquote>
</aside>
<p>You can always write your own loss function.  (If you implement using<br>
only differentiable pytorch tensor functions you don’t have to package<br>
it as an official <a href="https://pytorch.org/docs/1.8.1/autograd.html#torch.autograd.Function" rel="noopener nofollow ugc">torch.autograd.Function</a> nor implement explicit<br>
<code>forward()</code> and <code>backward()</code> methods.)</p>
<p>Best.</p>
<p>K. Frank</p>
Generate torchscript for Sentence-Bert with sentence inputs

aksem_esmani: <p>Hi There,</p>
<p>I’m a newbie to creating torch scripts from models. I have found <a href="https://huggingface.co/transformers/torchscript.html" rel="noopener nofollow ugc">this</a> helpful page that shows how to export a PyTorch model for ‘BertModel’ for inputs at the token level using JIT &amp; TRACE.</p>
<p>However, I want to export the <a href="https://www.sbert.net" rel="noopener nofollow ugc">SBert</a> model which is also PyTorch based. But the inputs that SBert gets are sentences.</p>
<p>Below is a sample code I have put together, my problem is that I do not know how should I create the <code>example_inputs</code> here. Shall I convert sentences directly to tensor? or tokenize them first? I appreciate your input on this. Thanks</p>
<pre><code class="lang-auto">import torch
from sentence_transformers import SentenceTransformer

sbert_model        = SentenceTransformer('paraphrase-mpnet-base-v2')
sbert_model.eval()

# Creating the trace
traced_model = torch.jit.trace(sbert_model, example_inputs = [])
torch.jit.save(traced_model, "traced_bsert.pt")
</code></pre>
How to jit annotate an instance property

Alexander_Soare: <p>Say I have something like:</p>
<pre><code class="lang-auto">import torch
from torch import Tensor
from torch import nn
import torch.nn.functional as F


class MyModule(nn.Module):
    def __init__(self, scale_factor: float):
        super().__init__()
        self.scale_factor = scale_factor

    def forward(self, x: Tensor) -&gt; Tensor:
        x = F.interpolate(x, scale_factor=self.scale_factor)
        return x


model = MyModule(5)
model(torch.zeros(16,1,20,20))

torch.jit.script(model)
</code></pre>
<p>This raises an error:</p>
<blockquote>
<p>Expected a value of type ‘Optional[float]’ for argument ‘scale_factor’ but instead found type ‘int’.</p>
</blockquote>
<p>Actually, for some reason this toy model wasn’t able to reproduce the error I get on my real model which is:</p>
<blockquote>
<p>Module ‘MyModule’ has no attribute ‘final_scale_factor’ (This attribute exists on the Python module, but we failed to convert Python type: ‘numpy.int64’ to a TorchScript type. Only tensors and (possibly nested) tuples of tensors, lists, or dictsare supported as inputs or outputs of traced functions, but instead got value of type int64… Its type was inferred; try adding a type annotation for the attribute.):</p>
</blockquote>
<p>In both cases it seems to be an issue of annotation. How would I annotate an instance property? I tried annotating in the <code>__init__</code> method like <code>self.scale_factor = torch.jit.annotate(float, scale_factor)</code> and that doesn’t help either.</p>
<p><strong>EDIT</strong> - I was able to sort my issue with <code>self.scale_factor  = torch.tensor(scale_factor)</code> then using <code>Tensor.item()</code> on the other end. But I’d still like to know the answer!</p>

googlebot: <pre><code class="lang-auto">class M:
  mutable_f : float
  const_f : jit.Final[float]
</code></pre>
Efficient implementation for input covariance

MasterSkepticista: <p>I need to compute covariance of input feature maps of a Conv2d layer.<br>
I’m using a <code>forward_pre_hook</code> to achieve this, as follows:</p>
<pre><code class="lang-python">def _forward_hook(self, module, input):
        if torch.is_grad_enabled():
            x = input[0].data
            if module not in self.m_aa:
                self.m_aa[module] = torch.zeros((x.size(1), x.size(1))).to(device)
            if isinstance(module, nn.Conv2d):
                # [N, C, H, W]
                for h in range(x.size(2)): # Parallelizable?
                    for w in range(x.size(3)):
                        self.m_aa[module] += (x[:,:,h,w].T @ x[:,:,h,w])
</code></pre>
<p>This is extremely slow. How can I better utilize the compute by parallelizing over <code>H</code> and <code>W</code> of the Kernel?</p>

eqy: <p>I think you can express this computation as either a batched matrix multiply or an <code>einsum</code> that implements a batched matrix multiply:</p>
<pre><code class="lang-auto">import time
import torch

x = torch.randn(64, 3, 64, 64, dtype=torch.double)
test = torch.randn(32, 48)
def orig(x):
    out = torch.zeros(x.size(1), x.size(1), dtype=torch.double)
    for h in range(x.size(2)):
        for w in range(x.size(3)):
            out += x[:,:,h,w].T @ x[:,:,h,w]
    return out

def new(x): # using batchmatmul
    n = x.size(0)
    c = x.size(1)
    permuted = x.permute(2, 3, 0, 1) # move matmul dimensions to end
    batched = permuted.reshape(-1, n, c) # combine h,w dimensions to be batched
    temp = torch.matmul(batched.permute(0, 2, 1), batched) # permute is like transpose here
    return torch.sum(temp, dim=0) # sum along batch (h,w) dimension

def new2(x): # using einsum
    temp = torch.einsum('kchw,kdhw-&gt;cdhw', x, x)
    return torch.sum(temp, dim=[2,3])

# warmup runs
for i in range(3):
    out = orig(x)
    out2 = new(x)
    out3 = new2(x)

print(torch.allclose(out, out2))
print(torch.allclose(out, out3))

t1 = time.time()
out = orig(x)
t2 = time.time()
out2 = new(x)
t3 = time.time()
out3 = new2(x)
t4 = time.time()

print("original", t2-t1)
print("batched matmul", t3-t2)
print("einsum", t4-t3)
</code></pre>
<p>However, you might want to time these implementations on your own system as I think the speedup is shape dependent (needs large enough h, w).</p>
Determine AnyModule type, need to use get<> + catch?

swilson314: <p>I’m recreating the python method from “Deep Learning with Pytorch” p. 295, shown below.</p>
<p>I’m curious about the recommended c++ method for duplicating “type(m)”. As far as I can tell, given an AnyModule, the only way I can determine the underlying type is to attempt a cast via get&lt;&gt;. My c++ code is below.</p>
<p>Is there any way to determine the type wo suffering a throw/catch?</p>
<pre><code class="lang-auto">def _init_weights(self):
for m in self.modules():
if type(m) in {
nn.Linear,
nn.Conv3d,
}:
nn.init.kaiming_normal_(
m.weight.data, a=0, mode='fan_out', nonlinearity='relu',
)
if m.bias is not None:
fan_in, fan_out = \
nn.init._calculate_fan_in_and_fan_out(m.weight.data)
bound = 1 / math.sqrt(fan_out)
nn.init.normal_(m.bias, -bound, bound)
</code></pre>
<pre><code class="lang-auto">    for (auto it = m_Model-&gt;begin(); it != m_Model-&gt;end(); ++it)
	{
		// Can't find any way to check type of AnyModule except get&lt;&gt; with potential throw.
		// https://pytorch.org/cppdocs/api/classtorch_1_1nn_1_1_any_module.html
		// type_info() deleted function. 
		// auto info = it-&gt;type_info();
		try
		{
			// throws: "Attempted to cast module of type class torch::nn::Conv1dImpl to type class torch::nn::LinearImpl"
			auto m = it-&gt;get&lt;torch::nn::Linear&gt;();
			torch::nn::init::kaiming_normal_(m-&gt;weight.data(), 0.0,torch::kFanOut, torch::kTanh);
			if (m-&gt;bias.numel())
			{
				auto Fan = torch::nn::init::_calculate_fan_in_and_fan_out(m-&gt;weight.data());
				auto FanOut = std::get&lt;1&gt;(Fan);
				auto Bound = 1.0/sqrt(FanOut);
				torch::nn::init::normal_(m-&gt;bias, -Bound, Bound);
			}
			continue;
		}
		catch (...)
		{
		}

</code></pre>

Matej_Kompanek: <p>You could try using dynamic_pointer_cast, for example</p>
<pre><code class="lang-auto">void InitWeights()
{
	for (auto&amp; m : modules(false))
	{
		auto conv3dPtr = std::dynamic_pointer_cast&lt;torch::nn::Conv3dImpl&gt;(m);

		if (conv3dPtr != nullptr)
		{
			torch::nn::init::kaiming_normal_(conv3dPtr-&gt;weight, 0.0, torch::kFanOut, torch::kReLU);
		}

		auto linearPtr = std::dynamic_pointer_cast&lt;torch::nn::LinearImpl&gt;(m);

		if (linearPtr != nullptr)
		{
			torch::nn::init::kaiming_normal_(linearPtr-&gt;weight, 0.0, torch::kFanOut, torch::kReLU);
		}
	}
}
</code></pre>

swilson314: <p>Perfect. Thanks so much. My attempts at casting didn’t include the “Impl”.</p>
Difference between criterion.backward et tensor.backward

twister9458: <p>Hello all;</p>
<p>I can understand the use of:<br>
loss.backward with loss = BCELoss()(pred, target), as it hook gradients in each leaf of the graph, so that the step() call implements the descent.</p>
<p>But I don’t understand the use of tensor.backward in the following code:</p>
<pre><code class="lang-auto">def generate_gradients(self, class, layer_name):
        activation = self.intermediate_activations[layer_name]
        activation.register_hook(self.save_gradient)
        logit = self.output[:, class]
        logit.backward(torch.ones_like(logit), retain_graph=True)
        # gradients = grad(logit, activation, retain_graph=True)[0]
        # gradients = gradients.cpu().detach().numpy()
        gradients = self.gradients.cpu().detach().numpy()
        return gradients
</code></pre>
<p>I’m trying to reimplement the TCAV paper with PyTorch. (TCAV ie Testing with Concept Activation Vectors), and the objective is to calculate gradients of inputs at a given layer.</p>
<p>Thank you very much</p>

albanD: <p>Hi,</p>
<p>Writing <code>logit.backward(torch.ones_like(logit), retain_graph=True)</code> is equivalent to <code>logit.sum().backward(retain_graph=True)</code> in case that makes more sense for you?</p>

Luesch: <p>hello,</p>
<p>When you call .backward pytorch automatically sets the starting gradient for the backwardpass to a scalar of 1 but if you want to set this starting gradient to something different than the default 1 you need to pass it into the .backward method itself (in this case the torch.ones_like(logit)) . As albanD said doing .sum().backward() would be the same .In this case the torch.ones_like(logit) is the derivative of the .sum() operation so it would probably make it cleaner and more redable to just call a .sum() on the logits ,so logit.sum().backward(retain_graph=True).</p>
<p>I hope that i was able to help you a bit and im sorry if something i said is wrong but i think it should be alright.</p>
<p>have a nice day <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>edit:<br>
if your question is why you call .backward on a tensor then the answer is a bit simpler.<br>
Every tensor in a graph can be the starting point for a backwardpass. So when you call<br>
.backward on the return of for example BCELoss() you dont call it on the loss object but on<br>
the returned tensor itself (so youre always calling .backward on a tensor).</p>
Load part of pretrained model with strict=False in load_state_dict

Evangelos_Kazakos: <p>Hi,</p>
<p>I have a pretrained model and I want to load just part of it in my model as Im replacing the classification layer. What Im doing is:</p>
<p>model.load_state_dict(state_dict, strict=False)</p>
<p>I need just a heads up that this is the correct way to go. Thank you very much.</p>

Zuanazzi: <p>That sounds about right. strict = False should load all layers in the intersection of the pretrained model and your model.</p>
<p>It won’t do any magic, so the dimensions must still match.</p>

Jaideep_Valani: <p>you can try this way simple and cool</p>
<pre><code class="lang-auto">current_model=net.state_dict()
keys_pre=torch.load('',map_location=device)

new_state_dict={k:v if v.size()==current_model[k].size()  else  current_model[k] for k,v in zip(current_model.keys(), keys_vin['model_state_dict'].values()) 
                 }
</code></pre>
Using NCHW format as default for Glow

speryt: <p>I’d like to ask if there is any way (or plan for) adding to Glow ability to use NCHW by default instead of NHWC? For now it is said in some places that Glow’s default is NHWC, but at the same time, many times methods to generate NCHW layout are used.</p>
<p>I strongly doubt that all of the backends will use NHWC, doing conversion instead - for example convertConvToNCHWConv. Such approach will significantly affect performance by introducing all of those Transpose operations. Instead it should be allowed from the very beginning to choose which format will be used. E.g. Tensorflow allows to chose how formatting will be done.</p>

Bert_Maher: <p>You’re right that many backends will use NCHW, but the conversions usually have no runtime cost – the graph optimizer completely eliminates transposes that cancel.  I just brought up an experimental NCHW backend and no transposes were needed to run ResNet or ResNeXt :-).</p>

speryt: <p><a class="mention" href="/u/bert_maher">@Bert_Maher</a> thanks for info. That sound’s really interesting. If I get it correctly low-level IR, which is input for my backend, should not have transpose operation (or significantly reduced number of those), while at the same time using properly formatted convolution.</p>
<p>So for example for convolution that works only with NCHW I should just use conversion approach from OpenCL and it should work with high and low level optimizations taking care of improving performance? That’s really impressive <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=6" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

Bert_Maher: <p>Yep, that’s exactly right!  The OpenCL approach is a good place to look for the NCHW.  And please do let us know if you’re seeing transposes that aren’t eliminated with that approach; it’s always possible to improve the optimizer.</p>

iviarcio: <aside class="quote no-group" data-username="Bert_Maher" data-post="2" data-topic="31272">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/bert_maher/48/7722_2.png" class="avatar"> Bert_Maher:</div>
<blockquote>
<p>I just brought up an experimental NCHW backend</p>
</blockquote>
</aside>
<p>Hi <a class="mention" href="/u/bert_maher">@Bert_Maher</a>. I’m trying to create an experimental NCHW backend like you did. For this, I reused the TensorLayout modules (.cpp, .h) from the OpenCL backend, but even in LIR I still find transpose operations and the Convolution operations in lenet-mnist are with the NHWC layout. What am I doing wrong or not doing? Thanks in advance.</p>
If I write a function using 'class Name(nn.Module)' and then call it 3 times, will each call have its own parameters or will they share parameters?

Bled_Clement: <p>Ok so I create a class and within the init I define some conv blocks (which are also classes). For example:</p>
<pre><code class="lang-auto">class WienerFilter(nn.Module):
    def __init__(self):
        super(WienerFilter, self).__init__()

        self.relu = nn.ReLU()
        self.relu2 = nn.LeakyReLU()
        self.bn = nn.BatchNorm2d(1)
        self.coringa = ConvBlock(2, 1, kernel_size=1, padding=0, with_bn=False)
    
    def forward(self, I, noise_std, block_size):
        example = self.coring(I)
        return example
</code></pre>
<p>If in another class I do:</p>
<pre><code class="lang-auto">class test(nn.module):
    def __init__(self):
         self.wiener_filter = WienerFilter()
    def forward(self, input_image, std):
        WINDOW_SIZE = 16
        filteringa = self.wiener_filter(t_map, std, WINDOW_SIZE)
        WINDOW_SIZE = 16
        filteringb = self.wiener_filter(t_map, std, WINDOW_SIZE)
        # WINDOW_SIZE = 32
        # filteringc = self.wiener_filter(t_map, std, WINDOW_SIZE)
</code></pre>
<p>My question is, will ‘coringa’ be considered a new filter/convolution or will the same filter be used in each of the 3 calls of the function.</p>

albanD: <p>Hi,</p>
<p>Everytime you instantiate the class <code>WienerFilter()</code> it will call the <code>__init__</code> function and thus create a new set of Paramters.</p>

Bled_Clement: <p>Thanks, I appreciate the quick repsonse. I fat fingered my post as I was typing it out and only finished it there. You answered my question though! I know it was more of a pythonic question!</p>
Indexing tensor based on another tensor value

treadstone: <p>I have 2 tesnor and I want to create 3rd tensor based on the value of 1st tensor as index in 2nd tensor as below-<br>
a=torch.tensor([8,1,2,3,4,6,5,7,8])<br>
b=torch.tensor([0,1,2,3,4,5,6,7,8])</p>
<p>since 1st value of a=8, 2nd value of a=1… So I want b[8], b[1] and so on… like below.<br>
output= ([8,1,2,3,4,6,5,7,8])<br>
So basically output is tensor b[value of a]. How to do that?</p>

treadstone: <aside class="quote no-group" data-username="treadstone" data-post="1" data-topic="123691">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/t/b5e925/48.png" class="avatar"> treadstone:</div>
<blockquote>
<p>a=torch.tensor([8,1,2,3,4,6,5,7,8])<br>
b=torch.tensor([0,1,2,3,4,5,6,7,8])</p>
</blockquote>
</aside>
<p>No need of solution solved it - b[a]</p>
Best practices for testing many different architectures

Eti49: <p>Hi,<br>
i am currently testing two model types (LSTM, Transformers) with different input representations (Embeddings, One-Hot-Encoding) and some different transformations applied beforehand. The transformations influence the shape of the input representation (e.g. the embedding).</p>
<p>I define the model, input and transformation in a yaml file for configuration.</p>
<p>At the moment i have basically specified two models (LSTM, Transformer). Based on the configuration i load the correct transformations and input representations and pass them to the model and dataset.</p>
<p>Is this a valid approach?<br>
The alternative would be to create one model definition for each model, embedding, transformation combination. This would create some repeated code, but less parts that are dependent on each other.</p>
<p>Thanks in advance</p>

googlebot: <p>As the alternative, you’re basically describing a rewrite with high <a href="https://en.wikipedia.org/wiki/Coupling" rel="noopener nofollow ugc">coupling</a>, that is mostly inferior when you have swappable model parts using abstract interfaces. Manually repeated code is always a thing to avoid.</p>
Tensor for 'out' is on CPU, Tensor for argument #1 'self' is on CPU, but expected them to be on GPU (while checking arguments for addmm)

yoonho: <p>Hi all,</p>
<p>I tried to solve simple code based on MNIST</p>
<pre><code class="lang-auto">import numpy as np
import os
import torch
import torch.nn as nn
import torch.nn.functional as F
import matplotlib.pyplot as plt
from torchvision import datasets, transforms
dev = 'cuda'
os.environ['KMP_DUPLICATE_LIB_OK'] = 'True'

batch_size = 1
test_batch_size = 1
train_loader = torch.utils.data.DataLoader (datasets.MNIST('dataset/', train=True, download=True,
                                                           transform= transforms.Compose([transforms.ToTensor(),
                                                            ])),
                                                            batch_size = batch_size, shuffle=True)

image, label = next(iter(train_loader))
image = image.to(dev)

with torch.no_grad(): 

    flatten = image.view(1, 28 * 28).to(dev) 
    print(flatten.shape)

    lin = nn.Linear(784, 10)(flatten) 
    print(lin)

</code></pre>
<p>Error occurs at lin = nn.Linear (784, 10)(flatten) even though I designated ‘flatten’ to ‘cuda’.</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "C:/Users/yoonh/Works/main.py", line 59, in &lt;module&gt;
    lin = nn.Linear(784, 10)(flatten) # 784를 받아서 10개로 내보냄
  File "C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\linear.py", line 94, in forward
    return F.linear(input, self.weight, self.bias)
  File "C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\functional.py", line 1753, in linear
    return torch._C._nn.linear(input, weight, bias)
RuntimeError: Tensor for 'out' is on CPU, Tensor for argument #1 'self' is on CPU, but expected them to be on GPU (while checking arguments for addmm)
</code></pre>
<p>I am looking forward to see any help to solve this error.</p>
<p>Kind regards,<br>
Yoonho</p>

ptrblck: <p>Double post from <a href="https://discuss.pytorch.org/t/error-tensor-for-out-is-on-cpu-tensor-for-argument-1-self-is-on-cpu-but-expected-them-to-be-on-gpu-while-checking-arguments-for-addmm/123668/2">here</a>.</p>
Gradient computation has been modified by an Inplace. operation. Warning: Error detected in GatherBackward

ashwani-bhat: <p>Hi. I am getting the following error when running the code:</p>
<p><code>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.LongTensor [14, 1, 768]] is at version 2; expected version 1 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!</code></p>
<p>Here is the stack trace:</p>
<blockquote>
<p>File “/content/drive/MyDrive/NLP projects/Emotion-cause-pair/temp-reccon/end2end/e2e_model.py”, line 351, in train<br>
end_positions=end_positions,<br>
File “/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py”, line 722, in _call_impl<br>
result = self.forward(*input, **kwargs)<br>
File “/content/drive/MyDrive/NLP projects/Emotion-cause-pair/temp-reccon/end2end/e2e_model.py”, line 111, in forward<br>
end_logits = self.end(out, start_positions=start_positions, p_mask=p_mask)<br>
File “/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py”, line 722, in _call_impl<br>
result = self.forward(*input, **kwargs)<br>
File “/content/drive/MyDrive/NLP projects/Emotion-cause-pair/temp-reccon/end2end/beam.py”, line 69, in forward<br>
start_states = hidden_states.gather(-2, st_pos)  # shape (bs, 1, hs)</p>
</blockquote>
<p>I have narrowed down to the portion of code that is giving this error, but I am not sure what needs to be changed to correct this. Here is the code snippet responsible for  this error:</p>
<blockquote>
<p>slen, hs = hidden_states.shape[-2:]<br>
st_pos = start_positions[:, None, None].expand(-1, -1, hs)<br>
start_states = hidden_states.gather(-2, st_pos)<br>
start_states = start_states.expand(-1, slen, -1)</p>
</blockquote>
<p>Can someone please help me overcome this error. Thanks.</p>
Error: Tensor for 'out' is on CPU, Tensor for argument #1 'self' is on CPU, but expected them to be on GPU (while checking arguments for addmm)

YoonhoRoh: <p>Hi all,<br>
I’m trying to solve simple problems based on MNIST.</p>
<pre><code class="lang-auto">import numpy as np
import os
import torch
import torch.nn as nn
import torch.nn.functional as F
import matplotlib.pyplot as plt
from torchvision import datasets, transforms
dev = 'cuda'
os.environ['KMP_DUPLICATE_LIB_OK'] = 'True'

batch_size = 1
test_batch_size = 1
train_loader = torch.utils.data.DataLoader (datasets.MNIST('dataset/', train=True, download=True,
                                                           transform= transforms.Compose([transforms.ToTensor(),
                                                            ])),
                                                            batch_size = batch_size, shuffle=True)

image, label = next(iter(train_loader))

with torch.no_grad():

    flatten = image.view(1, 28 * 28).to(dev) 
    print(flatten.shape)

    lin = nn.Linear(784, 10)(flatten) 
    print(lin)
</code></pre>
<p>Even though I designated flatten to ‘cuda’, error message still comes out.</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "C:/Users/yoonh/Works/main.py", line 59, in &lt;module&gt;
    lin = nn.Linear(784, 10)(flatten) 
  File "C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\linear.py", line 94, in forward
    return F.linear(input, self.weight, self.bias)
  File "C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\functional.py", line 1753, in linear
    return torch._C._nn.linear(input, weight, bias)
RuntimeError: Tensor for 'out' is on CPU, Tensor for argument #1 'self' is on CPU, but expected them to be on GPU (while checking arguments for addmm)
</code></pre>
<p>I am looking forward to see any help. Thanks in advance.<br>
Kind regards,<br>
Yoonho</p>

ptrblck: <p>The <code>nn.Linear</code> layer is created on the CPU, so you would need to transfer it to the GPU as well:</p>
<pre><code class="lang-python">layer = nn.Linear(784, 10).to(dev)
lin = layer(flatten)
</code></pre>
Unable to install pytorch>=1.6 with CUDA 9.0

solsol: <p>Hi. unable to install pytorch version higher than ‘1.1.0’.<br>
python 3.7<br>
cuda 9.0<br>
Any help?</p>

ptrblck: <p>Could you describe your installation attempts and what kind of errors you were seeing, please?</p>

solsol: <p>Now I could install pytorch==1.7.1  using ```<br>
conda install pytorch==1.7.1 torchvision==0.8.2 torchaudio==0.7.2 cudatoolkit=9.0 -c pytorch</p>
<p>However, I got “False” by torch.cuda.is_available()</p>

ptrblck: <p>Your machine might be lacking a proper installation of NVIDIA drivers, the GPU might not be recognized (or could be too old), or other errors could prevent PyTorch from using the GPU, so we would need more information to further help out.</p>
Detach and copy computation subgraph

mazabou3: <p>I have the following sequence of operations:</p>
<pre><code class="lang-auto">x = e(s)
y = f(x)
z1 = g1(y)
z2 = g2(y)
</code></pre>
<p>The gradient from <code>z1</code> would backprop through <code>g1</code>, <code>f</code> and <code>e</code> etc. But I want the gradient from z2 to stop at <code>x</code>.<br>
I can do this by using <code>.detach()</code> on <code>x</code></p>
<pre><code class="lang-auto">z1 = g1(f(x))
z2 = g2(f(x.detach()))
</code></pre>
<p>But this means that I now make two calls to <code>f</code>. Is there a better way of doing this? Like detaching and copying the relevant part of the computation graph?</p>

ptrblck: <p>I believe one approach would be to use the new <code>inputs</code> argument in the <a href="https://pytorch.org/docs/stable/autograd.html#torch.Tensor.backward"><code>backward</code></a> operation, which allows you to specify the tensors, which should accumulate the gradients. In your case you could pass the parameters from <code>f</code> and <code>g2</code> to the <code>backward</code> operation created by <code>z2</code>. Another possibility could be to <code>detach</code> the tensor as suggested by you, <code>retain_grad()</code> on <code>x</code> during the backward pass of <code>z1</code> and pass this gradient to <code>x.backward(grad)</code> to continue the backpropagation through <code>e</code>.</p>