akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Static build throws exception during static initialization

codesniffer13: <p>I built libtorch with static libs and when I run my app I get the following:</p>
<pre><code class="lang-auto">terminate called after throwing an instance of 'c10::Error'
  what():  Could not find schema for aten::empty.memory_format
Exception raised from findSchemaOrThrow at ../aten/src/ATen/core/dispatch/Dispatcher.cpp:81 (most recent call first):
frame #0: std::function&lt;std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt; ()&gt;::operator()() const + 0x13 (0x300844f in ./bin64/SA)
frame #1: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt;) + 0x2f (0x3007cc1 in ./bin64/SA)
frame #2: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt; const&amp;) + 0x59 (0x3006ce2 in ./bin64/SA)
frame #3: c10::Dispatcher::findSchemaOrThrow(char const*, char const*) + 0x16c (0xb34a1c in ./bin64/SA)
frame #4: ./bin64/SA() [0x1063ccb]
frame #5: at::_ops::empty_memory_format::call(c10::ArrayRef&lt;long&gt;, c10::optional&lt;c10::ScalarType&gt;, c10::optional&lt;c10::Layout&gt;, c10::optional&lt;c10::Device&gt;, c10::optional&lt;bool&gt;, c10::optional&lt;c10::MemoryFormat&gt;) + 0x58 (0x10bcf0e in ./bin64/SA)
frame #6: at::empty(c10::ArrayRef&lt;long&gt;, c10::TensorOptions, c10::optional&lt;c10::MemoryFormat&gt;) + 0x11b (0xa94b7b in ./bin64/SA)
frame #7: ./bin64/SA() [0xa96e23]
frame #8: ./bin64/SA() [0xa97b89]
frame #9: ./bin64/SA() [0xa3543a]
frame #10: __libc_csu_init + 0x4d (0x39199ad in ./bin64/SA)
frame #11: __libc_start_main + 0x78 (0x7f301bfecc18 in /lib/x86_64-linux-gnu/libc.so.6)
frame #12: _start + 0x2a (0xa7dc5a in ./bin64/SA)
</code></pre>
<p>I’ve include the “whole archive” (ie. -Wl,–whole-archive) for libtorch.a and libtorch_cpu.a .</p>
<p>pytorch-v1.12.1<br>
Ubuntu 18<br>
GCC 8.5.</p>
<p>I’m guessing libtorch has 1 or more pre-built operators that are supposed to be loaded during initialization, but are not.  Does that seem right?  Any pointers where that code sits?</p>
<p>By the way, a simple app like this builds and runs fine:</p>
<pre><code class="lang-auto">const auto tensor = torch::eye(3);
std::cout &lt;&lt; tensor &lt;&lt; '\n';
</code></pre>

sjmerel2: <p>I am having this same error on mobile.  Did you ever find a solution?</p>

Bilahari_Akkiraju: <p><a class="mention" href="/u/codesniffer13">@codesniffer13</a> did you ever find the solution for this ?</p>

codesniffer13: <p><a class="mention" href="/u/sjmerel2">@sjmerel2</a>  , <a class="mention" href="/u/bilahari_akkiraju">@Bilahari_Akkiraju</a>  - I confirmed the approach suggested by user VariantXYZ  at <a href="https://github.com/pytorch/pytorch/issues/14367#issuecomment-1258320460" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/issues/14367#issuecomment-1258320460</a> works.  I never completed the solution though.</p>
<p>It’d be great if someone can complete it and either merge it into mainline or make it available for others.</p>
Question about GPU memory garbage collect

XINGYU_ZHU: <h1><a name="problem-1" class="anchor" href="#problem-1"></a>Problem</h1>
<p><strong>Summary</strong>: The non-referenced tensor object still takes GPU memory.</p>
<p>I monitor the GPU memory usage through <code>nvidia-smi</code></p>
<p><strong>Code</strong>:</p>
<pre><code class="lang-auto">class FCBlock(nn.Module):
    def __init__(self, in_channel, hidden_channel, out_channel, n_blocks):
        super().__init__()
        self.net = nn.Sequential()
        self.net.append(nn.Linear(in_channel, hidden_channel))
        self.net.append(ReLU())
        for _ in range(n_blocks):
            self.net.append(nn.Linear(hidden_channel, hidden_channel))
            self.net.append(ReLU())
        self.net.append(nn.Linear(hidden_channel, out_channel))
   
    
    def forward(self, x):
        return self.net(x)

def T_forward(surf, y, direction):
    F, g = F_forward(surf, y)
    g = g / g.norm(dim=-1, keepdim=True).clamp(min=1e-6)
    return y + g * _D(y) * direction

def F_forward(surf, x):
    x.requires_grad=True
    F = surf(x)
    Fx = torch.autograd.grad(F, [x], grad_outputs=torch.ones_like(F),
                             retain_graph=True, create_graph=True)[0]
    return F, Fx

def test_leak(surf):
    y = torch.randn((8192, 100, 3)).to(V().cfg.device)
    direction = torch.ones((8192, )).view(8192, 1, 1).to(V().cfg.device)
    T_forward(surf, y, direction) # consumes 20GB GPU memory, and stay unreleased
    T_forward(surf, y, direction) # consumes another 20GB GPU memory, and stay unreleased

if __name__ == "__main__":
     surf = FCBlock(3, 512, 1, 2)
     test_leak(surf)
     print('hi')
     while True: pass
</code></pre>
<p><strong>Phenomenon</strong>:<br>
(1) The first <code>T_forward</code> consumes 20GB GPU memory, and stay unreleased<br>
(2) The second <code>T_forward</code> consumes another 20GB GPU memory, and stay unreleased<br>
(3) Both GPU memory are not released even after the end of <code>test_leak</code></p>
<p>I monitor the GPU memory usage through <code>nvidia-smi</code></p>
<p><strong>Analysis</strong>:<br>
I remember that gc in python auto collect the object with zero reference. But it seems like gc isn’t working in this code, since after the end of <code>test_leak</code>, no tensor object is referenced and all gpu tensor should be released. But <code>nvidia-smi</code> tells me they’re not released.</p>
<p>Why?</p>
<h1><a name="environment-2" class="anchor" href="#environment-2"></a>Environment</h1>
<pre><code class="lang-auto">torch 2.0.1+cu118
Ubuntu 22.04
</code></pre>

Diego: <p>In PyTorch when you move tensors to the GPU, the system has to allocate GPU memory for them. Since allocating/deallocating memory can be an expensive operation time-wise, when the tensors are no longer used PyTorch keeps this memory allocated as a form of cache in order to reuse it for future tensors without the overhead of allocating new memory. You can manually release this cache with <code>torch.cuda.empty_cache()</code>, at the cost of said overhead when allocating new tensors in the future.</p>
C++ for android

Jin_Pyo_Jun: <p>I am trying to implement a simple training code in C++.<br>
The target is the Android environment based on the ndk toolchain.<br>
Instead of an apk, I try to build it as a shard or executable with cmake.<br>
I’m struggling with the lack of manuals.<br>
I’m looking for a simple example with guide.<br>
help.</p>
Issue in changing mobilenetv3 width_mult parameter

Ashray_Saini: <p>I’m trying to change the width_mult=1.9 in Mobilenet_v3_large but getting the following error:</p>
<p>model = torchvision.models.mobilenet_v3_large(pretrained=True, width_mult=1.9,  reduced_tail=False, dilated=False)</p>
<p>RuntimeError: Error(s) in loading state_dict for MobileNetV3:<br>
size mismatch for features.0.0.weight: copying a param with shape torch.Size([16, 3, 3, 3]) from checkpoint, the shape in current model is torch.Size([32, 3, 3, 3]).<br>
size mismatch for features.0.1.weight: copying a param with shape torch.Size([16]) from checkpoint, the shape in current model is torch.Size([32]).</p>

ptrblck: <p>You won’t be able to load a pretrained <code>state_dict</code> into the model if you change <code>width_mult=1.9</code>. Remove it and it should work. In case you need this change you might need to train the model from scratch.</p>

Ashray_Saini: <p>Is their any other possible way I can get the weights without traning the model for width_mul=1.9</p>
Pipe Module throws cuda error

merkan: <p>I want to use (torch.distributed.pipeline.sync) module to split my model for 2 different GPUs. I was trying to implement documentation example in my case (<a href="https://pytorch.org/tutorials/intermediate/pipeline_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Training Transformer models using Pipeline Parallelism — PyTorch Tutorials 2.1.0+cu121 documentation</a>) but unfortunately PyTorch throws the following error.</p>
<p>“ValueError: nn.Module: Sequential(<span class="hashtag-raw">#skipped</span>#), should have all parameters on a single device, please use .to() to place the module on a single device.”</p>
<p>Following code is the example for my case.</p>
<pre><code>temp_list = [Transformer(1, 2, (192,448,480), 16, 12, 12, 0.1, [3, 6, 9, 12])]
module_list = []

for i in range(10):

    block = Transformer(1, 2, (192,448,480), 16, 12, 12, 0.1, [3, 6, 9, 12])
    
    device = 0 if i&lt;5 else 1

    temp_list.append(block.to(device))

module_list.append(nn.Sequential(*temp_list))

model = Pipe(torch.nn.Sequential(*module_list), chunks=2)
</code></pre>
<p>Since the main idea of Pipe is distributing model pipelines to the GPUs, how can it throw errors into use .to() ?</p>
Saving a Pytorch neural net (torch.autograd.grad included) as a Torch Script code

RabieF: <p>Hey everyone,</p>
<p>I have the following neural net setup:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/e/6ef7d6b14a4ccced1f2e91d4ead2b045780c1276.png" data-download-href="https://discuss.pytorch.org/uploads/default/6ef7d6b14a4ccced1f2e91d4ead2b045780c1276" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/e/6ef7d6b14a4ccced1f2e91d4ead2b045780c1276.png" alt="image" data-base62-sha1="fPFvQfu1B0muO9zOJWBB68Xoaqi" width="690" height="359" data-dominant-color="F1F2F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">918×478 15.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The neural net takes values of x as input, and is trained to output f(x) = x**2. The neural net is also trained to output the first-order derivative of f(x), i.e. f’(x) = 2*x. This derivative is computed using <strong>torch.autograd.grad</strong>, and is included as a second term in the loss function.</p>
<p>I have been unable to compile the neural net as a TorchScript code - for the example above, this is the error I have:</p>
<pre><code class="lang-auto">RuntimeError: Cannot insert a Tensor that requires grad as a constant. Consider making it a parameter or input, or detaching the gradient
Tensor:
 0.2816
-0.1232
-0.1976
 0.3143
[ torch.FloatTensor{4,1} ]
</code></pre>
<p>I have tried readjusting the neural net multiples times, but an error always comes up, and it always has to do with using torch.autograd.grad. How do I go about saving the above neural net as a Torchscript code? What am I missing here?</p>
<p>Thanks!</p>
Set dispatch key when m.impl("myadd", add_func)

jiangxiaobin96: <p>where set the dispatch key when i run m.impl(“myadd”, add_func).<br>
I just found code in function Library::<em>impl <code>aten/src/ATen/core/library.cpp</code>, but where init this dispatch_key</em>?</p>
<pre><code class="lang-c++">auto dispatch_key = f.dispatch_key_.has_value() ? f.dispatch_key_ : dispatch_key_;
</code></pre>

jiangxiaobin96: <pre><code class="lang-c++">class TorchLibraryInit final {
 private:
  using InitFn = void(Library&amp;);
  Library lib_;

 public:
  TorchLibraryInit(
      Library::Kind kind,
      InitFn* fn,
      const char* ns,
      c10::optional&lt;c10::DispatchKey&gt; k,
      const char* file,
      uint32_t line)
      : lib_(kind, ns, k, file, line) {    // init Library class
    fn(lib_);    // k in lib_ is dispatchKey
  }
};
</code></pre>
Freezing Part of The Computation Graph

VukGR: <p>Suppose that you have a PyTorch model (can be any model) and that your input tensor X = [x_1, x_2, x_3, …, x_n] is made of n tensors {x_i}. Now, if I wanna perform some kind of adversarial attack I wanna modify my input X such that I can only change one of {x_i} at a time. So my question is the following: say I want to change x_1 and keep other ones fixed, obviously now computing model(x_1) has a parts of computation graph dependent only on x_2, …, x_n and not x_1. Is it possible to somehow freeze this computation graph and create a new model, let’s call it model_1(x) = model(x, x_2, …, x_n) (changed x_1 to x to indicate that x is now a free parameters whereas other values are fixed), such that this model_1 only once computes parts of computational graph independent of x? This would make iterating and optimising over all x_i much quicker.</p>
Inconsistency in Data Type Requirements for torch.max and torch.mul Functions

linlang1837: <p>I’ve come across an interesting observation regarding the data type requirements for the <code>torch.max</code> and <code>torch.mul</code> functions. According to the documentation, when using <code>torch.max(a, out=(b, c))</code>, it is explicitly stated that the data types of the output tensors <code>b</code> and <code>a</code> must be the same.</p>
<p>However, I noticed that when using <code>torch.mul(a, 3, out=b)</code>, there is no strict requirement for the data types of <code>b</code> and <code>a</code> to be identical. I’m curious about the rationale behind this difference in behavior.</p>
<p>Could someone point me to the specific files or modules where this logic is implemented?  I guess the relevant codes are in TensorIterator.cpp. Any guidance on navigating the PyTorch source code to find this information would be greatly appreciated.</p>
Get error "tuple index with non-constant index" when trying to export a model to onnx format

feiyuhuahuo: <p>I’m trying to convert a model to onnx format. A function was wrapped by <code>torch.jit.script()</code> in the model forward method. No other <code>torch.jit.script()</code>  was uesd. The forward method is following:</p>
<pre><code class="lang-auto">    def forward(self, img):
        img = self.onnx_trans(img)
        x = self.backbone(img)
        x = self.neck(x)
        mask_feat_pred = self.mask_feat_head(x[self.mask_feat_head.start_level:self.mask_feat_head.end_level + 1])
        cate_preds, kernel_preds = self.bbox_head(x)
        get_seg = torch.jit.script(get_seg_scripted)
        seg_result = get_seg(cate_preds, kernel_preds, mask_feat_pred)
        print(seg_result)
        return seg_result
</code></pre>
<p>And  I use the following code to export the model:</p>
<pre><code class="lang-auto">    cfg = Custom_light_res50(mode='detect')
    cfg.print_cfg()
    model = SOLOv2(cfg).cuda()
    model.load_state_dict(torch.load(cfg.val_weight), strict=True)
    model.eval()

    input_size = (512, 512)
    input_img = cv2.imread('detect_imgs/test1.bmp', cv2.IMREAD_COLOR)
    input_img = cv2.resize(input_img, input_size)
    input_img = torch.from_numpy(input_img).cuda()

    torch.onnx.export(model,
                      input_img,
                      'seg.onnx',
                      input_names=['seg'],
                      output_names=['output'],
                      verbose=False,
                      opset_version=14)
</code></pre>
<p>I can get the correct value of variable <code>seg_result</code>. But still I get the error:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "/home/feiyu/SOLOv2_minimal/ttt.py", line 99, in &lt;module&gt;
    torch.onnx.export(model,
  File "/home/feiyu/.local/lib/python3.10/site-packages/torch/onnx/utils.py", line 504, in export
    _export(
  File "/home/feiyu/.local/lib/python3.10/site-packages/torch/onnx/utils.py", line 1529, in _export
    graph, params_dict, torch_out = _model_to_graph(
  File "/home/feiyu/.local/lib/python3.10/site-packages/torch/onnx/utils.py", line 1115, in _model_to_graph
    graph = _optimize_graph(
  File "/home/feiyu/.local/lib/python3.10/site-packages/torch/onnx/utils.py", line 582, in _optimize_graph
    _C._jit_pass_lower_all_tuples(graph)
RuntimeError: tuple index with non-constant index
</code></pre>
<p>Did I do anything wrong? Since I can already get the correct value, which means the computation is correct. Why there’s still an error?</p>

nivtorch: <p>Hi, did you get any solution to this error?<br>
I am getting the same issue.</p>
Installing Pytorch without CPU or GPU

Jasmine_Yahia: <h2><a name="i-installed-python-312-from-its-website-on-windows-10-then-by-using-command-prompt-for-python-i-checked-that-i-got-the-versions-of-python-312-and-pip-233python-i-ran-the-command-pip-install-torch-i-got-this-defaulting-to-user-installation-because-normal-site-packages-is-not-writeable-error-could-not-find-a-version-that-satisfies-the-requirement-torch-from-versions-none-error-no-matching-distribution-found-for-torch-how-can-i-solve-the-problem-1" class="anchor" href="#i-installed-python-312-from-its-website-on-windows-10-then-by-using-command-prompt-for-python-i-checked-that-i-got-the-versions-of-python-312-and-pip-233python-i-ran-the-command-pip-install-torch-i-got-this-defaulting-to-user-installation-because-normal-site-packages-is-not-writeable-error-could-not-find-a-version-that-satisfies-the-requirement-torch-from-versions-none-error-no-matching-distribution-found-for-torch-how-can-i-solve-the-problem-1"></a>I installed Python 3.12 from its website, on Windows 10. Then, by using Command Prompt for Python. I checked that I got the versions of Python (3.12) and pip (23.3)Python I ran the command: pip install torch. I got this: {Defaulting to user installation because normal site packages is not writeable ERROR: Could not find a version that satisfies the requirement torch (from versions: none) ERROR: No matching distribution found for torch.}. How can I solve the problem</h2>
<p>Note: There is not a CPU-only choice on the site. And the same problem I faced while using Conda to install it.</p>

ptrblck: <p>Python 3.12 is not supported yet in the binary builds so you would need to downgrade to e.g. 3.11.</p>

Pyrex: <p>Do you have any idea of when it will be supported ?</p>
Quantization of ssdlite from torchvision

TsukeBurrel: <p>Hello, I have applied post training static quantized the ssdlite model from torchvision. The model has been successfully quantized, however when I try to infer the model, the following error comes up.</p>
<p>“in Normalize raise TypeError( TypeError: Expected input images to be of floating type (in range [0, 1]), but found type torch.quint8 instead”</p>
<p>I can understand that the transform in the torchvision library is the reason of this issue. However, I couldn’t fix this issue just by commenting out “transform part” from my local library file.<br>
The issue on this part can be solved by not applying QuantStub but model part gives another error.</p>
<p>I’d like to ask the way to successfully quantize ssdlite model from torchvision.</p>

jcaip: <p>Hi <a class="mention" href="/u/tsukeburrel">@TsukeBurrel</a> do you have some example code to reproduce this error?</p>
Cuda_runtime_api.h not in $CUDA_HOME/include

XINGYU_ZHU: <h1><a name="environment-1" class="anchor" href="#environment-1"></a>Environment</h1>
<pre><code class="lang-auto">cuda-12.2
torch 2.1.0
</code></pre>
<h1><a name="problem-2" class="anchor" href="#problem-2"></a>Problem</h1>
<p>when using <code>pip install .</code> to install local <code>setup.py</code></p>
<p>I got an error <code>/home/xingyu/anaconda3/envs/wm/lib/python3.8/site-packages/torch/include/ATen/cuda/CUDAContext.h:5:10: fatal error: cuda_runtime_api.h: No such file or  directory</code></p>
<p>It appears that <code>cuda_runtime_api.h</code> is in</p>
<pre><code class="lang-auto">$CUDA_HOME/targets/x86_64-linux/include/thrust/system/cuda/detail/guarded_cuda_runtime_api.h
$CUDA_HOME/targets/x86_64-linux/include/cuda_runtime_api.h
</code></pre>
<p>instead of</p>
<pre><code class="lang-auto">$CUDA_HOME/include
</code></pre>
<p>Have anyone met the same problem?</p>
Pytorch version 1.13.1 vs pytorch version 1.13.1+cu117

bryan123: <p>is there a difference between pytorch version 1.13.1 and 1.13.1+cu117 (assuming my cuda version is 11.7)?</p>

ptrblck: <p>Your locally installed CUDA version won’t be used since the PyTorch binaries ship with their own CUDA dependencies. <code>1.13.1+cu117</code> indicates a wheel with CUDA 11.7 dependencies while <code>1.13.1</code> indicates either a CPU-only build (current versions should show <code>+cpu</code> but unsure if that was always the case) or a <code>conda</code> binary.</p>

bryan123: <p>Could you show me how to install PyTorch with a locally installed CUDA version? I tried it and failed because <code>torch.cuda.is_available()</code> returned <code>False</code> . Instead, installing it with <code>pip install torch==1.13.1+cu117 --extra-index-url https://download.pytorch.org/whl/cu117</code> is working (torch.cuda.is_available() returned ‘True’).</p>

ptrblck: <p>You could need to build from source as described <a href="https://github.com/pytorch/pytorch#from-source">here</a>.</p>
Why my nn.Identity function is changing the input shape?

hasan11085: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/d/2ded484ef00760b5b8957197ba2089413cf57ea7.png" data-download-href="https://discuss.pytorch.org/uploads/default/2ded484ef00760b5b8957197ba2089413cf57ea7" title="Screenshot 2023-08-17 at 11.14.07 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/d/2ded484ef00760b5b8957197ba2089413cf57ea7_2_649x500.png" alt="Screenshot 2023-08-17 at 11.14.07 PM" data-base62-sha1="6yhRpxQil9kJTYaCvfa6ZvpX3PF" width="649" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/d/2ded484ef00760b5b8957197ba2089413cf57ea7_2_649x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/2/d/2ded484ef00760b5b8957197ba2089413cf57ea7_2_973x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/2/d/2ded484ef00760b5b8957197ba2089413cf57ea7_2_1298x1000.png 2x" data-dominant-color="F2F2F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2023-08-17 at 11.14.07 PM</span><span class="informations">1691×1302 343 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Code:<br>
import torch<br>
import torchvision.models<br>
from torchinfo import summary<br>
import torch.nn as nn<br>
class Identity(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(Identity, self).<strong>init</strong>()</p>
<pre><code>def forward(self, x):
    print(x.shape)
    return x
</code></pre>
<p>model1=torchvision.models.vit_h_14(weights=‘IMAGENET1K_SWAG_E2E_V1’)<br>
for index in range(12,32):<br>
model1.encoder.layers[index]=nn.Identity()<br>
model1.encoder.ln=Identity()<br>
model1.head=Identity()<br>
model1.heads=Identity()<br>
model1.fc=Identity()<br>
print(model1)</p>

ptrblck: <p>If the original modules, which you are replacing with <code>nn.Identity</code> modules, changed the activation shape, the issue is expected.<br>
Did you check which layers you are replacing? And if so, did you check if these layers change the activation shape?</p>

paula_ramirez_rangel: <p>Hi <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"> Thank you. I have a similar problem, how do I know if the layers I’m replacing change the activation shape? And how do I correct it? I’m replacing classifier with nn.Identity in a VGG model</p>

ptrblck: <p>You could either manually check the input and output activation shape inside the <code>forward</code> method of your model:</p>
<pre><code class="lang-python">def forward(self, x):
    ...
    print(x.shape)
    x = self.classifier(x)
    print(x.shape)
    return x
</code></pre>
<p>and compare these or you could alternatively also use forward hooks and print the input/output shapes there.</p>
Multiple backprops through encoder part of the network

hermancollin: <p>Let me explain: I want to fine-tune the <a href="https://github.com/facebookresearch/segment-anything" rel="noopener nofollow ugc">segment-anything-model</a>, which consists of a heavyweight image encoder network, a small encoder for the prompts (points or bboxes) and a lightweight mask decoder.</p>
<p><img src="https://segment-anything.com/assets/section-3.1c.png" alt="https://segment-anything.com/assets/section-3.1c.png" width="" height=""></p>
<p>A typical training image will have ~50 bboxes. So for a single image, we do 1 forward pass through the image encoder. Assuming we prompt the model with a batch of 10 bounding boxes, we will thus do 50/10=5 forward passes through the other downstream smaller models (mask decoder). This is a problem because after the first backward pass, the gradients of the big image encoder are flushed.</p>
<p>What would be the canonical way to deal with this? Should I put the intermediate output of the image encoder in some sort of buffer?</p>
Pruning/Compressing heads in attention blocks

Namburi_Srinath: <p>I’ve a conceptual question</p>
<p>BERT-base has a dimension of 768 for query, key and value and 12 heads (Hidden dimension=768, number of heads=12). The same is conveyed if we see the BERT-base architecture</p>
<pre><code class="lang-auto">(self): BertSelfAttention(
              (query): Linear(in_features=768, out_features=768, bias=True)
              (key): Linear(in_features=768, out_features=768, bias=True)
              (value): Linear(in_features=768, out_features=768, bias=True)
              (dropout): Dropout(p=0.1, inplace=False)
)
</code></pre>
<p>Now, my question is:</p>
<p>Can I consider the first 64 neurons from the <em>out_features</em> as the first-head, the next 64 neurons from the <em>out_features</em> as the 2nd head and so on? (sec 3.2.2 from original paper; <a href="https://papers.nips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf" rel="noopener nofollow ugc">Link</a>)</p>
<p>Basically, I am wondering if the Linear module representing query matrix; which is 768x768 can be thought as (768x64), (768x64)…12 times? The same for key and value modules</p>
<p>If so, is it possible to provide some starter code as I am unable to wrap around my head. Any help is appreciated (and I’ve some sample in the contribution section)</p>
<p><strong>P.S:</strong> Here’s the issue I raised in Github (<a href="https://github.com/pytorch/pytorch/issues/111932" rel="noopener nofollow ugc">link</a>)</p>

vdw: <p>I don’t think(!) you can make this consideration.</p>
<p>Yes, internally, all attention heads are represented by a single <code>nn.Linear</code> layer. However, their combined output is then still pushed through another <code>nn.Linear</code> layer – W^O in the cited paper, top of Page 5. This is the output you “see”, and here you no longer can say that the first 64 neurons refer to the first head.</p>
<p>I have a transformer implementation from scratch available <a href="https://github.com/chrisvdweth/nus-cs4246x/blob/master/3-neural-nlp/src/transformer.py" rel="noopener nofollow ugc">here</a>; see class “MultiHeadAttention”. Please note that this implementation treats each head as its own <code>nn.Linear</code> layer. I also have an optimized version using asingle <code>nn.Layer</code> for all the heads, but not yet on Github. In any case, both implementations do a</p>
<pre><code class="lang-auto">out = self.Wo(out)
</code></pre>
<p>at the very end, as this last layer is the same.</p>

Namburi_Srinath: <p>Thanks for the response Chris. I understand as we are combining at the very end, it’s difficult to achieve this feature.</p>
<p>There’s an active discussion going in Huggingface Github on this and it turns out HF has few functions which support this functionality. I hope this helps <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>Link: <a href="https://github.com/huggingface/transformers/issues/27044" rel="noopener nofollow ugc">https://github.com/huggingface/transformers/issues/27044</a></p>
Does this management of the tensor dimension make sense?

javiersgjavi: <p>I have data with the following form [B, T, N, F], where each word stands for batch, time, node and features respectively. I pass this tensor through many modules and make some permutations to the dimensions, so I will try to explain them as best I can.</p>
<p>1- I pass the tensor through a forwarding module that extracts spatio-temporal representations in forwarding time and outputs [B, T, N, 1]. I squeeze and apply a permutation to this tensor to obtain [B, N, T].</p>
<p>2- I pass the tensor through a similar module but this one specialised in extracting backward spatio-temporal representations. Again, the output is [B, N, T].</p>
<p>3- I mix the information extracted by the two modules by making a torch.cat, obtaining a tensor with the form [B, N, 2*T].</p>
<p>4- Finally, I apply a MLP to the last tensor to obtain a prediction of shape [B, N, T] that combines the information obtained from the representations extracted by the two modules of points 1 and 2.</p>
<p>I did this process for a project I started and left incomplete a few months ago; however, I am reviewing it and I have the feeling that these permutations do not make sense. What am I missing?</p>

J_Johnson: <p>A couple questions.</p>
<ol>
<li>When you say a forwarding module, do you mean a Linear Layer, or a 1d/2d convolution layer?</li>
</ol>
<p>On a side note, a Linear layer won’t extract spatial-temporal representations effectively, if at all. Work through a few matmul examples in a spreadsheet, and it should become clear why that operation is essentially order blind.</p>
<p>Convolutions, on the other hand, are far more suitable for that job. RNNs are also sufficient, but suffer data loss on longer sequences.</p>
<ol start="2">
<li>
<p>Are you applying any attention layers?</p>
</li>
<li>
<p>Are you using positional/temporal embeddings?</p>
</li>
</ol>
<p>As far as permuting the time and node dims, it’s hard to say what that is/isn’t accomplishing without knowing the layer types involved.</p>

javiersgjavi: <p>Thank you very much for your answer. I will answer you point by point to keep the order of the topics.</p>
<ol>
<li>
<p>The module that receives as input the shape tensor [B, T, N, F] is a recurrent GNN. Essentially, it is composed of two submodules: The first which is an RNN, and the second which implements a GCN.</p>
</li>
<li>
<p>I do not use any attention layers. Only RNN + GCN layers in the first two modules and finally linear layers to blend the previously extracted representation.</p>
</li>
<li>
<p>I am not sure if I understood what you mean by positional/temporal embeddings, but I do not apply any further transformations to tensors able to operate on temporal and abstract data apart from the first two modules.</p>
</li>
</ol>
<p>I hope this post can shed some light on my problem setup. Thank you very much again!</p>
How to get the size of the memory needed to train a model?

Oussama_Bouldjedri: <p>Hi, I am working on am image classification problem using pythorch and I was wondering how do I get the required memory size to train the model ? some might suggest summary I tried that but I am doubting since I have a ViT (a basic one not a large or a tiny has less MB compared to a resnet 56 )?</p>
<p>Regrads</p>

J_Johnson: <p>As long you can fit a batch size of 1 on the gpu, you can use gradient accumulation.</p>
<p>See <a href="https://discuss.pytorch.org/t/why-do-we-need-to-set-the-gradients-manually-to-zero-in-pytorch/4903/20">here</a>.</p>
<p>As for calculating the GPU size, it’s a bit complicated with models using convolutions. This is because it also depends on your image size, the number and size of layers, the dtype, kernel size, optimizer, model.train() vs. model.eval(), and the batch size.</p>
<p>The chart on <a href="https://pytorch.org/vision/stable/models.html" rel="noopener nofollow ugc">this page</a> gives the parameter sizes between various pretrained vision models for Pytorch. That can generally be used as a relative comparison, but you’ll need to use some trial and error, depending on how you set the batch size, image size, etc.</p>

Oussama_Bouldjedri: <p>Hi thanks for the reply, probably I miss spoke my self, I want to measure how much my model require so, I can tell if this model can be trained on smart phone or raspberry pi or a Computer depending on it needed memory and which device hardware is able to handle it ?Thanks in advance</p>
`num_threads` in `torch.utils.benchmark` only for CPU?

OverLordGoldDragon: <p>The benchmark tutorial (below), and the function’s docs state that the default is <code>num_threads=1</code>, which doesn’t make much sense on a GPU. Is it only referring to the CPU, and does <code>benchmark</code> use all GPU threads? <code>torch.get_num_threads()</code> also returns the CPU count.</p>
<p><a href="https://pytorch.org/tutorials/recipes/recipes/benchmark.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/recipes/recipes/benchmark.html</a></p>

ptrblck: <p>From the <a href="https://pytorch.org/docs/stable/generated/torch.get_num_threads.html#torch-get-num-threads">docs</a>:</p>
<blockquote>
<p><code>torch.get_num_threads()</code> → <a href="https://docs.python.org/3/library/functions.html#int">int</a><br>
Returns the number of threads used for parallelizing CPU operations</p>
</blockquote>
<p>So yes, the threads setting refers to CPU-only workloads.</p>
<aside class="quote no-group" data-username="OverLordGoldDragon" data-post="1" data-topic="190647">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/overlordgolddragon/48/21887_2.png" class="avatar"> OverLordGoldDragon:</div>
<blockquote>
<p>and does <code>benchmark</code> use all GPU threads?</p>
</blockquote>
</aside>
<p>The <code>benchmark</code> util. won’t specify or change the device and your code is responsible to use the GPU or CPU.</p>
PyTorch-1.13.1 installation

hdzlety: <p>What file types (if any) does PyTorch create in a source code Linux installation?</p>

ptrblck: <p>The created files in the <code>build</code> folder should have these extensions based on a recent source build from <code>main</code>:</p>
<pre><code class="lang-auto">a
bin
bzl
c
cc
check_cache
cmake
cpp
cu
cubin
d
fatbin
gpu
h
ii
in
includecache
internal
json
log
make
marks
module_id
o
org
out
pc
ptx
rsp
so
tcl
ts
txt
</code></pre>
<p>The installation will of course copy python files etc. into your Python environment.</p>
Raft(small and large) - Memory usage in eval mode with pretrained weights

TimSchimansky: <p>(Beforehand: i use pytorch currently CPU only)</p>
<p>Hi there. I am not completely new to PyTorch but I am new to dense optical flow with PyTorch. The problem, that I am stumbling across is the following.</p>
<p>I have a image sequence and want to determine the optical flow for all consequtive frames. Therefore I have oriented myself with the code at the very bottom of:</p>
<p><a href="https://pytorch.org/vision/0.12/auto_examples/plot_optical_flow.html#sphx-glr-auto-examples-plot-optical-flow-py" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/vision/0.12/auto_examples/plot_optical_flow.html#sphx-glr-auto-examples-plot-optical-flow-py</a></p>
<p>Originally intended for generating GIFs. But should work for my intentions as well. I am using that structure on my own images and running into memory issues now.</p>
<p>My previous understanding was that the training of a network is the heavy lifting and the eval mode ‘just’ classifies based on the acquired weights. In my case, after each Iteration of the image pair loop, my 12GBs of RAM are filled further, ending with a SIGKILL after just 3-4 image pairs. (the same happens if i do batches of 2, just faster; or if i use raft_small, just slower).</p>
<p>What i dont understand:<br>
I use pretrained weights and the eval mode. Shouldn’t the RAM clear after each ‘classification’/call of the model() function?</p>
<p>The results and everything else looks fine. Just the RAM gets fuller and fuller.</p>

EdwardjkFeng: <p>Hello Tim,</p>
<p>I had the same problem and after doing some research, I finally figured out why the memory usage remains high after switching to <code>eval</code> mode.</p>
<p>The first thing we have to understand is that the majority of the memory is occupied to save the gradients of the tensors. However, switching the model from <code>train()</code> to <code>eval()</code> changes merely the behavior of some certain layers like dropout and batchNorm, but it does not disable the gradient, i.e. it will still build a computation graph for the tensors, compute and save gradients.</p>
<p>To mitigate the high RAM usage, you can use <code>torch.no_grad()</code>, which disables the gradient computation, and thus, the memory consumption should be significantly reduced during the forward pass.</p>
<p>Hope it helps. If anything is wrong, please correct me. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
Autoencoder to generate non image data

juliolopes: <p>Hello! I’m trying to build an autoencoder to generate a syntethic dataset, based on my real one. My dataset is composed of two features that are connected (they are both related). Although to ensure data consistency in the generation of consecutive samples, I think it’s necessary to do something like a sliding window approach, for the AE to learn this. The shape of my numpy array (windowed) is (892,10), in each 892 is the number of samples (windows) and 10 is the number of features of each window (5 of each of the features).</p>
<p>With this dataprocessing, now I’m ready to train the AE. I tried something like this:</p>
<pre><code class="lang-auto">train_dataset = TensorDataset(torch.from_numpy(numpy_array).double())
train_loader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True)

class Autoencoder(nn.Module):
    def __init__(self, feats, window_length):
        super(Autoencoder, self).__init__()
        self.encoder = nn.LSTM(feats, 4, batch_first=False)  
        self.decoder = nn.LSTM(4, feats, batch_first=False)

    def forward(self, x):
        encoded, _ = self.encoder(x)
        decoded, _ = self.decoder(encoded)
        return decoded

model = Autoencoder(numpy_array.shape[1], window_length)
model = model.double()
criterion = nn.MSELoss()
optimizer = optim.Adam(model.parameters())

# Training loop
for epoch in range(epochs):
    model.train()
    train_loss = 0.0
    for inputs in train_loader:
        optimizer.zero_grad()
        inputs = [item.double() for item in inputs]
        outputs = model(inputs[0])
        loss = criterion(outputs, inputs[0])  
        loss.backward()
        optimizer.step()
        train_loss += loss.item()
    print(outputs)
    print(f'Epoch {epoch + 1}/{epochs}, Train Loss: {train_loss / len(train_loader)}')

torch.save(model.state_dict(), 'autoencoder_model.pth')

</code></pre>
<p>Output of outputs:<br>
tensor([[nan, nan, nan, nan, nan, nan, nan, nan, nan, nan],<br>
[nan, nan, nan, nan, nan, nan, nan, nan, nan, nan],<br>
[nan, nan, nan, nan, nan, nan, nan, nan, nan, nan],<br>
[nan, nan, nan, nan, nan, nan, nan, nan, nan, nan]],<br>
dtype=torch.float64, grad_fn=)</p>
<p>Do you have any idea to fix this? Or a new idea to generate new data, in pytorch, preferably with an Autoencoder?</p>
Detect inf/nan in forward pass

AlbertZeyer: <p><code>autograd.detect_anomaly</code> detects inf/nan in the backward pass.</p>
<p>I want to have the same in the forward pass. With the possibility to whitelist a few special operations, modules or code blocks, e.g. masking attention energies to -inf, etc.</p>
<p>How?</p>
<p>Is there an easy way to install a hook after every operation?</p>
<p>Or otherwise, I was thinking about installing a post forward hook for every module, which would already be helpful.</p>
<p>Or I was also thinking about using <code>sys.settrace</code> to install some function which would inspect the locals.</p>
<p>Or what else would you suggest?</p>

ptrblck: <p>Naively you could use <code>torch.isnan/.isinf/.isfinite</code> inside the <code>forward</code> and add your custom logic to it. Alternatively, forward hooks should also work as these would allow you to return a modified output. However, the issue would be the operation causing the invalid outputs could still return an invalid gradient.</p>

AlbertZeyer: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="190514">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>Naively you could use <code>torch.isnan/.isinf/.isfinite</code> inside the <code>forward</code> and add your custom logic to it.</p>
</blockquote>
</aside>
<p>Yes sure but I don’t want to modify the code of the forward pass every time I want to do such checks.</p>
<p>Similarly, I might want to collect other statistics in the future, like e.g. finding very big activations, or so.</p>
MaskRCNN pertained versions other than maskrcnn_resnet50_fpn_coco

FiReTiTi: <p>Hi,<br>
I would like to change MaskRCNN backbone using resnet (other than resnet50).<br>
When looking at the mask_rcnn.py code (here <a href="https://github.com/pytorch/vision/blob/master/torchvision/models/detection/mask_rcnn.py" rel="nofollow noopener">https://github.com/pytorch/vision/blob/master/torchvision/models/detection/mask_rcnn.py</a>) I see that there is a fully pretrained version for resnet50 with FPN on coco ‘maskrcnn_resnet50_fpn_coco’.<br>
Does it exist other fully pretrained versions for MaskRCNN?<br>
Like for example using resnet101 or even resNext?<br>
Thanks in advance.</p>

cominder: <p>have you found other fully pretrained versions for MaskRCNN?</p>
Where's the at::empty() source code?

outis: <p>I just find</p>
<pre><code class="lang-auto">Tensor empty(
    IntArrayRef size,
    c10::optional&lt;DimnameList&gt; names,
    c10::optional&lt;ScalarType&gt; dtype,
    c10::optional&lt;Layout&gt; layout,
    c10::optional&lt;Device&gt; device,
    c10::optional&lt;bool&gt; pin_memory,
    optional&lt;MemoryFormat&gt; optional_memory_format) {
  // See [Note: hacky wrapper removal for TensorOptions]
  TensorOptions options = TensorOptions().dtype(dtype).layout(layout).device(device).pinned_memory(pin_memory);

  if (!names.has_value()) {
    return at::empty(size, options, optional_memory_format);
  }
  TORCH_CHECK(options.layout() == Layout::Strided,
      "NYI: named tensors only support strided layout");
  TORCH_CHECK(options.device().is_cpu() || options.device().is_cuda(),
      "NYI: named tensors only support CPU and CUDA tensors");
  auto result = at::empty(size, options, optional_memory_format);
  internal_set_names_inplace(result, names);
  return result;
}
</code></pre>
<p>but then there is nothing about at::empty(size, options, optional_memory_format);</p>

jiangxiaobin96: <p>build/aten/src/ATen/ops/empty.h, you can find the corresponding function from <strong>polymorphism</strong>.</p>
How to load many numpy arrays to DataLoader from disk?

Asya: <p>Is there more efficient way to load all <code>.npy</code> arrays to <code>DataLoader</code> ?<br>
this doesn’t work when there are too many arrays:</p>
<pre><code class="lang-auto">files = glob.glob('./*)

all_inputs = torch.tensor([])
all_labels = torch.tensor([])

for i in range(len(files)):
    input = torch.tensor(np.load('imgs_' + str(i) + '.npy'))
    all_inputs = torch.cat([all_inputs, input], dim=0)
    print(all_inputs.shape)

    labels = torch.tensor(np.load('labels_' + str(i) + '.npy'))
    all_labels = torch.cat([all_labels, labels], dim=0)
    print(all_labels.shape)


dataset = TensorDataset(all_inputs, all_labels)
dataloader = DataLoader(dataset)
</code></pre>

ptrblck: <p>You could write a custom <code>Dataset</code> and lazily load each numpy array in the <code>__getitem__</code>. <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html">This tutorial</a> might be a good starter.</p>

metatl: <p>Is there good ways to do transforms on numpy arrays that have say 10 channels using this method?</p>

ptrblck: <p>You could convert the <code>numpy.array</code> to a <code>tensor</code> and apply <code>transforms</code> on it, as <code>transforms</code> also support tensors:</p>
<pre><code class="lang-python">transform = transforms.Resize((20, 20))
x = np.random.randn(10, 200, 200)
out = transform(torch.from_numpy(x))
print(out.shape)
# torch.Size([10, 20, 20])
</code></pre>