akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How is DataLoader's dataset argument accessed by worker processes?

username: bhoeckendorf
cooked: <p>I’d like to ask how the “dataset” argument of <code>torch.utils.data.DataLoader</code> is accessed by worker processes during multi-process data loading.</p>
<p>My dataset object contains a database handle that can’t be shared across multiple processes. Single-process data loading works as intended. During multi-process data loading, all processes erroneously attempt to use the same database connection handle.</p>
<p>I’ve implemented custom copying and pickling that create dedicated database connections for each object. This works as intended when I deepcopy or pickle the object myself. However, it doesn’t seem to solve this problem.</p>
<p>How do I open dedicated database connections as my dataset object is sent to, or accessed by worker processes?<br>
Thanks for any pointers.</p>
<h3>
<a name="update-1" class="anchor" href="#update-1"></a>Update</h3>
<p>I’ve found a hacky solution by using <code>torch.utils.data.get_worker_info</code> to detect whether the current dataset object is inside a worker process, which triggers a one-time close+reopen of the database connection. This results in each loading process using a dedicated database connection as intended.</p>
<p>I’d much rather do this when the dataset object is handed over to the worker process, but still haven’t had any luck implementing that.</p>
Android C++ demo with android-ndk r21d and Pytorch 1.11

username: dsjochel
cooked: <p>Has anyone tried (with latest Pytorch 1.11) to run the build_android.sh script as found in this repo?</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://github.com/ljk53/pytorch-android-cpp-demo">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/ljk53/pytorch-android-cpp-demo" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/345;"><img src="https://opengraph.githubassets.com/2e97af2cb11e8cc2066ca651bdb82bebeaa6f87b1637992b62d57ab4d42d5a4f/ljk53/pytorch-android-cpp-demo" class="thumbnail" width="690" height="345"></div>

<h3><a href="https://github.com/ljk53/pytorch-android-cpp-demo" target="_blank" rel="noopener nofollow ugc">GitHub - ljk53/pytorch-android-cpp-demo</a></h3>

  <p>Contribute to ljk53/pytorch-android-cpp-demo development by creating an account on GitHub.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>I am running into issues getting this to build and wondering if others have run into problems.<br>
More specifically, I seem to be getting a linker error in the demo application after the Pytorch libraries get built:<br>
/opt/android-ndk-r21d/toolchains/llvm/prebuilt/linux-x86_64/lib/gcc/arm-linux-androideabi/4.9.x/…/…/…/…/arm-linux-androideabi/bin/ld: fatal error: -f/–auxiliary may not be used without -shared<br>
clang++: error: linker command failed with exit code 1 (use -v to see invocation)</p>
Python function to resize images and superimpose on one another to find the difference

username: alpha2210
cooked: <p>I am beginner in python. I have multiple images in single directory. I need to compare each image with other all images if they are of equal size if not then scale the other to make it equal size. After that in another python function I need to have 2 images which will convert be converted into red channel and blue channel each. Then superimpose red channel image to blue channel image to find difference between them. Finally store that superimposed image in another folder.</p>
<p>How should I do that? Thanks in advance.</p>

username: ptrblck
cooked: <p>The question doesn’t seem to be PyTorch-related so I would recommend to post it in another board (e.g. StackOverflow) to get a faster and better answer.</p>
Leftover in memory

username: yangzh
cooked: <p>I get confused when I try to profile the memory consumption of my simple program.</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
from memory_profiler import profile

def simple_func():
    x = torch.rand(3000, 2000, requires_grad=True)
    f = nn.Linear(2000, 1000)
    y = f(x)
    dd = torch.autograd.grad(y, (x,)+tuple(f.parameters()), grad_outputs=torch.rand(3000, 1000))

@profile
def compound_func():
    simple_func()
    simple_func()
    simple_func()
    simple_func()

compound_func()
</code></pre>
<p>The program output</p>
<pre><code class="lang-auto">Line #    Mem usage    Increment  Occurrences   Line Contents
=============================================================
    48    306.8 MiB    306.8 MiB           1   @profile
    49                                         def compound_func():
    50    319.4 MiB     12.6 MiB           1       simple_func()
    51    403.2 MiB     83.8 MiB           1       simple_func()
    52    403.2 MiB      0.0 MiB           1       simple_func()
    53    403.2 MiB      0.0 MiB           1       simple_func()
</code></pre>
<p>I don’t understand why the first two simple_func calls give increment in memory usage. It seems there is often leftover in memory even if all objects are local and should be deleted after the call.</p>
<p>I have tried an even simpler function</p>
<pre><code class="lang-auto">def simple_func():
    x = torch.rand(3000, 2000)
</code></pre>
<p>But there is still leftover</p>
<pre><code class="lang-auto">Line #    Mem usage    Increment  Occurrences   Line Contents
=============================================================
    48    306.8 MiB    306.8 MiB           1   @profile
    49                                         def compound_func():
    50    308.5 MiB      1.7 MiB           1       simple_func()
    51    331.2 MiB     22.7 MiB           1       simple_func()
    52    331.2 MiB      0.0 MiB           1       simple_func()
    53    331.2 MiB      0.0 MiB           1       simple_func()
</code></pre>
<p>Do we need to clean the memory manually? I have tried “del” before the end of the simple_func, but it doesn’t help.<br>
Or is it just the measurement issue by memory_profiler?</p>
Backprop through iterative process with in-place CUDA ops

username: A_A
cooked: <p>Hi, this is a bit of an unusual scenario:</p>
<p>I have a CUDA kernel implementation of an autograd Function <code>Func</code> with custom backward pass which I use as follows during forward pass:</p>
<blockquote>
<p>x_1 = from data loader<br>
for t = 1 : N<br>
x_(t + 1) = Func(x_t)</p>
</blockquote>
<p>Where <code>N</code> can very large e.g. 100 and so to decrease GPU memory usage, <code>Func</code> performs operations on <code>x</code> in-place and sets the <a href="https://pytorch.org/docs/stable/generated/torch.autograd.function.FunctionCtx.mark_dirty.html#torch.autograd.function.FunctionCtx.mark_dirty" rel="noopener nofollow ugc">mark_dirty</a> flag on it. During backward pass I would like to recompute the computation graph by basically trading compute for memory. Would I need to use <a href="https://pytorch.org/docs/stable/checkpoint.html" rel="noopener nofollow ugc">checkpointing</a> to achieve this goal or does setting <code>mark_dirty</code> flag on a Tensor automatically will recompute the input? Or said another way how does <code>mark_dirty</code> recomputes the lost Tensor, does it keep a copy in memory or computes it?</p>
<p>Thanks!</p>

username: albanD
cooked: <p>Hi,</p>
<p><code>mark_dirty</code> does not recompute the Tensor for you in any ways. It only allows autograd to know what was modified inplace for error checking purposes.</p>

username: A_A
cooked: <p>Thanks. How do you suggest I can let autograd know about the in-place operation happening in <code>Func</code>?</p>

username: albanD
cooked: <p>This is the right way to let the autograd know about the inplace.<br>
That’s actually the only think mark_dirty is doing: let the autograd know that an inplace op happened.</p>

username: A_A
cooked: <p>Sorry I was not clear earlier. How would autograd reconstruct original <code>x</code> for backprop, if it was sent to <code>Func</code> during forward pass and thus was overwritten?</p>

username: albanD
cooked: <p>No worries!</p>
<blockquote>
<p>How would autograd reconstruct original <code>x</code> for backprop</p>
</blockquote>
<p>It does not.<br>
If you modified it inplace, and you save it, you will get the modified version during the backward pass.</p>
<p>For example:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch


class MyFn(torch.autograd.Function):
    @staticmethod
    def forward(ctx, inp):
        print("fw in", inp)
        inp += 2
        print("fw out", inp)
        ctx.mark_dirty(inp)
        ctx.save_for_backward(inp)
        return inp

    @staticmethod
    def backward(ctx, grad_output):
        inp, = ctx.saved_tensors
        print("bw inp value", inp)
        return grad_output


a = torch.rand(2, requires_grad=True)

out = MyFn.apply(a.clone())

out.sum().backward()
</code></pre>
<p>Will print:</p>
<pre><code class="lang-auto">fw in tensor([0.3568, 0.4446], grad_fn=&lt;CloneBackward0&gt;)
fw out tensor([2.3568, 2.4446], grad_fn=&lt;CloneBackward0&gt;)
bw inp value tensor([2.3568, 2.4446], grad_fn=&lt;MyFnBackward&gt;)
</code></pre>
<p>As you can see the backward contains the modified value</p>
How can I turn off one loss but not the other?

username: jkr
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/f/6f18c4f52c5eb581832da202bca7afdcf4d4b350.png" data-download-href="https://discuss.pytorch.org/uploads/default/6f18c4f52c5eb581832da202bca7afdcf4d4b350" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/f/6f18c4f52c5eb581832da202bca7afdcf4d4b350_2_565x500.png" alt="image" data-base62-sha1="fQO4aRrSqxSXQNTK0JduzXUfoDm" width="565" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/f/6f18c4f52c5eb581832da202bca7afdcf4d4b350_2_565x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/6/f/6f18c4f52c5eb581832da202bca7afdcf4d4b350_2_847x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/6/f/6f18c4f52c5eb581832da202bca7afdcf4d4b350_2_1130x1000.png 2x" data-dominant-color="EFF0E5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1131×1000 116 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Say, I am training a network with two data types, A and B. The net always produces two outputs. But it the input is of type B, I only want to backpropagate loss1.</p>
<p>Here is some pseudo-code to illustrate the problem:</p>
<pre><code class="lang-auto">loss1 = Loss1(output1, target1)

if input_is_of_type_B:
	loss2 = torch.tensor(0.0)
else:
	loss2 = Loss2(output2, target2)

loss = (loos1 + loss2)/2
loss.backward()
</code></pre>
<p>Implemented this way, the code does not throw any errors, but will it do what I want/is there a better way?</p>
<p>Possible duplicates:<br>
• <a href="https://discuss.pytorch.org/t/how-to-define-a-suitable-zero-loss/11101" class="inline-onebox">How to define a suitable "zero loss"?</a><br>
• <a href="https://discuss.pytorch.org/t/how-to-attach-a-cost-to-a-graph/74744" class="inline-onebox">How to attach a cost to a graph</a></p>
Gdb debugging access index({}) or size()

username: lino202
cooked: <p>Hi,</p>
<p>I was playing with libtorch. I’ve configured and built with <code>cmake</code>  a really simple example.</p>
<p>cppFrontEnd.cpp</p>
<pre><code class="lang-auto">#include &lt;torch/torch.h&gt;
#include &lt;iostream&gt;

int main() {
  torch::Tensor tensor1 = torch::ones(5);
  std::cout &lt;&lt; tensor1 &lt;&lt; std::endl;
  tensor1.index_put_({2},5);
  std::cout &lt;&lt; tensor1 &lt;&lt; std::endl;
  std::cout&lt;&lt; tensor1.index({2});
  // std::cout&lt;&lt;tensor1.size(0)&lt;&lt;std::endl;


  return 0;
}
</code></pre>
<p>CMakeLists.txt</p>
<pre><code class="lang-auto">cmake_minimum_required(VERSION 3.0 FATAL_ERROR)
project(cppFrontEnd)
set(CMAKE_PREFIX_PATH "/path/to/cppFrontEnd/libtorch")
set(CMAKE_BUILD_TYPE "Debug")
find_package(Torch REQUIRED)
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} ${TORCH_CXX_FLAGS} -O0")
set(CMAKE_CXX_FLAGS_DEBUG "${CMAKE_CXX_FLAGS_DEBUG} ${TORCH_CXX_FLAGS_DEBUG} -O0")
add_executable(cppFrontEnd cppFrontEnd.cpp)
target_link_libraries(cppFrontEnd "${TORCH_LIBRARIES}")
set_property(TARGET cppFrontEnd PROPERTY CXX_STANDARD 14)
</code></pre>
<p>Then with <code>gdb</code> when I tried to access one element of the tensor<br>
with<br>
<code>p tensor1.index({1})</code><br>
I got<br>
<code>Cannot evaluate function -- may be inlined</code></p>
<p>and with</p>
<pre><code class="lang-auto">p tensor1.size(0)
Cannot resolve method at::Tensor::size to any overloaded instance
</code></pre>
<p>Otherwise<br>
p tensor.sizes()[0] works if I uncomment the line <code>std::cout&lt;&lt;tensor1.size(0)&lt;&lt;std::endl;</code></p>
<p>I assume the inline problem is generated by the compiler as <em>if you don’t use or explicitly instantiate some template method it is not generated in resulting binary.</em> then gdb cannot see it and that is why sizes()[] start working when I call tensor1.size() in the source code, am I right?</p>
<p>Is there a way to get a complete debug of the methods and attributes of the objects define in libtorch? Sorry if this is a basic question or if this is not the place to do it.</p>
<p>Thank you</p>

username: ptrblck
cooked: <p>You could try to build your application with debug symbols via <code>cmake --build . --config Debug</code>.<br>
I haven’t checked this approach and would guess that you would also need a debug <code>libtorch</code> build.</p>

username: lino202
cooked: <p>Thanks! I tried to build with the exact same command you provided but the lack of debug info still happens I assume it is due to the lack of debug symbols in <code>libtorch</code> but I have not try this yet.</p>
Neural coverage pytorch

username: Moon_Lee
cooked: <p>Follow the paper <a href="https://escholarship.org/uc/item/3c8107x5" class="inline-onebox" rel="noopener nofollow ugc">Is Neuron Coverage a Meaningful Measure for Testing Deep Neural Networks?</a></p>
<p>Could pytorch provide function to calculate neural coverage of CNN architecture?</p>
<p>Thanks</p>
Json file with poor structring

username: Oussama_Bouldjedri
cooked: <p>I am working on json file that has many dictionaries of the format:</p>
<p>{“serial_number”:"",“timestamp”:’’,“data”:},<br>
{“serial_number”:"",“timestamp”:’’,“data”:}</p>
<p>it is formed this way:</p>
<p>{<br>
“serial_number”:"",“timestamp”:’’,“data”:}<br>
{“serial_number”:"",“timestamp”:’’,“data”:<br>
}</p>
<p>it seems that this is a wrong way of structuring it,  is there a way to use it in this format or I have to edit it to make it look like this :</p>
<p>{<br>
“data”:[<br>
{“serial_number”:"",“timestamp”:’’,“data”:[]},<br>
{“serial_number”:"",“timestamp”:,“data”:[]}<br>
]<br>
}</p>

username: ptrblck
cooked: <p>The question doesn’t seem to be related to PyTorch so you would most likely get a faster and better answer in another board (e.g. StackOverflow).<br>
In any case, I would recommend to clean up your data once in order to avoid any issues further down in the training, which could yield unexpected results. <code>pandas</code> might come handy to process the data (in case it could load and understand it).</p>
How can we train on a complex tensor?

username: NullPointer
cooked: <p>I perform a Fourier transform with my data and then feed it into the network, but I get type mismatches. I believe that this is why <a href="https://github.com/wavefrontshaping/complexPyTorch" rel="noopener nofollow ugc">complexPyTorch</a> still exists, but I was wondering if there is a way to pass a complex tensor into a model without getting a type error. Any example (related to a GRU if possible) that could be provided would be very much appreciated.</p>
<p>Thank you</p>

username: NullPointer
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> if you’re able to, would you please offer some advice related to this? Thank you</p>

username: ptrblck
cooked: <p>Complex modules are not fully supported yet and you could thus try if the modules you want to use would already work. E.g. <code>GRU</code> seems to work on the CPU already:</p>
<pre><code class="lang-python">rnn = nn.GRU(10, 20, 2).to(torch.complex64)
input = torch.randn(5, 3, 10).to(torch.complex64)
h0 = torch.randn(2, 3, 20).to(torch.complex64)
output, hn = rnn(input, h0)
</code></pre>
<p>and yields a warning:</p>
<pre><code class="lang-python">UserWarning: Complex modules are a new feature under active development whose design may change, and some modules might not work as expected when using complex tensors as parameters or buffers. Please file an issue at https://github.com/pytorch/pytorch/issues/new?template=bug-report.md if a complex module does not work as expected.
</code></pre>
<p>so feel free to create a feature request for non-working modules.</p>
Why the checkpoint size becomes smaller after i resave it

username: wadewang
cooked: <p>I trained a model, but i need to modify one of the model state dict, so i use following code to modify the value of specified <code>step</code> key:</p>
<pre><code class="lang-auto">def modify_model_state_dict():
    path = f"/media/wwd/2THardDisk/models/base.pt"
    checkpoint = torch.load(path)  # my checkpoint contains two parts:  'model_state' and "optimizer_state"
    new_state_dict = checkpoint['model_state'].copy()
    new_state_dict['step'] = torch.tensor([1], device='cuda:0')  # reset the step to 1
    new_path = path[:-3] + '_new.pt'
    torch.save({
                "model_state": new_state_dict,
                "optimizer_state": checkpoint["optimizer_state"],
            }, new_path)

modify_model_state_dict()
</code></pre>
<p>After i saved the modified model, i checked its size, found that its size is 526,147,901 Bytes while original model size is 526,153,469 Bytes, so i am curious why the model size becomes smaller, where does the 5568 Bytes go ？ Does this means some infomation is lost ？</p>

username: ptrblck
cooked: <p>By default <code>torch.save</code> should compress the data, so I guess depending on the actually used algorithm changes in the data could result in different file sizes.</p>

username: wadewang
cooked: <p>I did not change the algorithm. Later, I used the modified model to inference, and it seemed to be normal. Recently, I continued training with this modified model, no error was reported, and I saw that the newly saved model size returned to the previous 526,153,469 Bytes. So it’s a mystery, although I don’t know what is the reason behind it, I guess it’s caused by the secondary compression, but I didn’t verify it, since I use it without problems, so I didn’t go into it.</p>
Fresh implementations of Faster-RCNN in PyTorch and TensorFlow

username: Bart
cooked: <p>Hi all,</p>
<p>Last year I was working on implementing Faster R-CNN from scratch using the original paper. Pleased to say that I got it working and spent some time this month porting it to TensorFlow as well and polishing things up a bit. I’ve documented some of my embarrassing setbacks and what I learned from them. Hopefully others will find this useful, even if working on a completely different model.</p>
<p>The repo is here: <a href="https://github.com/trzy/FasterRCNN" class="inline-onebox" rel="noopener nofollow ugc">GitHub - trzy/FasterRCNN: Clean and readable implementations of Faster R-CNN in PyTorch and TensorFlow 2 with Keras.</a></p>
<p>It works quite well, is easy to set up, and I’d like to think it is pretty clean and readable. If nothing else, it’s a fun comparison between PyTorch and TensorFlow.</p>
<p>Thanks,</p>
<p>Bart</p>
Moving a module to a device

username: bbharland
cooked: <p>I am trying to fix: “RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!” from a torchscript module.</p>
<p>Is there a convenient way to move a whole module onto a particular device?  I’ve tried <code>m.to(torch.device('cuda'))</code> and <code>m.cuda()</code></p>
<p>Here is a minimal (not quite working) example:</p>
<pre><code class="lang-auto">class TestModule(nn.Module):
    def __init__(self, net):
        super(TestModule, self).__init__()
        self.net = net
        self.a = torch.rand(2)

    def forward(self, x):
        s = self.net(x)
        return self.a * s

m = TestModule(net)
</code></pre>
<p>Initializing with a network on the cpu:<br>
<code>m.net[-1].weights</code> → cpu<br>
<code>m.a.device</code> → <code>device(type='cpu')</code></p>
<p>Then:<br>
<code>m = m.to(torch.device('cuda'))</code><br>
<code>m.net[-1].weight</code> → <code>device='cuda:0'</code><br>
<code>m.a.device</code> → <code>device(type='cpu'))</code></p>
<p>Thanks!</p>

username: Mazhar_Shaikh
cooked: <p>For an nn.Module, the .to(device) function will send all valid members of the module to the device.<br>
The valid members for this operation is other nn.Modules, Parameters and Buffers.</p>
<p>For your code, these changes would be required.</p>
<pre><code class="lang-auto">class TestModule(nn.Module):
    def __init__(self, net):
        super(TestModule, self).__init__()
        self.net = net
        self.a = nn.Parameter(torch.rand(2))

    def forward(self, x):
        s = self.net(x)
        return self.a * s

m = TestModule(net)
m.to(device)
##m will be on "device"
</code></pre>
Any reason using 2MB in CUDACachingAllocator?

username: devko
cooked: <aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/7ee0712642492ef221a69d3fdf13b607f406bd78/c10/cuda/CUDACachingAllocator.cpp#L102">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/7ee0712642492ef221a69d3fdf13b607f406bd78/c10/cuda/CUDACachingAllocator.cpp#L102" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/7ee0712642492ef221a69d3fdf13b607f406bd78/c10/cuda/CUDACachingAllocator.cpp#L102" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/7ee0712642492ef221a69d3fdf13b607f406bd78/c10/cuda/CUDACachingAllocator.cpp#L102</a></h4>



    <pre class="onebox">      <code class="lang-cpp">
        <ol class="start lines" start="92" style="counter-reset: li-counter 91 ;">
            <li>*/</li>
            <li>
            </li>
<li>namespace {</li>
            <li>
            </li>
<li>using stream_set = std::unordered_set&lt;cuda::CUDAStream&gt;;</li>
            <li>
            </li>
<li>constexpr size_t kMinBlockSize =</li>
            <li>   512; // all sizes are rounded to at least 512 bytes</li>
            <li>constexpr size_t kSmallSize = 1048576; // largest "small" allocation is 1 MiB</li>
            <li>constexpr size_t kSmallBuffer =</li>
            <li class="selected">   2097152; // "small" allocations are packed in 2 MiB blocks</li>
            <li>constexpr size_t kLargeBuffer =</li>
            <li>   20971520; // "large" allocations may be packed in 20 MiB blocks</li>
            <li>constexpr size_t kMinLargeAlloc =</li>
            <li>   10485760; // allocations between 1 and 10 MiB may use kLargeBuffer</li>
            <li>constexpr size_t kRoundLarge = 2097152; // round up large allocations to 2 MiB</li>
            <li>
            </li>
<li>typedef std::bitset&lt;static_cast&lt;size_t&gt;(StatType::NUM_TYPES)&gt; StatTypes;</li>
            <li>
            </li>
<li>void update_stat(Stat&amp; stat, int64_t amount) {</li>
            <li> stat.current += amount;</li>
        </ol>
      </code>
    </pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>When I check code, CUDACachingAllocator try to allocate GPU memory at best fit of 2MB size.</p>
<p>I find somethings like, “X86 support 4k, 2M, 4G page”, “CUDA driver need pinned memory for cudaMemcpy, So pytorch CPU allocator use cudaHostAlloc”…</p>
<p>So I think, 2MB is just good to manage between CPU and GPU.</p>
<p>Is there any more specific reason using 2MB size ?</p>
<p>Thanks</p>
C shared library and Bus error during training

username: LuChuan
cooked: <p>Hi,</p>
<p>I tried to do some preprocess with a C shared library during training, but got an BUS ERROR. ( The reason why I want to use a shared library instead of coding in python is I don’t want to code the same thing in two different language. In addition, I want to maintain the consistency between training and deployment. I found there is a deployment in skimage, but its result is different with mine. Figuring out the difference in the source code of skimage would cost too much time. )</p>
<p>ERROR: Unexpected bus error encountered in worker. This might be caused by insufficient shared memory.</p>
<p>This error should result from my .so library, because this code could work if the .so part was disabled. This shared library works individually, either.</p>
<p>All preprocess finished in a dataloader, I think there shouldn’t be any gpu issue involved and I didn’t sent img to cuda inside dataloader. I used nn.DataParallel to distribute data to multiple GPUs.</p>
<p>Below is how I loaded the library.</p>
<pre><code class="lang-auto">     try: 
            loaded_lib = ctypes.cdll.LoadLibrary( kwargs['libs_path'] )
            print("Successfully loaded ", loaded_lib)
        except Exception as e:
            print(e)
            exit()

        self.function_c = loaded_lib.function_c
        self.function_c.argtypes = [   ndpointer(ctypes.c_ubyte, flags = "C_CONTIGUOUS"),
                                ctypes.c_int,
                                ctypes.c_int,
                                ctypes.c_int, 
                                ndpointer(ctypes.c_float, flags = "C_CONTIGUOUS")
                                 ]
</code></pre>
<p>I didn’t use openmp or pthread  in my C++ code but opencv, and I enabled openmp and pthread when I compiled it.</p>
<p>Does anyone has any idea?</p>
<p>Thanks.</p>
Fix random result in lineaIndex RuntimeError

username: Ardeal
cooked: <p>Hi,</p>
<p>if I don’t fix random, the following code in yolov3 could be run correctly.<br>
if I fix random with the following code, I experience the following error:</p>
<pre><code class="lang-python">
import random
import numpy as np
torch.manual_seed(5)
random.seed(5)
np.random.seed(5)
torch.backends.cudnn.benchmark = False
torch.use_deterministic_algorithms(True)
</code></pre>
<pre><code class="lang-python"># t shaple: 75X80
#n=75
#i=0
#tcls shape: list with len 3, each shaple: 75, 55, 22

t[range(n), tcls[i]] = self.cp
</code></pre>
<pre><code class="lang-python">Exception has occurred: RuntimeError
linearIndex.numel()*sliceSize*nElemBefore == value.numel()INTERNAL ASSERT FAILED at "/pytorch/aten/src/ATen/native/cuda/Indexing.cu":253, please report a bug to PyTorch. number of flattened indices did not match number of elements in the value tensor751
  File "/trainer/yolov3_train/utils/loss.py", line 142, in __call__
    t[range(n), tcls[i]] = self.cp
  File "/trainer/trainer_yolo.py", line 558, in train
    loss, loss_items = compute_loss(pred, targets.to(device))  # loss scaled by batch_size
  File "/trainer/trainer_yolo.py", line 128, in trainer_yolov3
    self.train(hyp, opt, device, tb_writer=None)
  File "/runner_full.py", line 325, in compressor_runner
    results = trainer.trainer_yolov3(None, None, 99)
  File "runner_full.py", line 405, in &lt;module&gt;
    runner(ectp)
</code></pre>

username: ptrblck
cooked: <p>Could you update PyTorch to the latest nightly release (or <code>1.10.1</code>) as this was a known and already fixed issue?</p>

username: Ardeal
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="142122">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>r <code>1.10.1</code>)</p>
</blockquote>
</aside>
<p><a class="mention" href="/u/ptrblck">@ptrblck</a> ,<br>
Thank you!<br>
In order to use 1.10.1, I have to upgrade cuda.<br>
I will try it.</p>

username: ptrblck
cooked: <p>Note that the pip wheels and conda binaries ship with their own CUDA runtime (as specified during the installation) so you would only need a compatible NVIDIA driver (unless you want to build PyTorch or a custom CUDA extension from source which would then use your local CUDA toolkit installation).</p>
LBFGS optimizer

username: RuiZhangJerry
cooked: <p>Hi, I want to know how to implement the LBFGS optimization in libtorch?</p>

username: ptrblck
cooked: <p><a href="https://github.com/pytorch/pytorch/blob/e0643fa3fc5cd2dd41dce07b34eba8f45b118f15/test/cpp/api/optim.cpp#L20-L253">This test</a> might be helpful as a template.</p>

username: RuiZhangJerry
cooked: <p>Thank you very much! <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=12" title=":smile:" class="emoji" alt=":smile:"></p>
Bilinear interpolation Grid

username: Javier_Rodriguez_Pui
cooked: <p>Hi there,  I have a question regarding Bilinear Interpolation. According to Wikipedia, a Bilinear Interpolation depends on the pixel(x,y).<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/2/72a8c3d1af5b935a8d9371b60667e0ec9e7452a0.png" data-download-href="https://discuss.pytorch.org/uploads/default/72a8c3d1af5b935a8d9371b60667e0ec9e7452a0" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/2/72a8c3d1af5b935a8d9371b60667e0ec9e7452a0_2_690x126.png" alt="image" data-base62-sha1="gmk0xPxh8urbpvmM80SX0KqN9Uk" width="690" height="126" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/2/72a8c3d1af5b935a8d9371b60667e0ec9e7452a0_2_690x126.png, https://discuss.pytorch.org/uploads/default/optimized/3X/7/2/72a8c3d1af5b935a8d9371b60667e0ec9e7452a0_2_1035x189.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/7/2/72a8c3d1af5b935a8d9371b60667e0ec9e7452a0.png 2x" data-dominant-color="000000"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1065×196 17.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/8/7/8788e24bb6e2de6c337d039216c911d679775bcf.png" alt="image" data-base62-sha1="jkZJkbQQBsAl2awz91OAAQO429p" width="220" height="208"></p>
<p>I am wondering If is any way to know what is the value of  x0, x1, y1 and y2. Normally I use the following call:<br>
new_color = F.grid_sample(color, pixels, padding_mode=“border”)</p>
<p>For further research, I am calculating analytically the derivative of the function posted above but I need the constants x0, x1, y1 and y2.<br>
Any suggestion?</p>
I want to move all the tensor to GPU at once

username: HnuXt
cooked: <p>resently , I got a deep learning project from github , it  is about polar code decoder,(polar code : a channel code)，but when i try to train the model , i find that all the tensors are computing on the CPU,it is so slowly, so I want to make use of my GPU, I try to use the command:"  x=x.cuda()" but there will be a warning : all the tensors should be computed on one devise, it means that there are some tensors  that i haven’t paid attention to, so I want somebody can help me ! whether there exgist a comand that can move all tensors to GPU at once, oh~, I have already tried  this:<br>
“dd=locals()<br>
d_tmp=[value.cuda() for key,value in dd.items() if type(value)==torch.Tensor]”<br>
thank you!!!very much!</p>

username: ptrblck
cooked: <p>You could try to change the default tensor type to a device tensor via e.g. <code>torch.set_default_dtype</code>, but I would generally not recommend to try to push <em>all</em> tensors to the device, but use the explicit approach instead.<br>
The error should show the line of code which is failing and which tensors are included, so check the <code>.device</code> argument of these tensors and make sure they are on the GPU.</p>
Torchserve process

username: mscorp
cooked: <p>I’m a little confused about the torchserve process, even after successfully setting up the densenet example.  Specifically, the archiving of the model.</p>
<p>In all of the examples, the model file has to:<br>
subclass an nn module<br>
implement a forward function</p>
<p>the pth file has to be a trained, serialized model of the type classified above.</p>
<p>Here’s my confusion.<br>
using this example: <a href="https://github.com/pytorch/serve/blob/master/examples/image_classifier/densenet_161/model.py" class="inline-onebox" rel="noopener nofollow ugc">serve/model.py at master · pytorch/serve · GitHub</a></p>
<p>First of all, the model file included with the densenet example does not implement a forward function.<br>
I’m guessing its inherited somewhere, but if that’s the case, why complicate the example for noobs      like myself?</p>
<p>Second, what does the forward function actually need to contain?  Again. I’ve seen and read the references in the pytorch and torchserve tutorials:<br>
<a href="https://pytorch.org/tutorials/beginner/former_torchies/nnft_tutorial.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/beginner/former_torchies/nnft_tutorial.html</a><br>
Is the forward function just telling it where the input goes, and what the output looks like? why isn’t it defined per layer?</p>
<p>What would it look like for an lstm with several timesteps in its sequence?  (you have no lstm pytorch/torchserve examples in this tutorial. Just a simple “how to lstm” using pre-ordained weights)</p>
<p>Second, for the trained model, other than performing training as one would for any other type of model, and writing out the state dict, are there any other specifications that have to be followed?</p>
<p>So I guess i’m confused about both the process, and the torchserve’s actual requirements.  Unfortunately, the tutorials just caused more confusion. I’ve looked at probably 6 or 7 others, and they all do it a bit differently, and none of them specifically follow the archiver’s recommendation, or structure.</p>
<p>Is the process (for an lstm)<br>
train the model using the data i have<br>
write out a state dict representing the trained model<br>
include the model.py file that defines init and a forward function (still a little lost on the forward function)<br>
archive the whole thing together in a .mar file<br>
torchserve stuff (i’ve got that handled)</p>
<p>If possible, could one provide a detailed example of exactly what the archiver wants, what a forward function actually does (conceptually, from a torchserve perspective) and how this all fits together.</p>
<p>Thank you very much for your time</p>
How to define a vector function in pytorch from R to R^n in order to obtain its derivative?

username: Kingsman_Rojas
cooked: <p>If I have a tensor x.<br>
Can I define a vector function, say f(x)=(3x, x+2), in such a way that I can then obtain df/dx?</p>
<p>I hope my question is understood. In a nutshell: I want a way to define such a vector function, from which I can get its gradient.</p>

username: KFrank
cooked: <p>Hi Juan!</p>
<aside class="quote no-group" data-username="Kingsman_Rojas" data-post="1" data-topic="142111" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kingsman_rojas/48/45930_2.png" class="avatar"> Kingsman_Rojas:</div>
<blockquote>
<p>Can I define a vector function, say f(x)=(3x, x+2), in such a way that I can then obtain df/dx?</p>
</blockquote>
</aside>
<p>Does <a href="https://pytorch.org/docs/1.10.1/generated/torch.autograd.functional.jacobian.html" rel="noopener nofollow ugc">jacobian()</a> do what you want?</p>
<p>If you implement your function using (differentiable) pytorch tensor<br>
functions, autograd and <code>jacobian()</code> should compute for you the<br>
full set of partial derivatives.</p>
<p>Best.</p>
<p>K. Frank</p>
How to write backward function with two inputs

username: Maxwell_Albert
cooked: <p>Now here is a distributed computing problem.<br>
I write this function for communication between GPUs</p>
<pre><code class="lang-auto">class RemoteReceive(autograd.Function):
    @staticmethod
    def forward(ctx, input:torch.tensor = torch.tensor(1.0),from_rank:int = 0):
        dim = torch.tensor(1.0)
        dist.recv(dim,from_rank)
        size = torch.rand(int(dim))
        dist.recv(size,from_rank)
        x = torch.zeros(tuple(size.int()))
        
        dist.recv(x, from_rank)
        return x

    @staticmethod
    def backward(ctx, grad_output:torch.tensor = torch.tensor(1.0),to_rank:int = 0):
        dist.send(torch.tensor(grad_output.dim()*1.0),to_rank)
        #print("send to dim",to_rank,input.size.dim())
        dist.send(torch.tensor(grad_output.size())*1.0,to_rank)
        dist.send(grad_output, to_rank)
        return None
</code></pre>
<p>I need two input in backward. One for send to another GPU and another for choosing the rank of the GPU, but when I reference this function. like</p>
<pre><code class="lang-auto">        x = torch.rand(1)
        y = torch.rand(1)
        RemoteReceive.apply(x,1)
        print(rank,"finish forward")
        x.backward(x ,to_rank = 1)
</code></pre>
<p>A error occur<br>
TypeError: backward() got an unexpected keyword argument ‘to_rank’<br>
So how can I put these two arguments to backward,please.</p>

username: albanD
cooked: <p>Hi,</p>
<p>The backward that you implement is not the same as the one that gets called. The one you write will only work with gradients.</p>
<p>You can solve this by simply passing both the <code>from_rank</code> and <code>to_rank</code> to the apply (and so take both as input for your forward).<br>
And in your forward save anything that you need for the backward on the <code>ctx</code>: <code>ctx.to_rank = to_rank</code>.</p>

username: Maxwell_Albert
cooked: <p>Thank you!<br>
By the way, is there any possible platform that I can learn some pytorch distributed through tcp connection.<br>
What I saw on pytorch doc are some<code> to.(device)</code> type.</p>

username: albanD
cooked: <p>I would look into pytorch’s RPC system. It will allow you to send/receive Tensors (and other pytorch objects) easily over a few network layers.</p>
Help with policy gradient update with lstm

username: Intech
cooked: <p>Hi I have the following network with the forward function shown here:</p>
<pre><code class="lang-python">class RecurrentPNetwork(nn.Module):
    ''' Recurrent policy '''
    def __init__(self, state_space,action_space,hidden_space=64):
        super(RecurrentPNetwork,self).__init__()
        self.hidden_space=hidden_space

        self.fc1 = nn.Linear(state_space,hidden_space)
        self.rnn = nn.LSTM(hidden_space,hidden_space)#,batch_first=True)
        self.fc2 = nn.Linear(hidden_space,action_space)
        self.hidden_memory = []

    def forward(self,x):
        x = F.relu(self.fc1(x))
        if len(self.hidden_memory) == 0:
            h_t = None
        else:
            h_t = self.hidden_memory[-1]

        x, (new_h,new_c) = self.rnn(x,h_t)
        new_h = new_h.detach().requires_grad_()
        new_c = new_c.detach().requires_grad_()
        self.hidden_memory.append((new_h,new_c))
        out = F.softmax(self.fc2(new_h),dim=-1)
        return out
</code></pre>
<p>Here I encode the action history in the hidden state which I use to estimate the probability for the new action hence I have to manually unroll the model. When I optimize the model I want to take the history of the hidden states into account. For that I introduce the hidden_memory list. So that I can store them.</p>
<p>First the h_0 and c_0 are initialized to 0 and the model continues for 1 episode. and it works. When the second episode begins I get this error: one of the variables needed for gradient computation has been modified by an inplace operation.<br>
This happens at the lstm cell because now h_t is taken from the memory list. The way to solve that is to use detach() or detach().requires_grad_(), but then I miss out on these gradients and the model doesnt work. What should I do ?</p>
Autograd for creation of sparse_coo_tensor

username: yajivunev
cooked: <p>I am training a model with a custom loss function which involves sparse tensors being created.</p>
<p>Specifically, in the forward pass of the loss function:<br>
I <code>ravel</code> my prediction and labels arrays, <code>stack </code>them, and use them as indices to create a <code>sparse_coo_tensor</code>, with the values as <code>torch.ones</code> and shape given by <code>[(torch.amax(raveled_labels) + 1,(torch.amax(raveled_pred) + 1]</code>. I then make a few more sparse tensors by performing some <code>index_select</code> ops on the first created sparse tensor, perform some <code>sparse.sum</code> and <code>mul</code> ops on the newly created sparse tensors, and some more arithmetic with the returned values to output a final number.</p>
<p>Correct me if i am wrong, but the creation of a sparse tensor in the forward pass of a loss function does not have autograd support; all the other ops work with autograd. It would be really neat if autograd/backward works with this entire chain of ops – might just revolutionize computer vision <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:"></p>
<p>If I try to train a model with this custom loss function as is (i.e without writing my own backward), I get a <code>CUDA error: invalid configuration argument.</code></p>
<p>Thanks everyone! happy to answer more questions about this use case!</p>
Pytorch relu6 use more gpu memory than relu

username: KeyKy
cooked: <p>I found that torchvision.models.mobilenetv2+relu (i exchange all relu6 to relu) can support 450+ (450,3,224,224) batchsize in training (Tesla P40). However, the default mobilenetv2+relu6 can’t run in 384 batch size (384, 3, 224, 224). Why relu6 cost more gpu memory than relu? All activations use inplace=True mode. Pytorch 1.10.0</p>
Adding Prior Knowledge to Transformer

username: Edo_Dotan
cooked: <p>Hi,<br>
I’m interested in injecting a prior tensor (already computed) into a transformer. I think the best way to do it is to register a forward hook on the last encoder layer, detach the prior tensor, and fuse both tensors. Is this the correct way to do it?</p>
<h1>
<a name="something-like-consider-both-tensors-of-the-same-size-1" class="anchor" href="#something-like-consider-both-tensors-of-the-same-size-1"></a>Something like (consider both tensors of the same size):</h1>
<p>def hook(model, input, output):<br>
prior = prior.detach()<br>
input = torch.add(input, prior, alpha=0.1)</p>
<p>…<br>
transformer.models[0].encoder.layers[-1].final_layer_norm.register_forward_hook(hook)<br>
train model regularly…<br>
…</p>
<p>Thanks!</p>
Link-Time Optimization support

username: veritas
cooked: <p>I am curious as to whether there is any way to enable llink-time optimization (LTO) when building PyTorch from source.</p>
<p>NVIDIA has introduced LTO as a stable(?) feature as of CUDA 11.2. <a href="https://developer.nvidia.com/blog/improving-gpu-app-performance-with-cuda-11-2-device-lto" rel="noopener nofollow ugc">https://developer.nvidia.com/blog/improving-gpu-app-performance-with-cuda-11-2-device-lto</a></p>
<p>Also, both LLVM and GCC support LTO via compile flags.</p>
<p>Though I am no expert, LTO appears to give very good speedups at the cost of longer compile times.</p>
<p>Is LTO currently supported on PyTorch? If so, how do I enable it?</p>
<p>To be more precise, I would like to know how to specify compiler and linker options for PyTorch during the build.</p>
<p>To enable LTO and other optimizations, the compiler must be given -flto flags at both compile and link stages. However, the structure of PyTorch is so complicated that it is hard for me to tell how to give flags for specified stages and check if they have been given properly.</p>
Convert tensor to Parameter (while keeping the graph)

username: Zen
cooked: <p>I want to create a neural network generating the parameters of another NN. The NN ‘A’ generates ‘B’.</p>
<p>But there is a problem: the output of ‘A’ is a ‘tensor’ (with requires_grad = True) while the parameters of ‘B’ are ‘Parameter’. Since I want to backward the gradient up to the parameters of ‘A’, I am looking for a conversion tensor -&gt; Parameter that keeps the computational graph.</p>
<p>A naive conversion ‘w = Parameter(y)’, where w is the generated parameter of ‘B’, and y is the output of ‘A’, does not work: the computational graph is cut between w and y (the gradient of the parameters of ‘A’ are None, while the gradient of w, which should be non-leaf, is not None).</p>

username: InnovArul
cooked: <p>It depends on what ‘B’ does. Have you looked into <code>torch.nn.functional</code> APIs?</p>

username: Zen
cooked: <p>‘B’ is a NN with fully connected layers and convolutional layers, whose parameters are of type ‘Parameter’.<br>
So, I could replace the usual layers by layers using torch.nn.functional functions… but I need to rewrite each layer I want to use, then. Is this the only solution?</p>

username: InnovArul
cooked: <p>if you want the network ‘A’ to predict the parameters of network ‘B’,  this could be the only easier way than meddling around with default <code>nn</code> layers.</p>

username: Brando_Miranda
cooked: <p>Just responding to the actual title of the question. What is wrong with using <code>nn.Parameter(old_tensor)</code></p>
<p>where <code>old_tensor</code> is of type <code>torch.Tensor</code>. Does that not work for you? Why does it not?</p>

username: Zen
cooked: <p>As I mentioned, <code>new_tensor = nn.Parameter(old_tensor)</code> breaks the computational graph of autograd.</p>
<p>In other words, <code>new_tensor</code> does not contain any information about the way <code>old_tensor</code> has been computed, which I want.</p>

username: zacharynew
cooked: <p>Hi <a class="mention" href="/u/zen">@Zen</a> did you ever find a solution to this problem? I am now facing exactly the same issue.</p>

username: CNguyen
cooked: <p>For one that is still looking for a solution, you might find this <a href="https://discuss.pytorch.org/t/assign-parameters-to-nn-module-and-have-grad-fn-track-it/62677">thread</a> useful. There are 2 ways to do it:</p>
<ol>
<li>Delete the weights of each layer manually and re-assign by the tensor of interst (with history)</li>
<li>Use a package from <a href="https://github.com/SsnL/PyTorch-Reparam-Module" class="inline-onebox" rel="noopener nofollow ugc">GitHub - SsnL/PyTorch-Reparam-Module: Reparameterize your PyTorch modules</a>, but note that it might not work with <code>batchnorm</code> layer when <code>track_running_stats=True</code>.</li>
</ol>
<p>Another useful package that is supported directly from PyTorch is <code>functorch</code> with <code>make_functional</code>.</p>
Combining Parameterlist in PyTorch

username: blade
cooked: <p>I am trying to combine two <code>ParameterList</code>s in Pytorch. I’ve implemented the following snippet:</p>
<pre><code>import torch

list = nn.ParameterList()
for i in sub_list_1:
    list.append(i)
for i in sub_list_2:
    list.append(i)
</code></pre>
<p>Is there any functions that takes care of this without a need to loop over each list?</p>

username: ptrblck
cooked: <p>You could unwrap the sub lists and pass their content to <code>nn.ParameterList</code> as seen here:</p>
<pre><code class="lang-python">sub_list_1 = nn.ParameterList([nn.Parameter(torch.ones(1))])
sub_list_2 = nn.ParameterList([nn.Parameter(torch.ones(1) * 2)])
param_list = nn.ParameterList([*sub_list_1, *sub_list_2])

sub_list_1 = [nn.Parameter(torch.ones(1))]
sub_list_2 = [nn.Parameter(torch.ones(1) * 2)]
param_list = nn.ParameterList([*sub_list_1, *sub_list_2])
</code></pre>