akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
AttributeError: '_OpNamespace' 'falkon' object has no attribute 'mem_get_info'

username: Elisha
cooked: <p>Hi,<br>
I am trying to run Falkon with GPU and getting the below error:</p>
<p>File “/usr/local/lib/python3.10/dist-packages/falkon/utils/devices.py”, line 92, in _get_gpu_device_info<br>
mem_free, mem_total = mem_get_info(g)<br>
File “/usr/local/lib/python3.10/dist-packages/falkon/c_ext/<strong>init</strong> .py”, line 17, in call_cuda<br>
return getattr(torch.ops.falkon, name)(*args, **kwargs)<br>
File “/usr/local/lib/python3.10/dist-packages/torch/_ops.py”, line 569, in <strong>getattr</strong><br>
raise AttributeError(<br>
AttributeError: ‘_OpNamespace’ ‘falkon’ object has no attribute ‘mem_get_info’</p>
<p>Please help me resolve this.</p>
<p>Thanks!</p>

username: ptrblck
cooked: <p>This error seems to be specific to the <code>falkon</code> repository, so I would recommend checking with the authors if this error was already seen.</p>
Access 4D tensor with 2D indices

username: mshooter
cooked: <p>Hi everyone,</p>
<p>I am trying to access values using torch.gather<br>
I have a tensor of shape [B,K,64,64] and a tensor which represents the 2D indices [B,K,2]<br>
The resulting tensor should be [B,K,1]</p>

username: ptrblck
cooked: <p>Direct indexing should work:</p>
<pre><code class="lang-python">B, K = 2, 3
x = torch.randn(B, K, 64, 64)
idx = torch.randint(0, 64, (B, K, 2))

out = x[torch.arange(x.size(0)).unsqueeze(1), torch.arange(x.size(1)), idx[:, :, 0], idx[:, :, 1]]
print(out.shape)
# torch.Size([2, 3])
</code></pre>
How to extract the images patches from the feature maps after convolution operation?

username: MingHongL
cooked: <p>Hi all<br>
I don’t know how to extract the images patches from the feature maps after convolution operation in forward propagation. Because, I want to use this images patches to do some operations.<br>
I know that this operation which extract the images patches is to use the function such as  tf.image.extract_image_patches in Tensorflow , but I can’t find the similar function in Pytorch.</p>
<p>So… Could you tell me how to extract the images patches in Pytorch? Thanks！</p>

username: MariosOreo
cooked: <p>Hello,</p>
<p>You could return the feature maps in forward pass, and use <a href="https://pytorch.org/docs/stable/tensors.html#torch.Tensor.unfold" rel="nofollow noopener">torch.Tensor.unfold</a> to extract the images patches.</p>

username: MingHongL
cooked: <p>Thanks for your reply! Your suggestions are really helpful.</p>

username: abask61
cooked: <p>I am facing the same problem. Were you able to figure out how to do this?</p>
Why could not quantize LSTMCell using static custom quantization

username: Ahmed_Louati
cooked: <p>Is there a way to quantize LSTMCell using custom modules like LSTM?<br>
I tried to implement this manually (by replacing lstm_utils with lstmcell_utils that I have made) but I got this error:<br>
<code> AttributeError: 'tuple' object has no attribute 'numel'</code></p>
<p>I don’t understand why it is not possible when you have managed to do it for the LSTM layer (which normally is more complicated).</p>
<p>I could not have a graph module with quantizer and dequantizer blocks inserted!</p>
<p>I am using torch.fx</p>
<p>Looking forward to hearing from you!<br>
Thanks,</p>

username: andrewor
cooked: <p>Hi Ahmed,</p>
<p>The short answer is no, unfortunately. This would take significant engineering work since we made LSTM a special case throughout the quantization codebase. The error you’re seeing is saying that an observer got a tuple as input, which it doesn’t understand because it was expecting a single tensor. This is because LSTM (and LSTMCell) outputs a tuple, not a single tensor. To make this work you will need a lot of special code to mutate the graph so that each observer gets a single tensor as input instead. If you’re really determined to make this work you could follow this PR: <a href="https://github.com/pytorch/pytorch/pull/85068" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/pull/85068</a>. However, we probably won’t be able to merge this change into the main branch. Feel free to let me know if I can provide any additional context.</p>
<p>Best,<br>
-Andrew</p>

username: Ahmed_Louati
cooked: <p>Hello <a class="mention" href="/u/andrewor">@andrewor</a>,<br>
Thank you for your reply!</p>
<p>Actually, I changed my models’ architecture and replaced LSTMCell with the LSTM layer to make it quantizable.<br>
It would be so helpful if you could take a look at this issue where I want to custom quantize the LSTM layer. <a href="https://discuss.pytorch.org/t/torch-ao-quantization-fx-problem-with-custom-lstm-quantization/183138/7" class="inline-onebox">Torch.ao.quantization.fx: Problem with custom LSTM quantization - #7 by Ahmed_Louati</a></p>
How to load partitioned parquet files that doesn’t fit in memory

username: Cade_Reid
cooked: <p>Hi, I’m trying to find a good solution for creating a Dataset class to load batches from “chunked” parquet files (each file consists of thousands of records). The entire dataset (all parquet files) can’t fit on memory. The requirement is to have the batch size flexible and at least be able to shuffle within a buffer. (I also have a copy of the dataset but in saved as .npy files)</p>
<p>By the way, if you think that there is a better way to store the dataset besides storing them as multiple large parquet files, I would be very interested to know about it!</p>
<p>Thanks.</p>
<p>Cade</p>
Is this a assert a bug?

username: AT1999
cooked: <p>I am using the following code <a href="https://github.com/traveller59/spconv/blob/master/example/mnist/mnist_net_transform.py" rel="noopener nofollow ugc">link</a> where it uses the convert_fx from torch.ao.quantization.quantize_fx</p>
<p>However, the following assertion is fired:</p>
<pre><code class="lang-auto">_lower_to_native_backend.py", line 606, in _lower_static_weighted_ref_module
    assert(len(ref_node.args) == 1)
AssertionError
</code></pre>
<p>When I comment out this assert everthing seems to work fine. The question is if this assert is correct?</p>

username: andrewor
cooked: <p>Hi Ali,</p>
<p>Would you mind sharing two things that would help us debug?</p>
<ol>
<li>What op is “ref_node” when it fails? (<code>print(ref_node)</code>)</li>
<li>What are the args of “ref_node”? (<code>print(ref_node.args)</code>)</li>
</ol>
<p>Thanks,<br>
-Andrew</p>
Smarter Pooling Mechansims

username: sumona
cooked: <p>Hi, I am currently using global max pooling and global mean pooling for a Graph Convolutional Network using PyTorch Geometric. Since my input data is in a one-hot encoded format I believe such pooling mechanisms are not sufficient to hold enough information to train my model. Are there better pooling mechanisms suitable for one-hot encoded format data? As a beginner, any suggestions or guidance will be really appreciated.</p>
Warping layer in PWC-Net

username: hosein_warp
cooked: <p>i am researching about PWC_Net from below paper:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/6/064f9c8812c6700deb7c9fc6c250c71bd69c36cb.png" data-download-href="https://discuss.pytorch.org/uploads/default/064f9c8812c6700deb7c9fc6c250c71bd69c36cb" title="PWC_Net"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/6/064f9c8812c6700deb7c9fc6c250c71bd69c36cb_2_619x499.png" alt="PWC_Net" data-base62-sha1="TPqKXkze4Rq3HtPtnwLYwlMmcX" width="619" height="499" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/6/064f9c8812c6700deb7c9fc6c250c71bd69c36cb_2_619x499.png, https://discuss.pytorch.org/uploads/default/original/3X/0/6/064f9c8812c6700deb7c9fc6c250c71bd69c36cb.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/0/6/064f9c8812c6700deb7c9fc6c250c71bd69c36cb.png 2x" data-dominant-color="E9E9E8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">PWC_Net</span><span class="informations">753×608 135 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
i am confused in "warping layer " section in this paper.<br>
please explain me in simple words about warping layer<br>
thanks a lot.</p>
Delete model from GPU/CPU

username: mundus131
cooked: <p>Hi<br>
I have a big issue with memory. I am developing a big application with GUI for testing and optimizing neural networks. The main program is showing the GUI, but training is done in thread. In my app I need to train many models with different parameters one after one. To do this I need to create a model for each attempt. When I train one I want to delete it and train new one, but I cannot delete old model. I am trying to do something like this:</p>
<pre><code class="lang-auto">del model
torch.cuda.empty_cache()
</code></pre>
<p>but GPU memory doesn’t change,</p>
<p>then i tried to do this:</p>
<pre><code class="lang-auto">model.cpu()
del model
</code></pre>
<p>When I move model to CPU, GPU memory is freed but CPU memory increase.<br>
In each attempt of training, memory is increasing all the time. Only when I close my app and run it again the all memory is freed.</p>
<p>Is there a way to delete model permanently from GPU or CPU?</p>

username: ptrblck
cooked: <p>I cannot reproduce the issue using a recent master build:</p>
<pre><code class="lang-python">model = models.resnet18()
print(torch.cuda.memory_allocated())
&gt; 0

model.cuda()
print(torch.cuda.memory_allocated())
&gt; 46861312

del model
print(torch.cuda.memory_allocated())
&gt; 0
</code></pre>

username: naufqi
cooked: <p>i have this issue as well, but if you this</p>
<blockquote>
<p>model = model.to(‘cuda’)</p>
</blockquote>
<p>it will free the gpu memory when you delete the <code>model</code></p>

username: Surya_Narayanan
cooked: <p>How do you clear models from a list? <code>del</code> doesn’t work <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12" title=":frowning:" class="emoji" alt=":frowning:" loading="lazy" width="20" height="20"></p>
<p>For reference- check my colab notebook: <a href="https://colab.research.google.com/drive/1I4XagAgXBkKOic8hQTklAta3Chwnh6F1?usp=sharing" class="inline-onebox" rel="noopener nofollow ugc">Google Colab</a></p>

username: yiftach
cooked: <p><a class="mention" href="/u/surya_narayanan">@Surya_Narayanan</a> when I run your supplied code, the last check of allocated memory (<code>print(torch.cuda.memory_allocated()/1024**2, '3')</code>) gives me <code>0.0 3</code>. Is this not what you got?</p>
Loading CUDA and C++ extension slow

username: Kai_Lan
cooked: <p>I am using the <code>load</code> function from <code>torch.utils.cpp_extension</code>. However, the compile time is very long even when I make small changes to my <code>.cu</code> files. When I check the processes, it seems that it is only using one thread.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/e/ee5ce0950645a72d2f2035d10b24263cba24618a.png" data-download-href="https://discuss.pytorch.org/uploads/default/ee5ce0950645a72d2f2035d10b24263cba24618a" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/e/ee5ce0950645a72d2f2035d10b24263cba24618a.png" alt="image" data-base62-sha1="y0ECQGmuwF0clBAxcnszJLnsYFY" width="690" height="29" data-dominant-color="334D1A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1474×64 6.69 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Is there a way to speed it up? My argument list looks like this for now:</p>
<pre><code class="lang-auto">mymodule = load(name='mymodule',
                  sources=[f'{SOURCE_PATH}/torch_extension/mymodule.cpp', f'{SOURCE_PATH}/torch_extension/mymodule_kernel.cu'])
</code></pre>
Cuda tensor to cuda surface

username: keithlostracco
cooked: <p>I’m writing a plugin for a graphics application where I am able to copy data directly from a Cuda kernel to a Cuda surface abstraction provided by the host program. Is there a straightforward method to directly copy data from a <code>torch::Tensor</code> object on the GPU to a CUDA surface abstraction object?</p>
<p>Any suggestions or insights would be greatly appreciated</p>
<p>Thanks,<br>
Keith</p>

username: eqy
cooked: <p>You can call <code>.cpu()</code> to get a copy of the <code>Tensor</code> in host memory. Calling <code>.data_ptr()</code> on that host memory tensor will give you a pointer to the flattened data in host memory.<br>
The numpy conversion code in C++ provides an example of how this is done: <a href="https://github.com/pytorch/pytorch/blob/fbe2a7e50a940ba7a12b003241a2699f7a731afb/torch/csrc/utils/tensor_numpy.cpp#L164" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/blob/fbe2a7e50a940ba7a12b003241a2699f7a731afb/torch/csrc/utils/tensor_numpy.cpp#L164</a></p>

username: keithlostracco
cooked: <p>Thanks.</p>
<p>I would like to avoid the copy to CPU memory though and keep the data on the GPU (for performance). Basically I’m evaluating a relatively lightweight model that is outputting many times per second so the copy to the CPU and back again is a substantial hit. I’m looking for a GPU to (same) GPU solution… I’ve succeeding with this approach with Python but I’m trying to shave off a few more milliseconds of processing/copying time.</p>

username: eqy
cooked: <p>You should be able to access a GPU <code>data_ptr()</code> in your own CUDA kernels in a similar way.</p>

username: keithlostracco
cooked: <p>Yep that’s exactly what ended up doing. Thanks!</p>
Retrieve tensors after a cuda device side error

username: Samuele_Bortolato
cooked: <p>I’m writing some custom code (for a deep RL project) and I’m getting a device side assert triggered. I was wondering if it’s technically possible to clear the error flag and retrieve the tensors stored in the GPU memory regardless of the fact that some of them may have incorrect values.</p>

username: ptrblck
cooked: <p>Once an <code>assert</code> is triggered the CUDA context is corrupt and executing CUDA operations would reraise errors. I would not recommend trying to read any of the data as it could also be corrupt.</p>

username: Samuele_Bortolato
cooked: <p>What I don’t understand is, how can data not involved with the operations that caused the error be corrupted? I would guess that only the memory locations accessed in the kernell calls involved with the error can be corrupted, not the other tensors in device memory.<br>
Regardless of the fact that some tensors may be corrupted, getting the tensors at the moment of the error would greatly simplify the debugging, especially if the error is a rare logic error. Also, being able to access the tensors and selectively saving the parts that are “safe” would mean not wasting days of training in case one error shows up late in the training.<br>
I agree that it’s not advisable in general for the non advanced users, but is it possible to turn off the warning and still retrieve the data? If pytorch doesn’t natively support it, is it possible to bypass it and call a cudaMemcpy and retrieve the contet of the gpu memory in other ways?</p>

username: ptrblck
cooked: <p>No, there is no way to recover from a “sticky” device-side assert as it’s corrupting the <strong>CUDA context</strong>.<br>
Non-sticky errors, such as OOMs via <code>cudaMalloc</code>, are more forgiving and you can continue the execution by e.g. allocating less memory. However, once a sticky device-side assert is triggered, the CUDA context is in an invalid state. It would be similar to trying to read any data from the host after a segfault was triggered.</p>
Error creating latexpdf

username: ericauld
cooked: <p>Hello, I’m trying to create a pdf of the docs.</p>
<pre><code class="lang-auto">git clone https://github.com/pytorch/pytorch
cd pytorch/docs
make latexpdf
</code></pre>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "/Users/ericauld/pytorch/docs/source/scripts/build_opsets.py", line 74, in &lt;module&gt;
    main()
  File "/Users/ericauld/pytorch/docs/source/scripts/build_opsets.py", line 57, in main
    aten_ops_list = get_aten()
                    ^^^^^^^^^^
  File "/Users/ericauld/pytorch/docs/source/scripts/build_opsets.py", line 19, in get_aten
    parsed_yaml = parse_native_yaml(NATIVE_FUNCTION_YAML_PATH, TAGS_YAML_PATH)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/torchgen/gen.py", line 235, in parse_native_yaml
    _GLOBAL_PARSE_NATIVE_YAML_CACHE[path] = parse_native_yaml_struct(
                                            ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/torchgen/gen.py", line 167, in parse_native_yaml_struct
    error_check_native_functions(rs)
  File "/opt/homebrew/lib/python3.11/site-packages/torchgen/gen.py", line 277, in error_check_native_functions
    assert len(base_func_map[out_of_place_base_name]) &gt; 0, (
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: resize_as_ is marked with tag: inplace_view. The codegen expects there to be a corresponding out-of-place view op with the name 'resize_as_' and matching schema, but it didn't find one. 
make: *** [opset] Error 1
</code></pre>
<p>Any ideas? Thanks</p>
Simple cnn mnist c++ example crash,please help

username: Ziyun_Song
cooked: <p>example from pytorch/examples/blob/main/cpp/mnist/mnist.cpp<br>
my os is win10,  vs2019 x64, cpu, libtorch version: 1.12.1 cu113</p>
<p>cannot train or test<br>
train(epoch, model, device, *train_loader, optimizer, train_dataset_size);<br>
test(model, device, *test_loader, test_dataset_size);<br>
both will cause crash.</p>
<p>here is the last 2 call stack:</p>
<div class="md-table">
<table>
<thead>
<tr>
<th>&gt;</th>
<th>c10.dll!c10::detail::torchCheckFail(const char * func, const char * file, unsigned int line, const std::string &amp; msg) line 87</th>
<th>C++</th>
</tr>
</thead>
<tbody>
<tr>
<td></td>
<td>c10.dll!c10::detail::torchInternalAssertFail(const char * func, const char * file, unsigned int line, const char * condMsg, const char * userMsg) line 103</td>
<td>C++</td>
</tr>
</tbody>
</table>
</div>
Why does enumerating a dataloader show a different object than accessing the dataset by index?

username: nonathecoda
cooked: <p>I’m trying to use an existing model and adapt it for my own use case and dataset. To do that, I had to change the kind of information that the <strong>get_item</strong> method of the dataset returns (added some keys to a dictionary). For some reason, the new keys don’t get added when I try to loop over the dataloader that is supposed to load the dataset. During the debugging process I noticed that when accessing my dataset by index, it returns me the keys I added. When I access the dataset by using “enumerate”, it shows me the old keys.</p>
<p>Attached a code snippet from my trainer.fit() method with the outputs:</p>
<pre><code class="lang-auto">for batch_idx, batch in enumerate(train_loader):
                ic(type(train_loader))
                ic(type(train_loader.dataset))
                
                ic(train_loader.dataset[0].keys())
                ic(type(train_loader.dataset[0]))
    
                ic(batch.keys())
                ic(type(batch))
</code></pre>
<p>Outputs:</p>
<pre><code class="lang-auto">ic| type(train_loader): &lt;class 'torch.utils.data.dataloader.DataLoader'&gt;
ic| type(train_loader.dataset): &lt;class 'torch.utils.data.dataset.ConcatDataset'&gt;
ic| train_loader.dataset[0].keys(): dict_keys(['src_xyz', 'src_grey', 'tgt_xyz', 'tgt_grey', 'src_overlap', 'tgt_overlap', 'correspondences', 'pose', 'idx', 'overlap_p'])
ic| batch.keys(): dict_keys(['src_xyz', 'tgt_xyz', 'src_overlap', 'tgt_overlap', 'correspondences', 'idx', 'pose', 'overlap_p'])
ic| type(batch): &lt;class 'dict'&gt;
</code></pre>
<p>As you can see, the keys in train_loader.dataset[0] are different than in batch. Two questions arise from this:</p>
<ol>
<li>How is train_loader.dataset[0] not the same as the batch variable?</li>
<li>Why do we enumerate over the dataloader, not over dataloader.dataset?</li>
</ol>
<p>I hope the context is sufficient to understand the question!</p>

username: ptrblck
cooked: <ol>
<li><code>loader.dataset</code> will access the <code>dataset</code> that was passed to the <code>DataLoader</code> in the main process. If you are using multiprocessing via <code>num_workers&gt;0</code> each worker will create its copy of this <code>dataset</code> during its instantiation. You could get information about the worker process via <code>torch.utils.data.get_worker_info()</code> inside the <code>Dataset.__getitem__</code>.</li>
</ol>
<p>2.You don’t need to use a <code>DataLoader</code>, but it allows you to:</p>
<ul>
<li>shuffle the data</li>
<li>use pinned memory</li>
<li>use multiprocessing via <code>num_workers&gt;0</code></li>
<li>batches the data</li>
<li>use custom samplers, etc.</li>
</ul>

username: nonathecoda
cooked: <p>Thank you! It turned out there was a problem for the collate_fn.</p>
How encode_jpeg works?

username: Hoang_Le
cooked: <p>Hi all,</p>
<p>I wonder how exactly <code>encode_jpeg</code> works.<br>
<a href="https://pytorch.org/vision/main/generated/torchvision.io.encode_jpeg.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/vision/main/generated/torchvision.io.encode_jpeg.html</a><br>
I cannot trace the source code of this function.</p>
<p>Anyone knows how this function works internally.<br>
We know that JPEG compression algorithm has many different settings.<br>
But the pytorch documentation has nothing about it.</p>
<p>Thanks.</p>

username: ptrblck
cooked: <p>You can check the <a href="https://github.com/pytorch/vision/blob/08c9938f3e69b4fb2d710a038479264f3c23b133/torchvision/csrc/io/image/cpu/encode_jpeg.cpp#L27">source code</a> to see how the data is encoded via <code>libjpeg</code>.</p>

username: Hoang_Le
cooked: <p>Thank <a class="mention" href="/u/ptrblck">@ptrblck</a> . Look like Pytorch just calls libjpeg and uses default settings. I do not see any specific settings</p>
Hutchinson's trick and batch grad_outputs for autograd.grad

username: vmoens
cooked: <p>Hello<br>
I am trying to use Hutchinson’'s trick to compute an estimator of the trace of a hessian matrix.<br>
To make this estimator precise, I need to run the same backward pass several times with different stochastic vectors v that are used for vector-jacobian product.</p>
<p>The code using a for loop would look like this:</p>
<pre><code class="lang-auto">def hutch_loop(loss, params, niter, v):
    g = autograd.grad(loss, params, create_graph=True)
    out = 0.0
    for i in range(niter):
        vsplit = [torch.randn_like(p) for p in params]
        Hv = autograd.grad(g, params, vsplit, retain_graph=True)
        out += group_product(Hv, vsplit)
    return out/niter

def group_product(xs, ys):
    return sum([torch.sum(x * y) for (x, y) in zip(xs, ys)])

</code></pre>
<p>Ideally, I would like to be able to do this operation in batch mode, alleviating the need to compute as many backward passes than the number of samples i need to draw. Is there any way to do this efficiently?</p>
<p>Thanks!</p>

username: albanD
cooked: <p>Hi,</p>
<p>I’m afraid that right now this is the best that you can do.<br>
But we are actively working on changing that. If all goes according to plan, we should have a batched version for the fall of this year.</p>

username: xieyi4650
cooked: <p>Hi <a class="mention" href="/u/vmoens">@vmoens</a></p>
<pre><code class="lang-auto">Hv = autograd.grad(g, params, vsplit, retain_graph=True)
</code></pre>
<p>I remember the first parameter must be scalar, but you set it is <code>g</code><br>
wiil this trigger error ?</p>

username: vmoens
cooked: <aside class="quote no-group" data-username="xieyi4650" data-post="3" data-topic="87137">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/xieyi4650/48/60960_2.png" class="avatar"> xieyi4650:</div>
<blockquote>
<p><code>vsplit</code></p>
</blockquote>
</aside>
<p>not if you provide a tensor to do the reduction IIRC</p>
Latent distribution in transformer

username: Debodeep_Banerjee
cooked: <p>I am using a caption-generating transformer. My goal is to train a regression model on the top of the transformer. For this reason I need to have access to the latent representation of the predicted text. I am trying to extract the latent(/ vector representation) of the text while training the transformer. I will store the vector representation of the text and eventually train the regression model. But I am not able to figure out how to extract the vector representation of the final sequence. Can you please help me with that?</p>

username: J_Johnson
cooked: <p>You can either train a <code>nn.Embedding</code> layer or download a pretrained embedding layer.</p>
<p>Training a layer can be done either by this tutorial:</p>
<p><a href="https://pytorch.org/tutorials/beginner/transformer_tutorial.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/beginner/transformer_tutorial.html</a></p>
<p>or with Huggingface:</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://huggingface.co/blog/getting-started-with-embeddings">
  <header class="source">

      <a href="https://huggingface.co/blog/getting-started-with-embeddings" target="_blank" rel="noopener nofollow ugc">huggingface.co</a>
  </header>

  <article class="onebox-body">
    <img src="https://huggingface.co/blog/assets/80_getting_started_with_embeddings/thumbnail.png" class="thumbnail" width="" height="">

<h3><a href="https://huggingface.co/blog/getting-started-with-embeddings" target="_blank" rel="noopener nofollow ugc">Getting Started With Embeddings</a></h3>

  <p>We’re on a journey to advance and democratize artificial intelligence through open source and open science.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
Managing large datasets

username: as2475
cooked: <p>I have ~0.5 million files from which I am extracting features as torch.tensors with dimensions of approximately 100x100x100. Prrecomputing features for all files doesn’t seem like an option because I don’t have enough storage space. On the other hand, it takes ~10s to calculate the features per file so I’m worried this will be a bottleneck in training if I don’t precompute them.</p>
<p>I would be grateful if anyone would be willing to suggest what would be the standard/best practice in this case or point me to a relevant source?</p>
<p>thank you!</p>

username: yiftach
cooked: <p>Hi <a class="mention" href="/u/as2475">@as2475</a>, just to make sure I understand the situation - your initial files are smaller than the features you later compute? (your files are saved on the disk, but you don’t have enough storage for the features?)<br>
Eventually any continuously collected dataset becomes too large to be stored on a single machine, and it may then be the time to switch to a streaming based solution. But If you can afford a storage large enough to store them all that is an easier option.</p>
Modifying what model returns after training

username: Dibaloke_Chanda
cooked: <p>I am working with a multi-head architecture. The issue is when I complete training and am done with inference, I need to pass the model to an API to do another task. Now the API only works with a single prediction (single-head).  I can easily chop off the additional heads of the model and keep one specific head and pass it to the API. But the issue is the model definition in the forward() method I was returning multiple predictions generated by multiple heads. For the API to work I need to modify what I was returning and only return a single thing. Is there a way to do that ? I mean modifying what the model was returning from the trained model .pth file?</p>

username: dhruvbird
cooked: <p>You could wrap <code>forward()</code> in another method that calls <code>forward()</code> under the hood and returns only what you need. Alternatively load the model weights into another class that has the same structure but has the modifications applied in the forward() method itself, and you should be good to go!</p>

username: Dibaloke_Chanda
cooked: <p>Thank you. I understand what you mean. I can define another model with a modified forward method and name it “model2” and load the weights from “model” to “model2”. Can you direct me to any resource for how I do that (the transfer weight to another model part)?</p>

username: dhruvbird
cooked: <p>You don’t need to “transfer” weights explicitly. You can just load the weights from the saved file into the new model directly. Something like this:</p>
<pre><code class="lang-auto">class Model1(nn.Module):
    def __init__(self):
        super().__init__()
        self.lin = nn.Linear(10, 20)
        
    def forward(self, x):
        x = x + 10.
        return self.lin(x)
    
class Model2(nn.Module):
    def __init__(self):
        super().__init__()
        self.lin = nn.Linear(10, 20)
        
    def forward(self, x):
        x = x * 10.
        return self.lin(x)

m1 = Model1()
torch.save(m1.state_dict(), "test_model1.pt")

m2 = Model2()
m1.load_state_dict(torch.load("test_model1.pt"))
</code></pre>
Torchrun runs perfectly with 1 gpu node but fails on 4 giving a "[Errno 12] Cannot allocate memory" error

username: Rameswar_Sahu
cooked: <p>Hi,<br>
I have a CNN that I want to run in parallel across 4 GPU nodes. I tried running it with 1 GPU, and the program works fine. Here is the code I used:</p>
<p>OMP_NUM_THREADS=1 torchrun --nproc_per_node=1 model.py --exp_name=try1 --datadir=‘./data/try1’ --logdir=‘./logs/’ --num_block 3 3 3 --hidden 16 32 64 --num_workers=4 --lr=0.001 --batch_size=16<br>
Next, I tried running it with 4 GPU nodes by using the following:</p>
<p>OMP_NUM_THREADS=6 torchrun --nproc_per_node=4 model.py --exp_name=try1 --datadir=‘./data/try1’ --logdir=‘./logs/’ --num_block 3 3 3 --hidden 16 32 64 --num_workers=24 --lr=0.001 --batch_size=16</p>
<p>But this time, I got the following error :<br>
OSError: : [Errno 12] Cannot allocate memory</p>
<p>I don’t understand why I am getting this error and how to fix it. Any suggestion is highly appreciated.</p>
<p>Best,<br>
Ram</p>
How to obtain cell states from a bidirectional LSTM in pytorch?

username: Aaditya_Singh
cooked: <p>Hello, I would like to know how can we extract the cell states at each time step from a single-layer bidirectional LSTM. From the tutorials and discussions, this snippet works for unidirectional LSTM, but I don’t know how to convert this for the bidirectional one.</p>
<pre><code class="lang-auto">lstm = nn.LSTM(3, 3)  # Input dim is 3, output dim is 3
inputs = [torch.randn(1, 3) for _ in range(5)]  # make a sequence of length 5
# initialize the hidden state.
hidden = (torch.randn(1, 1, 3),
          torch.randn(1, 1, 3))
for i in inputs:
    # Step through the sequence one element at a time.
    # after each step, hidden contains the hidden state.
    out, hidden = lstm(i.view(1, 1, -1), hidden)
</code></pre>

username: ptrblck
cooked: <p><a href="https://discuss.pytorch.org/t/clarification-regarding-the-return-of-nn-gru/47363/2">This post</a> might be helpful, which visualizes the internal tensors, and <a href="https://discuss.pytorch.org/t/clarification-regarding-the-return-of-nn-gru/47363/4">this follow-up</a> for the bidirectional case. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

username: dhruvbird
cooked: <p>There are 2 main concepts with LSTMs:</p>
<ol>
<li><code>output</code>: PyTorch returns the final output corresponding to each time step (sequence length) in both directions. The output is returned only for the last layer in a multi-layer LSTM.</li>
<li><code>hidden state</code>: PyTorch returns only the final hidden state after the last time step (after the last element in the sequence) is processed. However, this final hidden state for each layer is returned in case of a multi-layer LSTM.</li>
</ol>
<p>To obtain the hidden state after the last layer in both directions use <code>hidden[-2]</code> and <code>hidden[-1]</code> respectively for the forward and reverse directions.</p>
How to select arbitrary rows from a tensor?

username: jagandecapri
cooked: <p>I have a 3D tensor as following</p>
<pre><code class="lang-auto">a = torch.tensor([
         [
          [ 1,  2,  3],
          [ 4,  5,  6]
         ],
         [
          [ 7,  8,  9],
          [10, 11, 12]
         ]
])
</code></pre>
<p>I would like to select the first row of the first matrix and the second row of the second matrix within the 3D tensor to give the following output:</p>
<pre><code class="lang-auto">[
    [1,2,3],
    [10,11,12],
]
</code></pre>
<p>I’m just thinking if there is a way in PyTorch without using for-loops? I tried looking at <code>torch.gather</code> and <code>torch.index_select</code> but it doesn’t seem to work with my use case.</p>

username: ptrblck
cooked: <p>Direct indexing will work:</p>
<pre><code class="lang-python">a[[0, 1], [0, 1]]
# tensor([[ 1,  2,  3],
#         [10, 11, 12]])
</code></pre>

username: jagandecapri
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thank you very much. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
Error on enumerate(dataloader)

username: chiheb48
cooked: <p>Hello, this my code yet when I run it I have this error appear.<br>
def train_step(model: torch.nn.Module,<br>
dataloader: torch.utils.data.DataLoader,<br>
loss_fn: torch.nn.Module,<br>
optimizer: torch.optim.Optimizer):<br>
# Put model in train mode<br>
model = model.to(device)<br>
model.train()</p>
<pre><code># Setup train loss and train accuracy values
train_loss= 0

# Loop through data loader data batches
for i, data in enumerate(dataloader):
    
    
    img, patches, _ = data

    patches = patches.cuda()

    # 1. Forward pass
    pat_pred = model(patches)

    # 2. Calculate  and accumulate loss
    loss = loss_fn(pat_pred, patches)
    train_loss += loss.item() 

    # 3. Optimizer zero grad
    optimizer.zero_grad()

    # 4. Loss backward
    loss.backward()

    # 5. Optimizer step
    optimizer.step()

# Adjust metrics to get average loss  per batch 
train_loss = train_loss / len(dataloader)
return train_loss
</code></pre>
<p>the Error:<br>
img, patches, _ = data<br>
18     patches = patches.cuda()<br>
20     # 1. Forward pass</p>
<p>ValueError: too many values to unpack (expected 3)</p>

username: ptrblck
cooked: <p>Based on the error message and the stacktrace it seems <code>data</code> does not contain 3 values so check its <code>len</code> and make sure the unwraping is using the expected number of internally stored objects.</p>

username: chiheb48
cooked: <p>Thank you for your  response, it did help me overcome the error.</p>
GPU out of memory with torch.nn.Bilinear

username: Todaime
cooked: <p>Hello,</p>
<p>I am currently trying to train a model which includes among other modules the torch.nn.Bilinear one :</p>
<pre><code class="lang-auto">self.batch_size = 1
input_len = 100
output_len = 2
self.bilinear = nn.Bilinear(
            input_len,
            input_len,
            output_len,
            bias=False,
        )
 self.left_indices = [
            index
            for index in range(config["max_seq_len"])
            for _ in range(config["max_seq_len"] - index)
        ]
self.right_indices = [
            higher_index
            for index in range(config["max_seq_len"])
            for higher_index in range(index, config["max_seq_len"])
        ]
</code></pre>
<p>Which I use this way during training :</p>
<pre><code class="lang-auto">...
output_tensor = torch.zeros(
            self.batch_size,
            self.max_seq_len,
            self.max_seq_len,
            self.output_len,
            device=used_device,
        )
output_tensor[
                :, self.left_indices, self.right_indices
            ] = self.bilinear(
            input_left[:, self.left_indices],
            input_right[:, self.right_indices],
        )
....
</code></pre>
<p>However I hit the traditional “CUDA out of memory.” (which seems to happen when processing a second batch)</p>
<p>I tried varying the size of the two input tensors (from 1 to 100) and the resource allocated to the GPU takes up to 8GB with a Batch size of 1.</p>
<p>Is this normal or am I doing something wrong ? (assigning the module output to a tensor like that ?)</p>
<p>Thank you for your help ! =)</p>

username: yiftach
cooked: <p>Hi, your answer has multiple aspects, I’ll try to answer all.</p>
<ol>
<li>Nothing here seems way off, but it’s hard to tell if this should result in CUDA OOM without the real numbers (e.g. the config params). Could you share a reproducible example?</li>
<li>The fact that it happens in the second batch (assuming you mean a second call to <code>self.bilinear</code> and not <code>batch_size=2</code> indicates some memory used in the first call is not getting freed. This might be unintentional, but again a full code would help.</li>
<li>Assigning the module output to a tensor is fine</li>
<li>You could replace those nested list comprehensions with PyTorch functions (e.g. see <code>torch.arange</code>, <code>repeat</code>).</li>
</ol>
How to split backpropegation for two parameters given one image

username: AviranRSX
cooked: <p>Hi,</p>
<p>I am trying to estimate two parameters, such as the length and angle of an object, from a given image using an EfficientNet. To achieve this, I split the output of the EfficientNet, which has 1280 classes, into two dense layers with 320 labels each. One dense layer is for the angle and the other for the length.</p>
<p>During training, I apply the Cross-Entropy (CE) loss separately to each parameter, sum them, and then divide by 2. After that, I perform backpropagation.</p>
<p>The training process is working fine, but I am wondering if there is a smarter way to train each parameter while maintaining this concept. Is it possible to perform backpropagation on each dense layer separately with its own loss and then combine them using the chain rule to backpropagate through the EfficientNet block?</p>
<p>Here is how the two dense networks are connected to the output of the EfficientNet:</p>
<pre><code class="lang-auto">class Model(nn.Module):
    def __init__():
        # bla bla
        eff_model.classifier = nn.Sequential(
            nn.Linear(in_features=1280, out_features=self.eff_b1_out_features),
            nn.LeakyReLU()
        )
        self.model = nn.Sequential(eff_model, net_add)  # add_net is a model containing two dense layers with a 1280-vector input

    def forward(self, matrix):
        len, angle = self.model(matrix)
        return len, angle

</code></pre>
<p>The training procedure follows these steps:</p>
<pre><code class="lang-auto">len_out, angle_out = self.model(matrix)
len_loss = criterion(len_out, len_label)
angle_loss = criterion(angle_out, angle_label)
total_loss = (len_loss + angle_loss) / 2
total_loss.backward()
</code></pre>
<p>Any suggestions on how to improve the training process would be greatly appreciated.<br>
Thank you!</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="AviranRSX" data-post="1" data-topic="183731">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/aviranrsx/48/62001_2.png" class="avatar"> AviranRSX:</div>
<blockquote>
<p>To achieve this, I split the output of the EfficientNet, which has 1280 classes, into two dense layers with 320 labels each. One dense layer is for the angle and the other for the length.</p>
<p>During training, I apply the Cross-Entropy (CE) loss separately to each parameter, sum them, and then divide by 2. After that, I perform backpropagation.</p>
</blockquote>
</aside>
<p>This sounds as if you are treating a regression use case as a multi-class classification where e.g. each possible angle corresponds to a class label.<br>
If so, I would expect your model could overfit to a few angles as I would also expect to see an imbalance in the dataset or did you guarantee each angle has the same frequency in the data?<br>
Did you try to output a single value for the angle and length each and apply e.g. <code>nn.MSELoss</code> on it?</p>
Cuda memory leak even when using cpu tensors

username: 0piero
cooked: <p>I have identified one part of my code that is causing memory leak in cuda:</p>
<pre><code class="lang-auto">for j in range(0, len(indices_unlbl)):

            inp_rep = inp[j].repeat(inputs.shape[0], 1)
            vec = inputs - inp_rep
            inp_rep = inp_rep.cpu()
            del inp_rep

            inf_entries = (torch.norm(input=vec, p=float('inf'), dim=1) &gt; 0.1)
            pvtmtx_samples[i1: i1 + blk_size, j][inf_entries] = float('inf')
            
            non_inf_entries = (pvtmtx_samples[i1: i1 + blk_size, j] &lt; float('inf'))
            norms_2 = torch.norm(input=vec[non_inf_entries], p=2, dim=1, dtype=float)
            norms_2 = norms_2.cpu()

            pvtmtx_samples[i1: i1 + blk_size, j][non_inf_entries] = norms_2

            inf_entries = inf_entries.cpu()
            non_inf_entries = non_inf_entries.cpu()
            vec = vec.cpu()

            del norms_2
            del inf_entries
            del non_inf_entries
            del vec

</code></pre>
<p>The line <code>pvtmtx_samples[i1: i1 + blk_size, j][non_inf_entries] = norms_2</code> is causing cuda memory increase in every iteration, if I remove this line the problem vanishes. what I don’t understand is why this is happening even if “pvtmtx_samples” and “norms_2” are both on cpu.</p>

username: ptrblck
cooked: <p>Could you post a minimal and executable code snippet reproducing the CUDA memory increase if CPUTensors are stored, please?</p>
Filling a 2D tensor at specific indices without detaching

username: robclouth
cooked: <p>Here’s my problem: I’m trying to fit a series of splines to a pixel image of wavy lines that go from left to right. I’ve found an autograd-friendly implementation of splines, then my plan was to reconstruct an image from the splines and pass the generated output and a target to the loss function.</p>
<p>For the optimization to work, the whole chain needs to be differentiable. However I can’t find a way of reconstructing the images from the splines in a way that doesn’t detach the chain.</p>
<p>I could take the spline and turn the values into x,y coords to set certain elements of a tensor, but when I int() them it detaches of course.</p>
<p>I found a grid_sample, which is a differentiable version of the exact opposite of what I need to do. I need to fill certain points in a tensor using float indices and interpolation.</p>
<p>Basically I need to “paint” into a tensor in an autograd-friendly way.</p>
<p>I’m a bit lost, any hints would be super useful.</p>

username: robclouth
cooked: <p>Ah, I figured it out.<br>
Basically you can floor the float index, then interpolate the int indices using the float one. That’s autograd friendly.</p>
<aside class="quote quote-modified" data-post="4" data-topic="2111">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/colesbury/48/26_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/indexing-a-variable-with-a-variable/2111/4">Indexing a variable with a variable</a> 
  </div>
  <blockquote>
    Your tensorflow link is to a private repository so I can’t view it. 
The index operation doesn’t have any gradients defined w.r.t. the index variable. That’s not a limitation of “define by run”, that’s a property of the operation: it has integer domain. 
You need a differentiable sampling operation for spatial transformer networks. You can implement STN in PyTorch in roughly the same ways as in Tensorflow: sample<a href="https://github.com/tensorflow/models/blob/7c42a178f67676467ca6e71d0628954a22396271/transformer/spatial_transformer.py#L84-L87" rel="noopener nofollow ugc"> floor(idx) and floor(idx) + 1</a> and linearly interpolate between the two. Note that t…
  </blockquote>
</aside>


username: minasmz
cooked: <p>Hello <a class="mention" href="/u/robclouth">@robclouth</a>,</p>
<p>I want to do similar work as you mentioned. I want to learn some parameters to paint an image in some positions (using some indexes) to maximize the classification performance of the images. I tried to understand the floor(idx) and interpolation. However, I could not figure out how I to access the positions in the image for changing the RGB channel of the image.<br>
In the <a href="https://discuss.pytorch.org/t/indexing-a-variable-with-a-variable/2111/4">mentioned</a> code. I could not relate the output tensor to the image I want to change the color values.<br>
I appreciate it if you help me!</p>
1d convolution with several identical input channels produced by heavy network

username: Daniele_Juda
cooked: <p>Hi all,</p>
<p>we have a network with this kind of behaviour:</p>
<pre><code class="lang-auto">embedding=heavy_shared_net(big_image_embedding)
for all node in nodes : 
    output[node]=output_net(cat(embedding,position(node)))
return output    
</code></pre>
<p>The number of nodes can reach thousands<br>
heavy_shared_net weights must be updated according to the error on all the output nodes</p>
<p>This solution should work in that regard but it would be nice to parallelize the loop that looks like a conv1d with kernel size 1.</p>
<p>if we use  the repeat method on the embedding will the backdrop still work on the  heavy_shared_net?</p>
<p>Any suggestions?</p>

username: ptrblck
cooked: <p>Yes, <code>repeat</code> is a differentiable operation but will of course increase the memory usage. Based on your code this should already be the case so you could try using it.</p>
FloatingPointError: Predicted boxes or scores contain Inf/NaN. Training has diverged

username: RadAlienware
cooked: <p>So, I am running detectron 2 mask rcnn for a specific dataset. So for transfer learning I have added another pretrained weight which is trained on a different but same type of  dataset. But everytime I am adding the pretrained weight, i am getting the aforementioned error. I have also added pretrained weight of Segment Anything Model (which is not similar to my dataset) I got the error. Then I am trying to use a model weight which is trained on kind of similar type of my dataset. But still  getting the same error.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/8/a872f0b17f87256399c3b6eb78fc0585ec37b66f.png" data-download-href="https://discuss.pytorch.org/uploads/default/a872f0b17f87256399c3b6eb78fc0585ec37b66f" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/8/a872f0b17f87256399c3b6eb78fc0585ec37b66f.png" alt="image" data-base62-sha1="o2atllK033wqPIYLdywhJ4v0L27" width="690" height="114" data-dominant-color="F8D9D9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1246×206 16 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><em>I have adjusted learning rate as well as batch size, still not getting any solution.</em></p>
<p>Thanks for helping me.</p>