akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Can we detach the tensor to manipulate and again cast it to tensor with requires grad = True and backpropagate on it?

username: mahesh_bhosale
cooked: <p>I am working on a code that does some averaging on the detached NumPy array of probabilities of the PyTorch tensor, can I recreate a tensor from the average NumPy array with required_grad=True and backpropagate?</p>

username: AlphaBetaGamma96
cooked: <p>I’m pretty sure once the graph has been freed it’s gone. Try and see if you can find the equivalent operations in PyTorch or you could define a custom <code>torch.autograd.Function</code> object which would allow you to backpropagate through it.</p>
<p>Could you share a minimal reproducible example?</p>

username: mahesh_bhosale
cooked: <p>Thanks for the reply.<br>
Sure, below is what operations it is doing,<br>
compute_batch_predictions() detached the tensor to the NumPy array, which is used to create a data frame that is doing averaging.</p>
<pre><code class="lang-auto">def train()
{
...
    batch_predictions = compute_batch_predictions(output, mode=parameters["model_mode"])
    pred_df = pd.DataFrame({k: v[:, 1] for k, v in batch_predictions.items()})
    pred_df.columns.names = ["label", "view_angle"]
    predictions = pred_df.T.reset_index().groupby("label").mean().T[LABELS.LIST].values
    l = loss(predictions, gt)
    optimizer.zero_grad()
    l.backward()
    optimizer.step()
...
}
</code></pre>

username: ptrblck
cooked: <p>As <a class="mention" href="/u/alphabetagamma96">@AlphaBetaGamma96</a> explained, once you’ve detached the computation graph you won’t be able to backpropagate through it anymore. Based on your code it seems you are trying to calculate the mean of some stats, so try to use PyTorch methods instead.</p>
How to fine tune BERT to get word embeddings on your own custom Corpus?

username: IrfanUmar
cooked: <p>I can have pretrained word embeddings using this, <a href="https://colab.research.google.com/drive/1RhmL0BqNe52FEbdSyLpkfVuCZxE7b5ke" rel="noopener nofollow ugc">https://colab.research.google.com/drive/1RhmL0BqNe52FEbdSyLpkfVuCZxE7b5ke</a></p>
<p>But Don’t know how to fine tune it on my custom dataset</p>

username: Abhilash_Srivastava
cooked: <p>What exactly do you want to do?<br>
Fine tune the model or fine tune the embeddings?</p>

username: IrfanUmar
cooked: <p>Fine tune the embeddingd</p>

username: Abhilash_Srivastava
cooked: <p>If you train the model E2E (not just fine-tune the task layer), it would modify the pre-trained parameters of all the layers (including the embedding layer).<br>
However, remember the BERT embeddings are different from the word2vec embeddings and they depend on the context. Hence, they cannot be used as it is for a different task (unlike word2vec embeddings which don’t have context).<br>
Until absolutely necessary to fine-tune the embeddings, you can fine-tune task layers (over BERT pretrained) model and adapt it to your specific problem set.</p>

username: Edgar_Platas
cooked: <p>Hi Irfan</p>
<p>were you able to solve the problem? I am in the same situation but I am working with the BioBERT model with a set of scientific articles of my own.</p>
<p>Greetings</p>
What is good Yolo metric scores

username: Tornike
cooked: <p>Hi, I’m doing object detection with yolov5 on a custom dataset. This is my first time training Yolo.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/8/6/863128cd93e6ecb45005746160626a818fb02967.png" alt="image" data-base62-sha1="j97iSEB29DCZhc5suhSO7noUGSb" width="600" height="50"></p>
<p>These are the results. what is a good result for these metrics?</p>
Can I build libtorch from source with mingw?

username: mesut92
cooked: <p>I need to know any chance to build torch with mingw? If it is possible, How? Did anyone build it?</p>
<p>OS:Windows 10</p>

username: peterjc123
cooked: <p>If you want to build the CPU version, it is possible but you will need to make some changes to the code. The answer is absolutely no if you want the CUDA version. Because nvcc doesn’t work with it under Windows.</p>

username: gps
cooked: <p>Hi <a class="mention" href="/u/peterjc123">@peterjc123</a><br>
I know this is old… but can you elaborate on the changes to make the package compile for CPU with mingw?<br>
is there a repo that can be cloned?</p>
<p>Thanks!</p>

username: peterjc123
cooked: <p>No, there isn’t a working build. You’ll need to switch the CMake generator to MinGW (currently MSVC) and adapt the code to the toolchain line by line. I’m afraid that’s a lot of work for you.</p>
Combine two datasets of **different** dimensions

username: daikkido
cooked: <p>I am working with some neural network models which I want to combine. My idea is to train the network with multiple datasets on <strong>multiple different losses</strong> simultaneously. For example, a batch consists of 8 samples of dataset 1, 8 samples of dataset 2, and 16 samples of dataset 3. I wanted to concatenate the dataset and just write a custom dataloader to ensure this kind of sampling. By doing this, the structure of all the code need not be changed which is a plus. The problem now is that the datasets are of different formats. For example, X1 = (21464, 500, 129), y1 = (21464, 3) and X2 = (30842, 500, 129), y2 = (30842, 2). In this example the labels are the problem. Are there different ways to approach this problem or does concatenating still somehow work?</p>
<p>Thanks for your help!</p>
Compile libtorch with g++

username: phithangcung
cooked: <p>Hi all,</p>
<p>I was trying to add the libtorch library to my C++ project which is being built by using g++. I tried to include the specified files in the g++ command but it does not seem to work. I follow the <code>example-app</code> from <a href="https://pytorch.org/cppdocs/installing.html#minimal-example" rel="noopener nofollow ugc">https://pytorch.org/cppdocs/installing.html#minimal-example</a></p>
<p>My command that was used to compile the c++ source code is:<br>
(I added <code>/path/to/libtorch/lib/</code> to <code>LD_LIBRARY_PATH</code> in <code>~/.bashrc</code>)</p>
<pre><code class="lang-auto">$ g++ -std=gnu++11 -o test *.cpp \
-I /path/to/libtorch/include/torch/csrc/api/include/ \
-I /path/to/libtorch/include/ \
-L /path/to/libtorch/lib/ \
-D_GLIBCXX_USE_CXX11_ABI=0 \
-lc10 \
-lCaffe2_perfkernels_avx512 -lCaffe2_perfkernels_avx2 -lCaffe2_perfkernels_avx \
-lnnpack -lpytorch_qnnpack -lqnnpack -lXNNPACK \
-lcaffe2_protos -lprotobuf-lite -lprotobuf -lprotoc \
-lonnx -lonnx_proto -lfoxi_loader -lfmt -lclog -lcpuinfo \
-lpthreadpool -lfbgemm -ldnnl -lmkldnn -lasmjit
</code></pre>
<p>I got the following error message:</p>
<pre><code class="lang-auto">/tmp/cchwVWtY.o: In function `at::operator&lt;&lt;(std::ostream&amp;, at::Tensor const&amp;)':
main.cpp:(.text+0x13c): undefined reference to `at::print(std::ostream&amp;, at::Tensor const&amp;, long)'
/tmp/cchwVWtY.o: In function `torch::autograd::AutogradMeta::AutogradMeta(c10::TensorImpl*, bool, torch::autograd::Edge)':
main.cpp:(.text._ZN5torch8autograd12AutogradMetaC2EPN3c1010TensorImplEbNS0_4EdgeE[_ZN5torch8autograd12AutogradMetaC5EPN3c1010TensorImplEbNS0_4EdgeE]+0x38): undefined reference to `vtable for torch::autograd::AutogradMeta'
/tmp/cchwVWtY.o: In function `torch::rand(c10::ArrayRef&lt;long&gt;, c10::TensorOptions)':
main.cpp:(.text._ZN5torch4randEN3c108ArrayRefIlEENS0_13TensorOptionsE[_ZN5torch4randEN3c108ArrayRefIlEENS0_13TensorOptionsE]+0x96): undefined reference to `at::rand(c10::ArrayRef&lt;long&gt;, c10::TensorOptions)'
collect2: error: ld returned 1 exit status
</code></pre>
<p>I also tried to compile with <code>-std=gnu++11</code> but it does not work.</p>
<p>I also tried the suggested solution that was mentioned by [pim] &amp; [thekchang] at <a href="https://discuss.pytorch.org/t/solved-build-the-c-frontend-using-g-or-any-other-compiler-no-cmake/35156">this thread</a></p>
<p>The compilation of the <code>example-app</code> with Cmake as describe worked great. However, it will be more convenient and save more time to compile and debug my project with g++ rather than Cmake. So my question “How can we use libtorch with the compilation with a normal compiler (e.g., g++) on terminal”</p>
<p>Thank you for your time and your effort. Any help would be greatly appreciated !!</p>

username: 111149
cooked: <p>Hi, I also faced similar build error info. I built the libtorch from source with MKL, MKLDNN(oneDNN), MAGMA and CUDA. The generated libs file include libtorch.so, libtorch_cpu.so. You need link them: “-ltorch -ltorch_cpu -lc10 …”</p>
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [3, 3, 3, 3]] is at version 3; expected version 2 instead

username: promach
cooked: <p>Why <strong>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation</strong> ?</p>
<p>The strange thing is that if I set <a href="https://github.com/buttercutter/gdas/blob/d56d5112b6783e38c3c4f824aad50c6dc0131756/gdas.py#L17" rel="noopener nofollow ugc">DEBUG = 1</a> , then the inplace runtime error is gone, why ??</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/2/f2e3cc249745f88a6becc0f7b05741858edc2913.png" data-download-href="https://discuss.pytorch.org/uploads/default/f2e3cc249745f88a6becc0f7b05741858edc2913" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/2/f2e3cc249745f88a6becc0f7b05741858edc2913_2_690x370.png" alt="image" data-base62-sha1="yEHBixODaOk3lkCQCMea16JBZgn" width="690" height="370" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/2/f2e3cc249745f88a6becc0f7b05741858edc2913_2_690x370.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/2/f2e3cc249745f88a6becc0f7b05741858edc2913_2_1035x555.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/f/2/f2e3cc249745f88a6becc0f7b05741858edc2913_2_1380x740.png 2x" data-dominant-color="373739"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1918×1031 164 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Loading a csv with a column of strings and a column of integers

username: Azzam
cooked: <p>Hello! I’m an amateur regarding pytorch. I have a csv file with a column of text and a column of integer labels. How can I load this csv into the dataloader so that I can train a model for classification?</p>
<p>This is my code which gives an exception.</p>
<pre><code class="lang-auto">import torch
import pandas as pd

train = pd.read_csv("/content/Q_V_1.08.csv")
train_tensor = torch.tensor(train.values)
</code></pre>
<pre><code class="lang-auto">TypeError                                 Traceback (most recent call last)
&lt;ipython-input-44-83fe164cf357&gt; in &lt;module&gt;()
      3 
      4 train = pd.read_csv("/content/Q_V_1.08.csv")
----&gt; 5 train_tensor = torch.tensor(train.values)

TypeError: can't convert np.ndarray of type numpy.object_. The only supported types are: float64, float32, float16, complex64, complex128, int64, int32, int16, int8, uint8, and bool.
</code></pre>

username: Andrei_Cristea
cooked: <p>Hi! The most direct way would probably be to <a href="https://pytorch.org/tutorials/beginner/basics/data_tutorial.html#creating-a-custom-dataset-for-your-files" rel="noopener nofollow ugc">create a custom Dataset</a> for your files. It’s quite straightforward, you just inherit the generic class and define the <code>__len__</code> (in your case just <code>return len(train)</code>) and <code>__getitem__</code> methods, as described in the tutorial. Good luck!</p>

username: Azzam
cooked: <p>In the tutorial, the getitem method converts image path to a tensor using <code>read_image</code>. The text in my csv file is not an image path. How would I convert it to a tensor?</p>

username: Azzam
cooked: <p>Also, after creating the custom dataset, how would I pass it into the dataloader and split the dataset into training and test sets?</p>

username: Andrei_Cristea
cooked: <p>Regarding <code>__getitem__</code>, you can customize it to return whatever it is you want to use in your training loop. For example, in your case you may try something like this:</p>
<pre><code class="lang-auto">class CustomDataset(Dataset):
    def __init__(self, csv_file):
        self.data = pd.read_csv(csv_file, header=None)
                                
    def __len__(self):
        return len(self.data)

    def __getitem__(self, idx):
        row = self.data.iloc[idx]
        sample = {'text': row[0], 'number': row[1]}
        return sample

dataset = CustomDataset("/content/Q_V_1.08.csv")
for foo in dataset:
    print(foo["text"], foo["number"])

Output:
alpha 100
bravo 200
charlie 300
delta 400
</code></pre>
<p>Regarding split and test, you might do <a href="https://discuss.pytorch.org/t/how-to-split-dataset-into-test-and-validation-sets/33987">this</a> which I found by Googling “pytorch train test split”.</p>
What is the reason for some nan value when training

username: dachun
cooked: <p><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/3/13753d2a42eed8c1c902034de51097abbf1cf5e6.png" alt="image" data-base62-sha1="2M8fOUmHZxm1DlmgBjjiSzX174a" width="540" height="372"><br>
As figure shows, I get some nan loss value on triangle position, what is the reason?</p>

username: Ehsan1997
cooked: <p>There could be many reasons, unnormalized data is one of them.</p>
<p>Please note that without seeing the code, it becomes very hard to answer question like these.</p>

username: Sairam954
cooked: <p>As mentioned by <a class="mention" href="/u/ehsan1997">@Ehsan1997</a>, it is difficult to identify without looking at code. However, here is a similar thread which might be helpful <a href="https://discuss.pytorch.org/t/nan-loss-coming-after-some-time/11568/31">https://discuss.pytorch.org/t/nan-loss-coming-after-some-time/11568/31</a></p>
I want to quantize my trained model 'model.pth'. model size 188mb

username: Muhammad_Raza
cooked: <p>I just trained my model using pytorch which has the size of 188mb, its very bad to run on real time inference , I want to reduce the size of my model  I know it can be done by quantization but I tried &amp; unable to quantize my trained model , there are many example to quantize during training but not available after you train your own model .</p>
<p>can you please give some code to easily quantize my heavy model ?</p>

username: Sairam954
cooked: <p>Here is the link to my colab with code snippets for applying post-training static quantization to a custom model and torchvision pre-trained model.<br>
<a href="https://colab.research.google.com/drive/1f3NuwvuIUF4NuXP_xzpxt-MiqwozOcdN?usp=sharing" rel="noopener nofollow ugc">Post Training Static Quantization on Pre-trained Torchvision Models</a></p>
<p>Hope this helps.</p>
Modifying loader to balance load between different GPUs

username: Felipe_bivort_haiek
cooked: <p>Hi, I am trying to figure out how to modify the DataLoader to permit a different batch size per device. I need to do this as I have different GPUs with different memory and tensor core sizes on the same machine.</p>
<p>I have been looking at the implementation of the DataLoader and it seems the appropriate thing to do would be to use a modified batch_sampler. A LoadBalancedBatchSampler class would inspect the rank of the process (similar to what DistributedSampler does) and then modify the batch_size per rank before the yield loop.<br>
DistributedSampler would also have to be modified so that I can get the same number of iterations in each device.</p>
<p>I am moving in the right direction? Have I missed something?</p>
<p>Thank you in advance.</p>

username: mrshenli
cooked: <p>cc <a class="mention" href="/u/vitalyfedyunin">@VitalyFedyunin</a> for data loader questions</p>
Should local rank be equal to torch.cuda.current_device()?

username: mesllo
cooked: <p>I’m running models using DistributedDataParallel, and I noticed that while the local rank is correctly up to the number of GPUs I’m using, when I print <code>torch.cuda.current_device()</code> I notice that the output is always <code>0</code>. Printing <code>torch.cuda.current_device()</code> correctly outputs <code>4</code>. What does this mean?</p>

username: mrshenli
cooked: <blockquote>
<p>Printing <code>torch.cuda.current_device()</code> correctly outputs <code>4</code> . What does this mean?</p>
</blockquote>
<p>I assume you mean <code>torch.cuda.device_count()</code> outputs <code>4</code>?</p>
<blockquote>
<p>I’m running models using DistributedDataParallel, and I noticed that while the local rank is correctly up to the number of GPUs I’m using, when I print <code>torch.cuda.current_device()</code> I notice that the output is always <code>0</code> .</p>
</blockquote>
<p>The current device is by default <code>cuda:0</code>, unless you explicitly set <code>torch.cuda.set_device(local_rank)</code>. When using DDP, it is recommended to either a) call <code>set_device</code> upfront, or b) setting <code>CUDA_VISIBLE_DEVICES</code> to <code>local_rank</code> before calling any <code>torch.cuda</code> APIs or create CUDA tensors.   If you do a), each process will still see all devices, but by default <code>tensor.cuda()</code> will move it to the specified device. If you do b), each process will only see one physical GPU that corresponds to its <code>local_rank</code>, i.e., <code>cuda:0</code> in different processes will map to a different physical device.</p>
<p>Both solution will work with DDP, as long as you make sure that each DDP instance exclusively operates on a dedicated device.</p>
How to distributed (cross-)correlation?

username: tsly123
cooked: <p>Hello,</p>
<p>I have a batch of tensor size (Batch, Patch, Dim) then after 1D batchnorm the tensor shape is (B, D).<br>
These tensor are in distributed mode (multi GPUs) and I’d like to compute the correlation of each tensor[i] to other tensors in the batch.<br>
The result correlation matrix should have size (B, B), so I will take the max-min value to decide which tensor would correlate the most or the least to which.<br>
Would the code be like this?</p>
<pre><code class="lang-auto"># https://github.com/facebookresearch/moco-v3/blob/main/moco/builder.py#L126
# concat all_gather tensors (B, D) from GPUs
@torch.no_grad()
def concat_all_gather(gpus_tensor):
    """
    Performs all_gather operation on the provided tensors.
    *** Warning ***: torch.distributed.all_gather has no gradient.
    """
    tensors_gather = [torch.ones_like(gpus_tensor)
        for _ in range(torch.distributed.get_world_size())]
    torch.distributed.all_gather(tensors_gather, gpus_tensor, async_op=False)

    output = torch.cat(tensors_gather, dim=0)
    return output

# gather tensor and compute correlation 
tensors = concat_all_gather(gpus_tensor)
corr = tensors @ tensors.T
</code></pre>
<p>Another problem is after compute correlation, I have tensor[i] and its least correlated tensor. These two tensors might in different GPUs, and I need to get the tensor before 1Dbatchnorm (j, Patch, Dim).<br>
One possible solution is to gather tensor before 1Dbatchnorm. Is it correct? and Is there other way to do it?</p>
<p>Thank you.</p>

username: mrshenli
cooked: <p>Hey <a class="mention" href="/u/tsly123">@tsly123</a>, the allgather part looks correct to me, except that you can use <code>torch.empty_like()</code> instead of <code>torch.ones_like()</code>, which should be a little bit faster as it does not need to set all values to 1.</p>
<blockquote>
<p>Another problem is after compute correlation, I have tensor[i] and its least correlated tensor. These two tensors might in different GPUs, and I need to get the tensor before 1Dbatchnorm (j, Patch, Dim).</p>
</blockquote>
<p>After <code>concat_all_gather</code>, I assume <code>tensors</code> already contains values from all processes/device. Is it possible to just reuse that instead of doing another gather?</p>
Libtorch MacOs --no-as-needed not supported

username: mreister
cooked: <p>I am trying to compile a pytorch c++ example based on the instruction here: <a href="https://pytorch.org/cppdocs/installing.html" class="inline-onebox" rel="noopener nofollow ugc">Installing C++ Distributions of PyTorch — PyTorch master documentation</a></p>
<p>I am running into an issue when i run:</p>
<p><code>cmake --build . --config Release</code></p>
<p>I get an error:</p>
<pre><code class="lang-auto">[ 50%] Linking CXX executable block_reader
ld: unknown option: --no-as-needed
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[2]: *** [block_reader] Error 1
make[1]: *** [CMakeFiles/block_reader.dir/all] Error 2
make: *** [all] Error 2
</code></pre>
<p>It looks like the error is that the linker option <code>--no-as-needed</code> is not supported on my os MacOs.</p>
<p>I made sure to download the Max version of libtorch: <a href="https://download.pytorch.org/libtorch/cpu/libtorch-macos-1.11.0.zip" rel="noopener nofollow ugc">https://download.pytorch.org/libtorch/cpu/libtorch-macos-1.11.0.zip</a></p>
<p>I am sure i am not the first person to try this on a Mac…?</p>
<p>I did some searching and it looks like the location in libtorch that support this are:</p>
<pre><code class="lang-auto">include/ATen/detail/CUDAHooksInterface.h
44:  "One common culprit is a lack of -Wl,--no-as-needed in your link arguments; many "

share/cmake/Caffe2/public/utils.cmake
220:# a dynamic library, it will be pruned unless you are --no-as-needed)
267:          "-Wl,--no-as-needed,\"$&lt;TARGET_FILE:${SRC}&gt;\" -Wl,--as-needed")

share/cmake/Caffe2/Caffe2Targets.cmake
76:  INTERFACE_LINK_LIBRARIES "-Wl,--no-as-needed,\"\$&lt;TARGET_FILE:torch_cpu&gt;\" -Wl,--as-needed;\$&lt;TARGET_PROPERTY:torch_cpu,INTERFACE_LINK_LIBRARIES&gt;"
94:  INTERFACE_LINK_LIBRARIES "-Wl,--no-as-needed,\"\$&lt;TARGET_FILE:torch&gt;\" -Wl,--as-needed;\$&lt;TARGET_PROPERTY:torch,INTERFACE_LINK_LIBRARIES&gt;"
</code></pre>
<p>any ideas?</p>
To achieve image regression problem

username: lalalalalyh
cooked: <p>Hi,</p>
<p>I am trying to implement some image regression problem. Basically, I want to use MINST data as the dataset, input the image and the output is the corresponding number, maybe not necessarily the exact number, but some closed one.</p>
<p>In tensorflow, autokeras has the function ImageRegressor, which have the model that can achieve this ( I attached the model below).<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/5/555098704f9a59f9c76e0c8705cab976421f4a05.png" data-download-href="https://discuss.pytorch.org/uploads/default/555098704f9a59f9c76e0c8705cab976421f4a05" title="Screen Shot 2022-04-28 at 10.38.44 AM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/5/555098704f9a59f9c76e0c8705cab976421f4a05_2_690x362.png" alt="Screen Shot 2022-04-28 at 10.38.44 AM" data-base62-sha1="caJg7kO2FrKkpIhbbXceN1k92Rf" width="690" height="362" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/5/555098704f9a59f9c76e0c8705cab976421f4a05_2_690x362.png, https://discuss.pytorch.org/uploads/default/optimized/3X/5/5/555098704f9a59f9c76e0c8705cab976421f4a05_2_1035x543.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/5/5/555098704f9a59f9c76e0c8705cab976421f4a05_2_1380x724.png 2x" data-dominant-color="F2F2F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2022-04-28 at 10.38.44 AM</span><span class="informations">1500×788 51 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>My current idea so far is to “copy” the ResNet152 function to the PyTorch. However, it doesn’t train well. I am wondering whether there are some other alternative ways to achieve this. Thanks!!</p>

username: Ehsan1997
cooked: <p>MNIST images are very small in size (28x28), I don’t think using ResNet152 is a good idea. I would suggest to create your own model from scratch.</p>
Access a tensor entry programatically

username: aliutkus
cooked: <p>hi, I may be missing something, but assume you have a tensor <code>v</code> of shape, say <code>(a,b,c,d)</code>, and another tensor say <code>ind=torch.Tensor([3,2,1,0])</code>.</p>
<p>How can I access <code>v[3,2,1,0]</code> programatically, i.e. by exploiting the variables <code>v</code> and <code>ind</code> ?</p>
<p>edit: I know that <code>v[tuple(ind)]</code> works, but I want this to be jitable, and this solution apparently is not</p>

username: Matias_Vasquez
cooked: <p>You can use this ↓ as your index</p>
<pre><code class="lang-python">ind.long().split(1)
</code></pre>
<pre><code class="lang-python"># Example
a, b, c, d = 10, 20, 30, 40

v = torch.rand(a, b, c, d)
ind=torch.Tensor([3,2,1,0])

print(torch.all(v[ind.long().split(1)] == v[3, 2, 1, 0]))

# Output:
tensor(True)
</code></pre>

username: aliutkus
cooked: <p>awesome. It works, but unfortunately not in a <code>torch.jit.script</code> environment.</p>
<pre><code class="lang-auto">@torch.jit.script
def access(v, index):
  return v[index.split(1)]

v = torch.randn((3, 5, 5))
index = torch.tensor([0,2,2])

print(index)
print(v[0, 2, 2])
print(v[index.split(1)])
print(access(v, index))
</code></pre>
<p>any idea ?</p>

username: Matias_Vasquez
cooked: <p>If you use the <code>@torch.jit.export</code> decorator it works.</p>
<pre data-code-wrap="python"><code class="lang-python">@torch.jit.export
def access(v, index):
  return v[index.split(1)]

v = torch.randn((3, 5, 5))
index = torch.tensor([0,2,2])

print(index)
print(v[0, 2, 2])
print(v[index.split(1)])
print(access(v, index))
</code></pre>
<p><a href="https://pytorch.org/docs/stable/jit.html#torch.jit.export" rel="noopener nofollow ugc">Here</a> is the documentation. But I do not have that much experience with <code>jit</code> so this might not be what you are looking for.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/4/a42647c7dd38ec21bef1c1a41cc4ffb93bef24fb.png" data-download-href="https://discuss.pytorch.org/uploads/default/a42647c7dd38ec21bef1c1a41cc4ffb93bef24fb" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/4/a42647c7dd38ec21bef1c1a41cc4ffb93bef24fb.png" alt="image" data-base62-sha1="nq8jS4uvFIikqQlzam4NLQSL9G3" width="690" height="96" data-dominant-color="F3F3F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">854×119 13.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: aliutkus
cooked: <p>thanks a lot. This will be called from forward unfortunately. I will investigate</p>

username: Matias_Vasquez
cooked: <p>I got it to work a little more with <code>@torch.jit.script</code>.</p>
<pre><code class="lang-python">@torch.jit.script
def access(v, index: torch.Tensor):
  return v.index(index.split(1))
</code></pre>

username: aliutkus
cooked: <p>awesome, thanks it indeed works fine !!!</p>
Uncertainty Estimation

username: Nerolf05
cooked: <p>Hello together,</p>
<p>I am trying to estimate a conservative description of distribution p(y|x) (see image)<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/8/38b1093f0857cdfa0d659a4d9d8d83eb4c17d9c0.png" data-download-href="https://discuss.pytorch.org/uploads/default/38b1093f0857cdfa0d659a4d9d8d83eb4c17d9c0" title="grafik"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/8/38b1093f0857cdfa0d659a4d9d8d83eb4c17d9c0_2_690x370.png" alt="grafik" data-base62-sha1="85w2gdUw6T6I5g20VXsuBA715ew" width="690" height="370" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/8/38b1093f0857cdfa0d659a4d9d8d83eb4c17d9c0_2_690x370.png, https://discuss.pytorch.org/uploads/default/optimized/3X/3/8/38b1093f0857cdfa0d659a4d9d8d83eb4c17d9c0_2_1035x555.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/3/8/38b1093f0857cdfa0d659a4d9d8d83eb4c17d9c0_2_1380x740.png 2x" data-dominant-color="D3E4F0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">grafik</span><span class="informations">3075×1653 298 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>So far I am working with a MixtureDensityNetwork (estimating only 1 gaussian mixture and a negLogLikelihoodGaussian Loss) (Bishop94), resulting in a $\mu(x)$ and $\sigma^2(x)$.<br>
But I would like to include a more conservative penalty term into the cost/loss function, as the underlying distribution is not necessarily a Gaussian one, such that in the end $\mu(x)$ and $\sigma^2(x)$ describe the data.<br>
I came across the “Karush Kuhn Tuckert” conditions (for constraint optimization), so I wanted to ask you if any of you have already worked with it and incorporate it into the Loss function?</p>
<p>Best<br>
Nerolf</p>
Stateful LSTM Pytorch

username: Andre_Amaral_IST
cooked: <p>Hey,</p>
<p>Where should I initialize the hidden state and cell state to have a LSTM stateful?</p>
<p>Regrads<br>
André</p>

username: tom
cooked: <p>One way could be to add a wrapper <code>nn.Module</code> that contains the LSTM as a submodule and calls it with the hidden state.<br>
Do you want the state to be carried over from the previous (forward) call or do you want the initial state to be learned (but the same for consecutive calls)? For the former you would need to avoid autograd desasters by detaching the state (or at least doing so often enough) and probably use a buffer (as in <code>module.register_buffer</code>), for the latter you could use a <code>nn.Parameter</code>.</p>
<p>Best regards</p>
<p>Thomas</p>

username: Andre_Amaral_IST
cooked: <p>What I am doing is just initialize the states in the init method of the model class</p>
Running backward cause memory leak

username: Zhenduo_Wang
cooked: <p>Hi,</p>
<p>I cannot solve this memory leak problem:</p>
<p>My code is:</p>
<blockquote>
<pre><code>def trex_reward_update(self, all_traj_pairs):
    batch_loss = 0
    error = 0
         
    for k in range(self.n_rewards):
        L = T.zeros(1).to(self.device)
        for traj_pair in all_traj_pairs:
            with T.no_grad():
                pos_traj, neg_traj = [(traj[0].tolist(), traj[1]) for traj in traj_pair[0]], [(traj[0].tolist(), traj[1]) for traj in traj_pair[1]]
                traj_sample_length = min(min(len(pos_traj), len(neg_traj)), self.traj_sample_length)
                pos_traj, neg_traj = [T.tensor(traj[0]).to(self.device) for traj in pos_traj if traj not in neg_traj], [T.tensor(traj[0]).to(self.device) for traj in neg_traj if traj not in pos_traj]  
                pos_traj = random.sample(pos_traj, traj_sample_length)
                neg_traj = random.sample(neg_traj, traj_sample_length)

                pos_input_tensor = T.stack(pos_traj).to(self.device)
                neg_input_tensor = T.stack(neg_traj).to(self.device)
    
            pos_output = self.reward[k].forward(pos_input_tensor)
            neg_output = self.reward[k].forward(neg_input_tensor)
            for error_iter in range(len(pos_output)):
                if pos_output[error_iter] &lt; neg_output[error_iter]:
                    error += 1
            L -= T.log(T.exp(pos_output.sum()) / (T.exp(pos_output.sum()) + T.exp(neg_output.sum())))

        self.reward_optimizer[k].zero_grad()
        L.backward(retain_graph = False)
        self.reward_optimizer[k].step()
        self.scheduler[k].step()
        batch_loss += L.detach().item()
        del L

    return batch_loss, error
</code></pre>
</blockquote>
<p>If I don’t run line of <code>L.backward()</code>, then it is fine. Running <code>L.backward()</code> gives memory leak. Can anyone please advise me? What is happening and how to solve this.</p>

username: InnovArul
cooked: <aside class="quote no-group" data-username="Zhenduo_Wang" data-post="1" data-topic="150912">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zhenduo_wang/48/49310_2.png" class="avatar"> Zhenduo_Wang:</div>
<blockquote>
<p>If I don’t run line of <code>L.backward()</code>, then it is fine. Running <code>L.backward()</code> gives memory leak.</p>
</blockquote>
</aside>
<p>Can you elaborate on how did you decide it is a memory leak?</p>

username: Zhenduo_Wang
cooked: <p>I use nvidia-smi -l 1 to watch the memory change. It gradually increases by ~2MB each epoch. After a long time (about 20 hrs), any line that allocates gpu will raise CUDA out of memory error.</p>
Using libtorch with wsl2 with gpu support

username: alfie-nsugh
cooked: <p>I tried to use libtorch on WSL2 with GPU support. This didn’t work however, but pytorch can be used with cuda support without any issue. Will GPU support on wsl2 for libtorch be added? Or if this is already supported will there be a tutorial that shows you how to compile something with libtorch with gpu support?</p>
Get the value of crossentropy loss for each sample in the batch

username: Shantanu_Ghosh
cooked: <p>Hi,</p>
<p>I want to get the value of the nn.crossentropy loss for each sample in the batch. Can someone please guide me?</p>
<p>For example: i have 5 samples in the batch. So i have 5 target values and 5 prediction values. So if do something like this:</p>
<p>Loss = nn.crossentropy(ypred, ytarget)</p>
<p>Now this loss is a scalar for all the samples in the batch. Now i want to get the loss as a vector of size equals to the elements in the batch. So if the batch has 5 elements, so the loss has vector of size 5 elements. I want to know this for nn.bce and nn.mse and nn.kld loss. I guess it will be same for all the losses. I want to do it because i want to multiply it with a factor to each loss per sample. For the final loss value i will do it by torch.sum</p>

username: Matias_Vasquez
cooked: <p>The default reduction is set to <code>mean</code>. You just need to specify <code>reduction='none'</code> and you will get this vector that you want.</p>
<pre><code class="lang-python">#Example
criterion= nn.CrossEntropyLoss(reduction='none')

loss = criterion(y_pred, y_target) # ← This will give you a tensor
</code></pre>
<p><a href="https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html" rel="noopener nofollow ugc">Here</a> is the documentation for <code>nn.CrossEntropyLoss</code>.</p>
Pytorch on Jetson Nano 2GB

username: Ondrej_Zeleny
cooked: <p>Hi,<br>
I’ve been trying to replicate this (<a href="https://pytorch.org/blog/running-pytorch-models-on-jetson-nano/#overview" class="inline-onebox" rel="noopener nofollow ugc">Running PyTorch Models on Jetson Nano | PyTorch</a>)o) blog post and run my own implementation of the yolov5. I followed the steps in the post and installed pytorch 1.9 which however did not installed properly since python enviroment says its pytorch 1.8., secondly the link for torchvision is no-longer working and i did not managed to install it. Im trying to run inference using containers and i even downloaded the docker image for pytorch 1.8. The issues for me is i have no idea how to run is this way and i dont really understand things like container,image, docker. With that being said, could anyone help me with the step-by-step how to run my python script in the <a href="http://nvcr.io/nvidia/l4t-pytorch" rel="noopener nofollow ugc">nvcr.io/nvidia/l4t-pytorch</a>   r32.6.1-pth1.8-py3 image ?</p>
Register layers within list as parameters

username: Syzygianinfern0
cooked: <p>Due to some design choices, I need to have the pytorch layers within a list (along with other non-pytorch modules). Doing this makes the network un-trainable as the parameters are not picked up with they are within a list.</p>
<p>This is a dumbed down example.</p>
<pre><code class="lang-python">from torch import nn
import numpy as np


class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.modules = [nn.Linear(16, 8), np.multiply]


print(list(Net().parameters()))  # prints an empty list - expected to print weights of nn.Linear
</code></pre>
<p>How can I achieve what I expect?</p>

username: Syzygianinfern0
cooked: <p>One answer is to use <code>nn.ModuleList</code>. But this does not allow non-pytorch modules to be a part of it. Again, due to some design choices, I require both the pytorch and non-pytorch modules to be part of the same iterable.</p>

username: InnovArul
cooked: <aside class="quote no-group" data-username="Syzygianinfern0" data-post="1" data-topic="150761">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/syzygianinfern0/48/46517_2.png" class="avatar"> Syzygianinfern0:</div>
<blockquote>
<p><code>np.multiply</code></p>
</blockquote>
</aside>
<p>I am not sure of your goal here.<br>
Technically you could write a dummy <code>torch.nn</code> Module to hold non-pytorch modules, as below:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import numpy as np

class NonTorchContainer(nn.Module):
    def __init__(self, fn):
        self.fn = fn

    def forward(self,): pass

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.modules = nn.ModuleList([nn.Linear(16, 8), 
                                      NonTorchContainer(np.multiply)])

    def forward(self,): pass

if __name__ == '__main__':
    x = Net()
</code></pre>
<p>I hope you are aware that non-pytorch modules will break the computation graph and the gradients won’t be backpropagated.</p>

username: Syzygianinfern0
cooked: <p>Although <a class="mention" href="/u/innovarul">@InnovArul</a>’s answer is perfectly OK, a more favourable solution for me was this.</p>
<p>I’ve solved it by just adding a line which creates a class variable which refers to the list element.</p>
<pre><code class="lang-python">import numpy as np
import torch
from torch import nn


class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.modules = [nn.Linear(3, 1), np.multiply]
        self.one = self.modules[0]  # Just have this line extra


model = Net()
print(list(model.parameters()))

print(model.modules[0].weight)  # View original weights
optim = torch.optim.SGD(model.parameters(), lr=0.1)
x = model.modules[0](torch.ones(1, 3) * 5) - 1.0  # Still using from self.modules
print(x)
x.backward()
print(model.modules[0].weight.grad)  # Grad WILL be calculated!
optim.step()
print(model.modules[0].weight)  # View updated weights
</code></pre>
To freeze layers, is it enough to set requires_grad to False?

username: tstandley
cooked: <p>Very similar questions have been asked before, but this one is subtly different.</p>
<p>Like many, I want to freeze some layers of my neural network. I understand that I can not add those parameters to the optimizer. I understand that I should set requires_grad to False for these layers, so that back propagation doesn’t do extra work.</p>
<p>But I want to know if only setting requires_grad to False is sufficient for the right thing to happen.</p>
<p>I’m worried that parameters without gradients are treated as having zero gradients by the optimizer. That would mean that it would still apply weight decay to those layers.</p>
<p>Does anybody know for sure if that’s a real problem or not?</p>
<p>I’m going to be changing which layers are frozen during training, and I’d rather not have to create a new optimizer every time I change requires_grad for a layer.</p>
<p>Thanks!</p>

username: Matias_Vasquez
cooked: <p>Hi, setting it to <code>False</code> is sufficient for the right thing to happen.</p>
<p><a href="https://discuss.pytorch.org/t/passing-to-the-optimizers-frozen-parameters/83358/2">Here</a> is a very good explanation.</p>
<p>But just to be sure, here is the source code for the <code>SGD</code> optimizer. As you can see, when updating the parameters the flag is checked and if set to <code>False</code> then nothing will happen. So it will no be taken as having zero gradient, it will just do nothing for those parameters.</p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/b7b99ab0c8f82100177729b9751481852d83e77e/torch/optim/sgd.py#L95-L96">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/b7b99ab0c8f82100177729b9751481852d83e77e/torch/optim/sgd.py#L95-L96" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/b7b99ab0c8f82100177729b9751481852d83e77e/torch/optim/sgd.py#L95-L96" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/b7b99ab0c8f82100177729b9751481852d83e77e/torch/optim/sgd.py#L95-L96</a></h4>



    <pre class="onebox">      <code class="lang-py">
        <ol class="start lines" start="95" style="counter-reset: li-counter 94 ;">
            <li>if p.grad is None:</li>
            <li>    continue</li>
        </ol>
      </code>
    </pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


username: tstandley
cooked: <p>Thanks <a class="mention" href="/u/matias_vasquez">@Matias_Vasquez</a>!</p>
How can I get the supported type of native function？

username: yi_yang
cooked: <p>hi, I want to get the supported type of native function which is defined in native_function.yaml, such as, we know add op in gpu support Double, Float, Half…, but I found all dtype defined in native functions is tensor/scalar, beacuse they are unboxed function. I want to add some log to get the supported dtype information, but I dont know where to add. Thanks very much!</p>
Attention Between 3D Matrix and a vector

username: hadarshavit
cooked: <p>Hi,</p>
<p>I’m trying to implement channel attention. I have a matrix of shape NxCxWxH where N is batch size, C is channels, W width, H height and a vector of size NxC. I’m trying to perform a multipication between them.<br>
I already tried to x * y, torch.matmul(x, y) x * y.expand_as(y) but they all resulted in an error that there is a dimension mismatch.<br>
How can I implement it?<br>
Thanks</p>

username: InnovArul
cooked: <p>There are many ways to achieve it. One of the easy ways to multiply tensors with multiple dimensions is to make use of <a href="https://pytorch.org/docs/stable/generated/torch.einsum.html"><code>torch.einsum()</code></a>.</p>
<p>In your case, you can code something like below:</p>
<pre><code class="lang-auto">N,C,H,W = 2,3,4,5

import torch
x = torch.randn(N,C,H,W)
y = torch.randn(N,C)
torch.einsum('bchw,bc-&gt;bhw', x, y)
</code></pre>

username: hadarshavit
cooked: <p>Thanks! However I need the output to be in shape (N, C, H, W), like the first input.</p>

username: InnovArul
cooked: <p>use <code>torch.unsqueeze(dim=1)</code> to introduce a singleton dimension after the multiplication.</p>
<pre><code class="lang-auto">result = torch.einsum('bchw,bc-&gt;bhw', x, y) # b,h,w
result = result.unsqueeze(dim=1)  # b,1,h,w
</code></pre>
<p>You can look at <a href="https://pytorch.org/docs/stable/tensors.html#torch.Tensor.mH"><code>torch.Tensor</code></a> API page to know the different functions available to manipulate a tensor.</p>
Quantization post exporting to ONXX

username: chhana
cooked: <p>Pytorch model converted to ONXX model and then I tried quantizing the ONXX model expecting faster inference. Unfortunately it didn’t. Any hints to proceed with quantization for an ONXX model would be really helpful.</p>
<p>Thanks in advance.</p>

username: andrewor
cooked: <p>Hi Harish, did you mean ONNX models? Unfortunately exporting quantized models to ONNX is not an area the pytorch team is actively maintaining.</p>
CPU Full without any reason

username: Andre_Amaral_IST
cooked: <p>I am trying to run the code in the same way i did previously n times and it worked but now it is giving me the following error:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/a/2ae25ae6f4bc0ca67a304d8c76c39ed4b1851300.png" alt="image" data-base62-sha1="67n14tX2BTtOGntvwgxdLQDqDFm" width="689" height="48"></p>

username: ptrblck
cooked: <p>The screenshot is a bit hard to read (you can post code snippets by wrapping them into three backticks ```), but it seems you are trying to allocate <code>11714362584 / 1024**3 ~= 10.9GB</code> of data while your RAM is already full.<br>
Check the memory usage via e.g. <code>htop</code> and make sure you have enough free RAM.<br>
Also, check the <code>X</code> dimensions and verify that <code>~10.9GB</code> are indeed expected as it seems you are surprised by this allocation.</p>

username: Andre_Amaral_IST
cooked: <p>Thanks for the tips.<br>
I already figure out how to workaround this problem!<br>
Thanks again for the help and sorry for the double post <img src="https://discuss.pytorch.org/images/emoji/apple/pensive.png?v=12" title=":pensive:" class="emoji" alt=":pensive:" loading="lazy" width="20" height="20"><br>
Regards,<br>
André</p>