akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Torch script and eager-mode hybrid interface

Scott_Hoang: <p>Hi all,<br>
While I am very familiar with PyTorch, I am a noob when it comes to torch.jit.<br>
I have several questions:</p>
<ol>
<li>Is it possible to start writing the scripting module now and still train your model following the standard PyTorch training regime?</li>
<li>does torch. Script support forward hooks?</li>
<li>Is it possible to write a hybrid model with some sub-modules that are script/trace while others are just standard nn.Module and still enjoy some benefit?</li>
</ol>
<p>I’m doing this because my model is still a work-in-progress, and many parts are subjected to change, while some are entirely static. So I want to test what speed-up I can get starting the transition now.<br>
Thanks</p>
Expand 2D sparse Tensor to do multiplication with 3D tensor

Gul_Zain: <p>Hello,<br>
I am trying to do a graphconv operation for a 3D data. Data ‘x’ is in shape [batch,seq_dim,feature_dim]. I have a sparse adjacency matrix ‘adj’ of shape [seq_dim,seq_dim]. I want to multiply adjacency matrix with input for all the samples. So i simply try to expand the adjacency matrix as adj.unsqueeze(0).expand([x.shape[0],-1,-1]). However, i get following error at this operation</p>
<blockquote>
<p>RuntimeError: sparse tensors do not have strides</p>
</blockquote>
<p>This error occurs at .expand operation. Unsqueeze() operation does not trigger this error. Please let me know if there is any other way around to do this operation with pytorch as well?</p>

Gul_Zain: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> quick help will be much appreciated <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

Matias_Vasquez: <p>This is just a workaround, maybe someone has a better solution.</p>
<p><a href="https://pytorch.org/docs/stable/sparse.html#supported-linear-algebra-operations" rel="noopener nofollow ugc">Here</a> is a list with the available linear algebra operations with sparse matrices.</p>
<p>However some of them do not support batch multiplication yet. (<a href="https://github.com/pytorch/pytorch/issues/14489" rel="noopener nofollow ugc">github issue</a>)</p>
<p>In this issue they mention that <code>bmm</code> can be used even with sparse matrices. However, it does not support broadcasting as shown in the <a href="https://pytorch.org/docs/stable/generated/torch.bmm.html" rel="noopener nofollow ugc">docs</a>. And sparse tensors cannot use the <code>repeat</code> method.</p>
<p>So the solution posted <a href="https://discuss.pytorch.org/t/torch-repeat-for-sparse-matrix/127581">here</a> is to stack your <code>sparse</code> tensors in the desired dimension.</p>
<p>This should work.</p>
<pre><code class="lang-python">batch = 3
seq_dim = 150
feature_dim = 300

i = [[0, 1, seq_dim-1], [0, 2, seq_dim-1]]
v =  [3, 4, 5]
adj = torch.sparse_coo_tensor(i, v, dtype=torch.float)
new_adj = torch.stack([adj for _ in range(3)], dim=0)

x = torch.rand(batch, seq_dim, feature_dim)

print(torch.bmm(input=new_adj, mat2=x).shape)
</code></pre>
Member values return wrong results if JITed

avitase: <p>Hi,</p>
<p>I have a custom <code>nn.Module</code> specialization that uses an <code>nn.ModuleList</code> internally:</p>
<pre><code class="lang-auto">class A(nn.Module):
    def __init__(self):
        super().__init__()
    
    def forward(self, x):
        return x if self.training else x - 1.0


class B(nn.Module):
    def __init__(self):
        super().__init__()
        self.a = nn.ModuleList([A(), A()])
    
    def forward(self, x):
        return self.a[1](self.a[0](x))
    
    
model = torch.jit.script(B())
</code></pre>
<p>Toggling between <code>train</code> and <code>eval</code> triggers the correct branch in <code>forward</code> but the reported values are wrong:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; model.train();
&gt;&gt;&gt; [a.training for a in model.a]
[False, False]

&gt;&gt;&gt; model.eval();
&gt;&gt;&gt; [a.training for a in model.a]
[False, False]
</code></pre>
<p>whereas the correct values are returned when the module is not JITed. The same seems to be true for any other member variables.</p>
<p>Is this a bug of <code>nn.ModuleList</code>?</p>
<p>Thanks for your help.</p>
Difference between torch.distributed and python run

base-y: <p>Hi, can I know whats the difference between the following:</p>
<ol>
<li><code>python train.py .....&lt;ARGS&gt;</code></li>
<li><code>python -m torch.distributed.launch &lt;ARGS&gt;</code></li>
<li><code>deepspeed train.py &lt;ARGS&gt;</code></li>
</ol>
<p>Even option 1 seem to be using some sort of distributed training when there are multiple gpus. My understanding was option 1 will only use 1 gpu if not given <code>distributed.launch</code>.</p>
<p>Also, does deepspeed use torch.distributed in the backend or is it something different?</p>
<p>thanks in advance</p>

ptrblck: <aside class="quote no-group" data-username="base-y" data-post="1" data-topic="150494">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/b/d2c977/48.png" class="avatar"> base-y:</div>
<blockquote>
<p>Even option 1 seem to be using some sort of distributed training when there are multiple gpus. My understanding was option 1 will only use 1 gpu if not given <code>distributed.launch</code>.</p>
</blockquote>
</aside>
<p>It depends what <code>train.py</code> uses as inside the script still multiple processes could be spawned.</p>

cbalioglu: <p>In addition to <a class="mention" href="/u/ptrblck">@ptrblck</a>’s answer; there is really no magic inside <code>python -m torch.distributed.launch</code>. It can be considered mostly a convenience launcher script that helps you to run distributed jobs while avoiding most of the boilerplate code. Ultimately it set ups the process group(s), environment variables, etc. that you can consume in your training script passed in <code>&lt;ARGS&gt;</code>. Technically there is nothing preventing you from replicating all that code in a custom <code>train.py</code> script (i.e. option 1), but in most use cases using <code>python -m torch.distributed.launch</code> is much easier and future-proof.</p>

base-y: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="150494">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>It depends what <code>train.py</code> uses as inside the script still multiple processes could be spawned.</p>
</blockquote>
</aside>
<p>I am using huggingface trainer inside <code>train.py</code>. Could be that the trainer is spawning multiple processes. Maybe thats why <code>option 1</code> too is using dist.training.</p>
Custome dataset : multiple input in pytorch

Hijab_Sehar: <p>I have a csv file having two columns image name(around 300) and  ID(say 1,2). Different images may have same ID so there can be more than one image against one ID.I want to assign 0 label for images sharing same ID and 1 for different. My model should read two images in testing phase and reply 1 if images have different IDS(1 and 2) and 0 if same images. There are two folders ID1 and ID2 and it has all images against each ID.<br>
Problem:<br>
what is best piece of code to override getitem,either converting images to numerical and concatenate or what?</p>
Can DistributedDataParallel still work efficiently on a single GPU?

mesllo: <p>From what I tested, DDP still runs for a single GPU, and essentially the DistributedSampler makes the dataset split by a single GPU - meaning that we get the same dataset unchanged. With this being said, are there any benefits to using DDP on a single GPU over the normal method?</p>

ptrblck: <p>No, I don’t think you should expect to see any benefits in launching <code>DDP</code> on a single GPU only, but would also assume to see no difference (between source code changes) to a standalone single-GPU script.</p>
Retrain pth model without model structure

Otis_Chang: <p>According to <a href="https://pytorch.org/tutorials/beginner/saving_loading_models.html" rel="noopener nofollow ugc">PyTorch documents</a>, it seems we have to define the model structure before we <code>model.load_state_dict(model_state_dict) </code>.<br>
If we save entire model to pth file and load pth by <code>model = torch.load(PATH)</code>, the directory structure should not be changed.</p>
<p>Actually, it seems we have to know model structure if we want to retrain model.</p>
<p>Does it exist any method or any weight file type that we can retrain the PyTorch model without knowing the model structure?</p>

ptrblck: <p>Scripting a model via <code>torch.jit.script</code> and storing it via <code>torch.jit.save</code> should create a model file which you could directly load via <code>torch.jit.load</code> and execute the model.</p>

Otis_Chang: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thanks for replying.</p>
<p>After I load scripted model via <code>model = torch.jit.load('model.pth')</code>, could I directly resume training by <code>model.train()</code>? or I need to save optimizer and loss, and load them while resuming training as well?</p>

ptrblck: <p>If you want to continue the training, you should save the <code>optimizer</code>'s <code>state_dict</code> as well (potentially also the learning rate scheduler’s <code>state_dict</code>). However, if you want to re-train or fine-tune the model, you could create a new <code>optimizer</code> and treat the model as a pre-trained one.</p>
Image patch Data Augmentation during training

cbd: <p>In some research paper, authors are using image patches as augmentation during training and use whole image in testing phase( Input and Output to CNN is image).</p>
<ol>
<li>How image patch helps instead resize image as CNN input?</li>
<li>Can we use this technique to all types of CNN architecture?</li>
<li>Should it be fail in some specific cases?</li>
</ol>

Sehaba95: <aside class="quote no-group" data-username="cbd" data-post="1" data-topic="148487">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/13edae/48.png" class="avatar"> cbd:</div>
<blockquote>
<p>Should it be fail in some specific cases</p>
</blockquote>
</aside>
<p>I have recently read this paper: <a href="https://hal.archives-ouvertes.fr/hal-02969811v2" rel="noopener nofollow ugc">Patch-based CNN evaluation for bark classification</a>, and the authors did a great comparison about classification using patches, with and without rescaling the original image. In short, the obtained results shows that using patches for classification (without rescaling) helps improve the accuracy, since we will not lose details of the object. But when classifying patches from a rescaled image we will lose in term of patch accuracy. I really suggest you to go through the paper, and you will understand more from the authors.</p>
<p>In general, this paper doesn’t reply to the exact question that you are asking but it could give you an idea about patch classification.</p>
How could I solve "No such file or directory" when using libtorch

Ehwartz: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/f/af67af3a28126804ae9bac183d9bce6597ae57cb.png" data-download-href="https://discuss.pytorch.org/uploads/default/af67af3a28126804ae9bac183d9bce6597ae57cb" title="6"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/f/af67af3a28126804ae9bac183d9bce6597ae57cb_2_690x441.png" alt="6" data-base62-sha1="p1HHpYgYyBPmc6FzLyzh1I6BpRp" width="690" height="441" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/f/af67af3a28126804ae9bac183d9bce6597ae57cb_2_690x441.png, https://discuss.pytorch.org/uploads/default/optimized/3X/a/f/af67af3a28126804ae9bac183d9bce6597ae57cb_2_1035x661.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/a/f/af67af3a28126804ae9bac183d9bce6597ae57cb_2_1380x882.png 2x" data-dominant-color="333132"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">6</span><span class="informations">3000×1920 374 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
FileNotFoundError: python setup.py develop

yukino: <p>I try <code>python setup.py develop</code> and <code>python setup.py build develop</code> ,but both get the error</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "setup.py", line 719, in &lt;module&gt;
    build_deps()
  File "setup.py", line 285, in build_deps
    build_dir='build')
  File "C:\Users\Jay\pytorch\tools\build_pytorch_libs.py", line 259, in build_caffe2
    my_env)
  File "C:\Users\Jay\pytorch\tools\build_pytorch_libs.py", line 239, in run_cmake
    check_call(cmake_args, cwd=build_dir, env=my_env)
  File "C:\Users\Jay\Anaconda3\lib\subprocess.py", line 306, in check_call
    retcode = call(*popenargs, **kwargs)
  File "C:\Users\Jay\Anaconda3\lib\subprocess.py", line 287, in call
    with Popen(*popenargs, **kwargs) as p:
  File "C:\Users\Jay\Anaconda3\lib\subprocess.py", line 729, in __init__
    restore_signals, start_new_session)
  File "C:\Users\Jay\Anaconda3\lib\subprocess.py", line 1017, in _execute_child
    startupinfo)
FileNotFoundError: [WinError 2] 系統找不到指定的檔案。
</code></pre>
<p>someone knows how to deal with this?<br>
python version: 3.6.8<br>
anaconda version: 1.7.2</p>

hvbsizssycnqfhaqupb: <p>It happened to me when I forgot to install some python packages.</p>
<pre><code class="lang-auto">pip install --upgrade setuptools
</code></pre>
<p>and</p>
<pre><code class="lang-auto">pip install astunparse numpy ninja pyyaml mkl mkl-include setuptools cmake cffi typing_extensions future six requests dataclasses
</code></pre>
<p>solved the issue. If you’re using Anaconda just replace pip with conda in the second line. More info here: <a href="https://datagraphi.com/blog/post/2021/9/13/building-pytorch-from-source-on-windows-to-work-with-an-old-gpu" class="inline-onebox" rel="noopener nofollow ugc">Building PyTorch from source on Windows to work with an old GPU</a></p>
How to build a new Dataset,such as ImageFolder？

casual_dong: <p>I have my own dataset. So, I want to build a CNN model based on my own dataset. I have to write a new Dataset function to substitute the official Dataset. But, I get an  error and will post picture in the end. My question is how to read an image in the official Dataset? In my code, i use torchvision.io.read_img to read an image, but get error “TypeError: pic should be PIL Image or ndarray. Got &lt;class ‘torch.Tensor’&gt;”.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/9/a903027d073a2598e71f64b54be7f8e51a071964.png" data-download-href="https://discuss.pytorch.org/uploads/default/a903027d073a2598e71f64b54be7f8e51a071964" title="uTools_1651817274401"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/9/a903027d073a2598e71f64b54be7f8e51a071964_2_690x414.png" alt="uTools_1651817274401" data-base62-sha1="o798Jf1qYPAp6uweGxeWnXgxGXq" width="690" height="414" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/9/a903027d073a2598e71f64b54be7f8e51a071964_2_690x414.png, https://discuss.pytorch.org/uploads/default/optimized/3X/a/9/a903027d073a2598e71f64b54be7f8e51a071964_2_1035x621.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/a/9/a903027d073a2598e71f64b54be7f8e51a071964.png 2x" data-dominant-color="F3F0F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">uTools_1651817274401</span><span class="informations">1149×690 92.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

casual_dong: <p>I have solved this problem, because I use trochvision.io.read_img to read IMG. So I get a Tensor, and think that transform.resize  operate in Img. So i need to convert tensor to img.  using transform.ToPILImage to do that.</p>

tom: <p>Good job finding your own solution.</p>
<aside class="quote no-group" data-username="casual_dong" data-post="2" data-topic="150918">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/casual_dong/48/48069_2.png" class="avatar"> casual_dong:</div>
<blockquote>
<p>So I get a Tensor, and think that transform.resize operate in Img</p>
</blockquote>
</aside>
<p>The transforms should also work on tensors these days.</p>
<p>Best regards</p>
<p>Thomas</p>
What is the usage of meta tensor and how it helps large models reduce memory footprint?

xiacijie: <p>Hello,</p>
<p>I am reading <a href="https://devblog.pytorchlightning.ai/experiment-with-billion-parameter-models-faster-using-deepspeed-and-meta-tensors-2e9c255edd71" rel="noopener nofollow ugc">this article</a> about meta tensor helps speed up billion-parameter models. I am very interested in the notion “meta tensor” but I couldn’t find much documentation about it.</p>
<p>So as I understand, the meta tensor is a tensor without content, and it only has meta data, for example, its shape. So when two meta tensors are combined together by an operator, the result tensor is also a meta tensor with an inferred shape. However, I cannot find any more usage of meta tensor beyond shape inferring. Therefore, I would like to know:</p>
<ol>
<li>Is there any other usage of meta tensor?</li>
<li>In the article I mentioned above, how exactly does the meta tensor help speed up the large models? What is the role meta tensor played with DeepSpeed?</li>
</ol>
Pytorch multiprocessing RuntimeError: Attempted to send CUDA tensor received from another process; this is not currently supported. Consider cloning before sending

Forceless: <p>Hi, I tried to copy many models and put them into a multiprocessing.manager.queue<br>
If I pass a model as an arg to a subprocess and use queue.put(deepcopy(model))<br>
It raise<br>
RuntimeError: Attempted to send CUDA tensor received from another process; this is not currently supported. Consider cloning before sending.<br>
Seems like this was becase copied model still have old model 's data<br>
reproduce script and more description is under replenishing</p>

Forceless: <p>I tried state_dict also, still don’t work</p>
How to solve the last conv_layer and linear layer fitting problem?

casual_dong: <p>I have created a CNN model. But I have a question that if I would use hyperparams to create a CNN<br>
model, I don’t know the output size of the last conv layer.  I need to write a code to special compute the output size of the last conv layer?</p>

ptrblck: <p>Assuming you would like to let PyTorch automatically compute the <code>in_features</code> of e.g. the first <code>nn.Linear</code> layer based on the used input shape, you could use the <code>nn.LazyLinear</code> module.</p>

casual_dong: <p>Thank you very much, I have solved this problem</p>
Dealing with multiple sequences in T5ForConditionalGeneration

Rafi: <p>Here I remark that the output of individual sequences are different from batched sequences using <code>T5ForConditionalGeneration</code></p>
<p>Here is an example to reproduce the result: <code>batch_size=1 vs 2</code></p>
<pre><code class="lang-auto">import torch

import torch.nn as nn

import torch.optim as optim

import torch.nn.functional as F

import pandas as pd

from torchtext.legacy.data import Field, BucketIterator, TabularDataset

# prepare data
data = {"text": ["summarize: i am very happy. i am very happy",
        "summarize: i am very safe. i am very safe"],
        "summary": ["i am very happy", "i am very safe"]}
df = pd.DataFrame(data)
df.to_csv("debug.csv", index=False)

# set tokenizer of T5-small

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

tokenizer = AutoTokenizer.from_pretrained("t5-small")

pad_index = tokenizer.convert_tokens_to_ids(tokenizer.pad_token)

unk_index = tokenizer.convert_tokens_to_ids(tokenizer.unk_token)

eos_index = tokenizer.convert_tokens_to_ids(tokenizer.eos_token)

model = AutoModelForSeq2SeqLM.from_pretrained("t5-small")

model.resize_token_embeddings(len(tokenizer))

model.to("cuda")

from transformers import T5Tokenizer, T5ForConditionalGeneration

SRC = Field(tokenize = tokenizer.encode, 
            use_vocab=False,
            lower = False,
            init_token = None, 
            eos_token = eos_index, 
            pad_token=pad_index,
            unk_token=unk_index,
            include_lengths = True)

TRG = Field(tokenize = tokenizer.encode, 
            use_vocab=False,
            init_token = None, 
            eos_token = eos_index, 
            pad_token=pad_index,
            unk_token=unk_index,
            lower = False)

fields = {"text": ("src", SRC), "summary": ("trg", TRG)}
train_data, valid_data, test_data = TabularDataset.splits(
    path="./",
    train="debug.csv",
    validation="debug.csv",
    test="debug.csv",
    format='csv',
    fields=fields)


BATCH_SIZE = 1

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

train_iterator, valid_iterator, test_iterator = BucketIterator.splits(
    (train_data, valid_data, test_data), 
     batch_size = BATCH_SIZE,
     sort_within_batch = True,
     sort_key = lambda x : len(x.src),
     device = device)

for i, batch in enumerate(train_iterator):
    src, _ = batch.src
    trg = batch.trg
    logits = model(input_ids=src.view(src.shape[1], src.shape[0]),
                   labels=trg.view(trg.shape[1], trg.shape[0])).logits
    X = logits.view(logits.size(1), logits.size(0), logits.size(-1))
    X = F.softmax(X, dim=-1)
    ids = X.argmax(dim=-1)
    y = tokenizer.batch_decode(sequences=ids, skip_special_tokens=False)
    z = tokenizer.batch_decode(sequences=trg, skip_special_tokens=False)
    print(" ".join(y))
    print("*********")
    print(" ".join(z))
    print("*********")
</code></pre>
Libtorch seems does not compatible with ITk

Fuzhiyuan: <p>My libtorch version is libtorch-shared-with-deps-1.11.0+cu113<br>
ITK version is 5.2.1</p>
<p>When i annotate  <em><strong>target_link_libraries(libtorch_test “${TORCH_LIBRARIES}” )</strong></em>,  <em><strong>make</strong></em> can work perfectly. But i delete these annotation ,some problem will occur, which will throw an error : undefine reference to Itk::::ExceptionObject::SetLocation(std::string const&amp;)<br>
The following is CmakeList(No annotation):</p>
<pre><code class="lang-auto">cmake_minimum_required(VERSION 3.0.0 FATAL_ERROR)
project(libtorch_test)


set(CMAKE_CUDA_ARCHITECTURES 70 75 80)
set(CUDNN_LIBRARY_PATH "/usr/local/cuda-11.3/lib64")
set(CUDNN_INCLUDE_PATH "/usr/local/cuda-11.3/include")
set(ITK_DIR "/home/xxxx/Lib/ITK/build") 

set(CMAKE_PREFIX_PATH "/home/xxxx/Lib/libtorch-shared-with-deps-1.11.0+cu113/libtorch/share/cmake/Torch" "/home/xxxx/Lib/InsightToolkit-5.2.1/build" ) 
find_package(Torch REQUIRED)
find_package(ITK REQUIRED)
include(${ITK_USE_FILE})
add_executable(libtorch_test deploy.cpp)
target_link_libraries(libtorch_test "${TORCH_LIBRARIES}" )
target_link_libraries(libtorch_test "${ITK_LIBRARIES}")
set_property(TARGET libtorch_test PROPERTY CXX_STANDARD 14)

</code></pre>
Save numpy as image in pytorch (data type : float32, clipped between [0,1])

gesm: <p>Hi there,</p>
<p>is there any way to save a NumPy array as image in pytorch (I would save the numpy and not the tensor) without using OpenCV… (I want to save the NumPy data as an image without multiplying by 255 or adding any other prepro)</p>
<p>Thanks</p>

Andrei_Cristea: <p>What exactly do you mean by saving it as an image “in PyTorch”? An image could take several forms:</p>
<ul>
<li>a numpy array</li>
<li>a Tensor (pytorch object)</li>
<li>a PIL Image object</li>
</ul>
<p>Are you trying to save an image on disk? Convert a numpy array to a Tensor? Something else?</p>
<p>A little more detail about what you’re trying to accomplish would help us answer.</p>

gesm: <p>Thanks for your reply.</p>
<p>What I did is I first loaded a subset of images and convert the tensors to NumPy and now I want to save these images on disk in (.png, jpeg) format but without doing any preprocess as the data is float 32 with values between 0 and 1. Using OpenCV for example requires the multiplication of the array values by 255. But I want to save the array as an image on disk without changing any values.</p>
<p>Thanks,</p>

Andrei_Cristea: <p>I don’t know if there’s a way to save something as png without multiplying by 255 first, however you don’t need to go through OpenCV, since you can do it through PIL directly:</p>
<pre><code class="lang-auto">from PIL import Image
image_numpy = np.random.rand(224, 224, 3)
img = Image.fromarray(np.uint8(255 * image_numpy))  # no opencv required
img.save("file.png")
</code></pre>
<p>You can also save the numpy array directly to disk, however your filesystem won’t recognize it as an image (like you can’t open it as a png):</p>
<pre><code class="lang-auto">np.save("file.npy", image_numpy)
</code></pre>
Pytorch lightning log (events file) - unreadable format

natgolovach: <p>Hi All!</p>
<p>For some reason I can not read the logs from logs files as they look like that:</p>
<pre><code class="lang-auto">ahparams"`u∑œŸ‹2	≥jÍ∏ÕúÿA*

val_lossêéí&gt;3¯°ıe
û	≠lÍ∏ÕúÿA*

epochzÙçwŸ‹2	ÿ
fªÕúÿA*

val_loss‰ÇÃ=ˆeﬁ4e
û	ÛfªÕúÿA*
</code></pre>
<p>Why is that happening? this is “…lightning_logs/version_X/events.out.tfevents.1650580020.run-625dc32b6711b8302f650b5a-s5w9h.2705.0” file</p>

ptrblck: <p>If I’m not mistaken, <code>tfevent</code> files are stored as <code>protobuf</code> files so you might be able to read them via a <code>protobuf</code> library.</p>

natgolovach: <p>thank you! that helped</p>
How to perform weight regularization in pytorch?

Jabu: <p>I have a model implemented in pytorch that applies a final fully connected layer before running the softmax function. The architecture is defined to solve a 4-class Speech Emotion Recognition task: given an audio track, it transforms it into its spectrogram and uses it to predict the emotion between happiness, sadness, neutrality and anger.</p>
<p>Unlike the architecture of the paper, it attempts to adapt the implementation of the Compact Convolutional Transformer found on Github at the link <a href="https://github.com/SHI-Labs/Compact-Transformers/blob/main/src/cct.py" class="inline-onebox" rel="noopener nofollow ugc">Compact-Transformers/cct.py at main · SHI-Labs/Compact-Transformers · GitHub</a>.</p>
<p>To improve the performance of the model I am following some tricks defined in the paper <a href="https://arxiv.org/abs/2104.07288" class="inline-onebox" rel="noopener nofollow ugc">[2104.07288] Speaker Attentive Speech Emotion Recognition</a>. Like what is described in the paper, however, my model also suffers from a “class collapse” problem: even by balancing the dataset, it tends to predict the anger and sadness classes well and the other two badly.</p>
<p>In the paper to solve this problem they apply a particular weight regularization technique to the fully connected layer, described in chapter 2.4.</p>
<p>Unfortunately, however, I cannot understand how I should modify my fully connected layer in pytorch to implement this type of regularization.</p>
<p>Can someone help me?</p>
Altering a Tensor Along a Variable Dimension

bchen: <p>Hi all,</p>
<p>Is there a way to alter a tensor given an input dimension and indexes? As an example, suppose I wanted to multiply the first and second index of the third dimension of a tensor by 2. I could do:</p>
<p>t = torch.randn([5, 5, 5, 5, 5])<br>
t[:, :, [1,2], :, :] *= 2</p>
<p>But suppose I don’t know ahead of time the dimension. Is there a way to do the following in a nice clean way?</p>
<pre><code class="lang-auto">def multiply_by_2(t, idx, dim):
  if dim == 0:
    t[idx, ...] *= 2
  if dim == 1:
    t[:, idx, ...] *=2
</code></pre>
<p>In other words, is there something like index_select that allows me to change the original tensor?</p>
<p>Thanks</p>

Andrei_Cristea: <p>Perhaps there is a better way, but here’s an ugly solution involving transposing the desired dimension into the first position, making the change there, and then transposing it back.</p>
<pre><code class="lang-auto">t = torch.randn([4, 4, 4])  # our starting tensor

 # your way
t0 = torch.clone(t)
t0[:, [1, 2]] = 1.

# proposed way, via transposing
dim = 1  # desired dimension to edit
t1 = torch.clone(t)
dims = list(range(t1.ndim))
dims.remove(dim)
dims = [dim] + dims
t1 = t1.permute(dims)
t1[[1, 2]] = 1
t1 = t1.permute(tuple(np.argsort(dims)))

print(torch.allclose(t0, t1))
Output:
True
</code></pre>
<p>Yuck!</p>

Matias_Vasquez: <p>You can use <code>index_select</code> and then <code>index_copy_</code>.</p>
<pre><code class="lang-python">a = torch.rand(5,5,5,5,5, requires_grad=True)
dim = 2
index = torch.tensor([2, 3]) 

def index_select_mult(a, dim, index, mult):
  b = a.clone()
  temp = torch.index_select(a, dim, index)
  temp = temp * mult
  b.index_copy_(dim, index, temp) 
  return b

print(a)
a = index_select_mult(a, dim, index, 10) 
print(a)
</code></pre>
<p>Hope this helps <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=12" title=":smile:" class="emoji" alt=":smile:" loading="lazy" width="20" height="20"></p>

bchen: <p>Thanks guys - both approaches seem to work!</p>
How to use pytorch profiler record_function for c++ code

annyan: <p>Hi,<br>
I want to add some datapoints for custom c++ code I have for training. I know I can use profile record_function to add to profiler from python code. But my c++ op is pretty heavy and I want to add some more fine grain output from c++ side. I was able to track down the impl of record_function and found those c++ impl code:<br>
third_party/pytorch/torch/csrc/autograd/record_function_ops.cpp</p>
<p>But when I called record_function_enter_new and record_function_exit_new from my c++ code, it didn’t show up in final profiling result? I cannot find any useful examples in code base for how to call those functions. Any pointers are appreciated.</p>
<p>Thanks</p>
Assign two models to two GPUs and run concurrently

Z_Huang: <p>Hi community,</p>
<p>Let’s say I have 2 CNN models and I want to assign each of the models to a GPU (2 GPUs on my device). Then I want to make these 2 models run concurrently. How may I do that? I tried lots of methods, but they are all running sequentially. Any suggestion?</p>

ptrblck: <p>Your approach should be possible, but assuming you are trying to drive both GPUs from a single process, the CPU might not be able to run ahead and schedule the kernels fast enough.<br>
Also, if you have any synchronization points, the CPU will of course be blocked and would have to wait until it can continue with the program execution.<br>
The easiest way would probably be to launch different processes (scripts) or make sure that you are not blocking the CPU in your single script.</p>

Z_Huang: <p>I made two models to work on two GPUs concurrently just now. I am so excited.</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torchvision
from concurrent.futures import ThreadPoolExecutor, wait
import time

device_0 = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
device_1 = torch.device("cuda:1" if torch.cuda.is_available() else "cpu")

start_time = time.time()

dense = torchvision.models.densenet121(pretrained=False).to(device_0)
rest = torchvision.models.resnet101(pretrained=False).to(device_1)

dummy1 = torch.ones((200,3,32,32)).to(device_0)
dummy2 = torch.ones((200,3,32,32)).to(device_1)

def train(model,value,epoch):
    ti = time.perf_counter()
    for _ in range(epoch):
        ret = model(value)
    to = time.perf_counter()
    return ret.shape, to-ti

output = []

with ThreadPoolExecutor() as executor:
    futures = []
    futures.append(executor.submit(train, dense, dummy1, 1000))
    futures.append(executor.submit(train, rest, dummy2, 1000))
    complete_futures, incomplete_futures = wait(futures)
    for f in complete_futures:
        output.append(f.result())
        print(str(f.result()))

elapsed = (time.time() - start_time)
print(f"Total time of execution {round(elapsed, 4)} second(s)")
print("Output is:",output)
</code></pre>
<pre><code class="lang-auto">(torch.Size([200, 1000]), 25.267832748999353)
(torch.Size([200, 1000]), 28.169526882003993)
Total time of execution 28.7672 second(s)
Output is: [(torch.Size([200, 1000]), 25.267832748999353), (torch.Size([200, 1000]), 28.169526882003993)]
</code></pre>
How to automatically detect the foot length from a 3D scan image?

tanvir_14: <p>I have some 3D scan images of the foot. How can I detect the foot length automatically from that image? I have the following information from the images only:</p>
<ol>
<li>Toes</li>
<li>Metatarsals</li>
<li>Heel</li>
<li>weight</li>
<li>sex</li>
</ol>

Andrei_Cristea: <p>As a starting point, how would you as a human determine the foot length based on that data?</p>

tanvir_14: <p><a class="mention" href="/u/andrei_cristea">@Andrei_Cristea</a> I can measure both length and width by using tape.</p>

Andrei_Cristea: <p>In that case, it sounds like you need to do something like the following flow. I’m afraid this isn’t going to be anywhere close to a step-by-step guide.</p>
<ul>
<li>gather as much data as you can</li>
<li>label the data yourself, so you end up with pairs of (3d scan, actual length)</li>
<li>train a 3d CNN model (utilizing Conv3d layers) whose last layer is a Linear layer with out_features=1. perhaps <a href="https://github.com/timzhang642/3D-Machine-Learning" rel="noopener nofollow ugc">something in this repo</a> can inspire your choice of model. use data augmentations that leverage the fact that what you’re measuring is invariant to rotation (e.g. rotate the data, add some pixel noise, and so on). perhaps you can transfer-learn the feature layers from a comparable pretrained model, since you may not have quite enough data yourself to train it.</li>
</ul>
<p>This isn’t a very helpful reply, it sounds like a challenging problem. Good luck!</p>

tanvir_14: <p><a class="mention" href="/u/andrei_cristea">@Andrei_Cristea</a>  unfortunately i don’t have enough data…do u know about any models that would be helpful to detect the landmarks of the foot?</p>

Andrei_Cristea: <p>Unfortunately I do not.</p>
Understanding of LSTM

DanielC: <p>Hi there,<br>
I am trying to implement a LSTM network which uses batch of noise as input, and get output with same size. The input size is 1(single number), time step is 10, which means the input noise and output shape are both (batch_size, 10, 1).</p>
<p>Therefore I built my network as following:</p>
<pre><code class="lang-auto">class GeneratorNet(torch.nn.Module):

    def __init__(self, hidden_dim, z_dim):
        super(GeneratorNet, self).__init__()

        self.rnn = nn.LSTM(
            input_size=1,
            hidden_size=32,
            num_layers=1,
            batch_first=True,
        )
        self.out = nn.Sequential(
            nn.Linear(32,10),
            nn.Sigmoid())

    def forward(self, z):
        r_out, (h_n, h_c) = self.rnn(z, None)
        out = self.out(r_out[:, -1, :])
        return out
</code></pre>
<p>The problem is that if I use different random input, the output is almost same. Here is the code I test result</p>
<pre><code class="lang-auto">generator = GeneratorNet()
for i in range(1, 5):
    # the size of noise is (1,10,1), batch_size is 1.
    noise = torch.randn(size=(1,10,1))
    # the output size is (1, 10, 1)
    out = generator(noise)
    plt.plot(out.detach().numpy()[0])
plt.show()
</code></pre>
<p>And here is the output result I get:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/6/d/6dbd3e4af2030784884f29b1dab44af7c69281f4.png" alt="Screen Shot 2022-05-05 at 1.20.15 PM" data-base62-sha1="fENuLzhrfFXCOOZNgl7tG6ZFJ1W" width="280" height="207"><br>
Does anyone can explain why the output is so similar?</p>
Pretrain Resnet34 based on mel spectrogram features

Gorgen: <p>Hello everyone:<br>
mel spectrum was sent to resnet34 for pre-training to extract high-dimensional audio representation. If log_S(mel, frames) (128,196), each sample needs to be filled or intercepted (128,500) and sent to the resnet34 model for training?<br>
Question 1. Does the padding need to be normalized before it can be sent to the model for training?<br>
Question 2. Does resnet34 need to be consistent for the latter two dimensions (height, width), analogous to mel, frames, (128, 500), for better results? Looking at resnet34, most of the last two dimensions are 224, 224 or the same dimension.<br>
Question 3: Transforms are required for watching the tutorial. What is the normal operation? transforms = transforms.Compose([<br>
transforms.Resize(256),<br>
transforms.CenterCrop(224),<br>
transforms.ToTensor(),<br>
transforms.Normalize(mean=[0.485], std=[0.229])<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/1/51cae353477e1a85bea18df2bdbb3a63ea324b9a.png" data-download-href="https://discuss.pytorch.org/uploads/default/51cae353477e1a85bea18df2bdbb3a63ea324b9a" title="1"><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/1/51cae353477e1a85bea18df2bdbb3a63ea324b9a.png" alt="1" data-base62-sha1="bFzmmdD7NpSYpwoYA2CO0qTcgY2" width="690" height="360" data-dominant-color="D9E5E6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">1</span><span class="informations">810×423 20.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Efficient way to crop and resize

jhp: <p>Hi Im doing <code>crop and resize</code> during training, but the number of boxes in each image is too large so that the time consumption in <code>crop and resize</code> operation for each box seems to be heavy.<br>
I wonder is there any tools that can efficiently operate <code>crop and resize</code>. (e.g, batched mode for crop and resize)</p>

subhashnerella: <p>Try roi_pool or roi_align in torchvision.</p>

jhp: <p>Thanks. That might be a good option.<br>
I forgot to mention I have to zero pad to the cropped image to maintain aspect ratio…<br>
I.E. crop-&gt;pad-&gt;resize</p>
<p>A lot of searching but it seems that the operation should be done with each box solely…</p>

subhashnerella: <p>roi_align takes output_size as an input. You dont have to zero pad it to maintain aspect ratio unless you want to do it that way.</p>
RuntimeError: mat1 and mat2 shapes cannot be multiplied (2x64 and 1024x1)

nour_nour: <p>I would like to output two information based on the below unet model, a classification at image level using a linear layer and a segmentated image, but the linear layer does’nt work and I got this error:   RuntimeError: mat1 and mat2 shapes cannot be multiplied (2x64 and 1024x1)</p>
<p>"</p>
<pre><code class="lang-auto">class UNet(nn.Module):

    def __init__(self, num_classes, in_channels=3, depth=5, start_filts=64, up_mode='transpose', merge_mode='concat'):
      
        super(UNet, self).__init__()

        self.up_mode = up_mode


        self.num_classes = num_classes
        self.in_channels = in_channels
        self.start_filts = start_filts
        self.depth = depth

        self.down_convs = []
        self.up_convs = []

        # create the encoder pathway and add to a list
        for i in range(depth):
            ins = self.in_channels if i == 0 else outs
            outs = self.start_filts*(2**i)
            pooling = True if i &lt; depth-1 else False

            down_conv = DownConv(ins, outs, pooling=pooling)
            self.down_convs.append(down_conv)

        # create the decoder pathway and add to a list
        for i in range(depth-1):
            ins = outs
            outs = ins // 2
            up_conv = UpConv(ins, outs, up_mode=up_mode,merge_mode=merge_mode)
            self.up_convs.append(up_conv)

        self.conv_final = conv1x1(outs, 1, 1)

        # add the list of modules to current module
        self.down_convs = nn.ModuleList(self.down_convs)
        self.up_convs = nn.ModuleList(self.up_convs)
        
        self.linear = nn.Linear(1024, 1)
        self.activation = nn.ReLU()
       

    def forward(self, x):
        encoder_outs = []
        
        # encoder pathway, save outputs for merging
        for i, module in enumerate(self.down_convs):
            x, before_pool = module(x)
            encoder_outs.append(before_pool)

        
            x_ = torch.sum(self.activation(x), [2,3])
            out_linear = self.linear(x_)
        
        
    #def decoder(self)     
        for i, module in enumerate(self.up_convs):
            before_pool = encoder_outs[-(i+2)]
            x = module(before_pool, x)
        
        x = self.conv_final(x)
        return x, out_linear
</code></pre>

Alvaro-Ciudad: <p>You are trying to multiply a 2 by 64 to 1024 by 1. If you change the 1024 of your linear layer to 64 it will work</p>
Output tensor for critic network in A2C

sofiane: <p>The output tensor of the critic in A2C algorithm contains probability distribution parameters, Those parameters will be used to sample from pd.<br>
Suppose we are on the continuous case, I would like to sample an action from a normal distibution with those parameter to create an action tensor,</p>
<ul>
<li>Do I need to have twice the size of the action space at the output of the critic to take into consideration the mean and the std of the normal distributions (suppose diagonal covariance matrix)?</li>
<li>How can I interpret p(<strong>a|s</strong>) with a neural network ? I cannot see the difference with p(<strong>s|a</strong>), is <strong>s</strong> or <strong>a</strong> the input of the neural network or both?</li>
</ul>
BCELoss + Automatic Mixed Precision

SofiaCP: <p>Dear all,</p>
<p>I wanted to use automatic mixed precision to train my model.</p>
<p>The output of my model is a weighted average of the output of several components, <code>Y= w1y1 + w2y2 + ... + wkyk</code>,</p>
<p>where <code>y1,...yk</code> is the output of each component <strong>after applying the sigmoid function</strong> (like a weighted averaged ensemble).</p>
<p>For this reason, I cannot use BCEWithLogitsLoss since simply taking the sigmoid of Y is not equivalent to the weighted average of the sigmoid outputs.</p>
<p>Any suggestions?<br>
Many thanks!</p>

ptrblck: <p>From the docs:</p>
<blockquote>
<p>The backward passes of <a href="https://pytorch.org/docs/stable/generated/torch.nn.functional.binary_cross_entropy.html#torch.nn.functional.binary_cross_entropy"><code>torch.nn.functional.binary_cross_entropy()</code></a> (and <a href="https://pytorch.org/docs/stable/generated/torch.nn.BCELoss.html#torch.nn.BCELoss"><code>torch.nn.BCELoss</code></a>, which wraps it) can produce gradients that aren’t representable in <code>float16</code> . In autocast-enabled regions, the forward input may be <code>float16</code> , which means the backward gradient must be representable in <code>float16</code> (autocasting <code>float16</code> forward inputs to <code>float32</code> doesn’t help, because that cast must be reversed in backward). Therefore, <code>binary_cross_entropy</code> and <code>BCELoss</code> raise an error in autocast-enabled regions.</p>
</blockquote>
<p>Assuming you can guarantee the numerical stability, remove the loss calculation (and maybe the weighting operations beforehand) from the <code>autocast</code> region to use <code>float32</code> for them.</p>

SofiaCP: <p>Sounds good. Will give it a try. Thank you!</p>
Match CNN output with LSTM input dimensions

natalie-sch: <p>I would like to build a hybrid CNN-LSTM model and have training samples of the following shape: (21,6000)</p>
<p>I built the CNN with the following layers</p>
<pre><code class="lang-auto">from torchsummary import summary
summary(model, (21, 6000))
</code></pre>
<p>returns<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/f/3fb9fdcc91eda2fdaf97b362437efbde4eb09a79.png" data-download-href="https://discuss.pytorch.org/uploads/default/3fb9fdcc91eda2fdaf97b362437efbde4eb09a79" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/f/3fb9fdcc91eda2fdaf97b362437efbde4eb09a79_2_434x375.png" alt="image" data-base62-sha1="95Kz3k5IsICgh9J30smpkL6NzqF" width="434" height="375" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/f/3fb9fdcc91eda2fdaf97b362437efbde4eb09a79_2_434x375.png, https://discuss.pytorch.org/uploads/default/original/3X/3/f/3fb9fdcc91eda2fdaf97b362437efbde4eb09a79.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/3/f/3fb9fdcc91eda2fdaf97b362437efbde4eb09a79.png 2x" data-dominant-color="F5F5F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">608×525 101 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I would like to use the output of the CNN model (the output from the layer before softmax, for example) and use it as an input for the LSTM. To align the CNN output with the LSTM input, I looked into examples using <code>padding=same</code> and <code>permute</code>.<br>
Could you give me a hint on what input shape the LSTM would expect here and how I could transform the output shape of the CNN to match that input shape?</p>
<p>As mentioned here <a href="https://discuss.pytorch.org/t/cnn-with-lstm-input-shapes/86910/9" class="inline-onebox">CNN with LSTM input shapes - #9 by ananda2020</a>, the following gets the required input shape of the LSTM</p>
<pre><code class="lang-auto">x = torch.randn(1, 2, 394, 1)
x = x.view(x.size(0), -1, x.size(3)) # [batch_size, features=channels*height, seq_len=width]
x = x.permute(2, 0, 1) # [seq_len, batch_size, features]
x.size()
</code></pre>
<p>returns</p>
<pre><code class="lang-auto">torch.Size([1, 1, 788])
</code></pre>
How to keep track of which model parameters are frozen?

yuhang_jiang: <p>I was using</p>
<pre><code class="lang-auto">for p in model.parameters():
    print(p)
</code></pre>
<p>to keep track of parameters. However, after trained the model, the unfrozen parameter values do not change.</p>

ptrblck: <p>You could check if these parameters were getting a valid gradient accumulated into their <code>.grad</code> attribute after the first <code>backward()</code> operation. If not, then the computation graph might have been detached.</p>
Pytorch code to approximate a sin function

joepareti54: <p><a href="https://pytorch.org/tutorials/beginner/pytorch_with_examples.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/beginner/pytorch_with_examples.html</a><br>
code in paragraph : nn module<br>
PyTorch: nn</p>
<p>This code only works for me after replacing:</p>
<p>loss_fn = torch.nn.MSELoss(reduction=‘sum’)</p>
<p>with:</p>
<p>loss_fn = torch.nn.MSELoss()</p>

ptrblck: <p>Could you describe what kind of error you are seeing if you don’t replace the <code>loss_fn</code>?</p>
Model for continuous outcome in [0, 1]

John_Grace: <p>My target variable is continuous in the [0, 1] interval, what the best model for this I can build in pytorch? I am wondering if there’s some form of logistic regression (with appropriate loss function) that works for this?</p>

InnovArul: <p>To keep the outcome within <code>[0,1]</code>, you can formulate any model with the final layer and the objective functions as follows:</p>
<ol>
<li>Final layer: <code>Sigmoid</code>, Objective function: <code>BCELoss</code>,</li>
<li>Final layer: <code>Linear</code>, Objective function: <code>BCEWithLogitsLoss</code>
</li>
</ol>