akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Android GPU access, NNAPI vs Vulkan

username: julien-thierry
cooked: <p>Hi,</p>
<p>I noticed that on Android, the NNAPI pytorch backend might decide to dispatch some operations to the GPU:</p>
<pre><code class="lang-auto"> I ExecutionPlan: Device qti-default can't do operation RESIZE_NEAREST_NEIGHBOR
 I ExecutionPlan: Device qti-dsp can't do operation RESIZE_NEAREST_NEIGHBOR
 I ExecutionPlan: Device qti-gpu can't do operation RESIZE_NEAREST_NEIGHBOR
 I ExecutionPlan: Device qti-hta can't do operation RESIZE_NEAREST_NEIGHBOR
 I ExecutionPlan: ModelBuilder::findBestDeviceForEachOperation(RESIZE_NEAREST_NEIGHBOR) = 4 (nnapi-reference)
 I ExecutionPlan: Device qti-dsp can't do operation CONCATENATION
 I ExecutionPlan: Device qti-hta can't do operation CONCATENATION
 I ExecutionPlan: ModelBuilder::findBestDeviceForEachOperation(CONCATENATION) = 0 (qti-default)

</code></pre>
<p>My understanding is that for the concatenation operation, the qti-gpu will be used through qti-default. What are the benefits of using Vulkan over NNAPI and vice-versa for running inference on GPU?</p>
<p>Thanks,</p>
<p>Julien</p>

username: David_Reiss
cooked: <blockquote>
<p>What are the benefits of using Vulkan over NNAPI and vice-versa for running inference on GPU?</p>
</blockquote>
<p>The main difference is that with Vulkan, the code that runs on the GPU (shaders) ships as part of the PyTorch runtime.  With NNAPI, we give a high-level description of the model to the OS, which then sends it to the vendor backend for execution.  Right now, any updates to the backed require Android OS updates.</p>
<h2><a name="benefits-of-vulkan-1" class="anchor" href="#benefits-of-vulkan-1"></a>Benefits of Vulkan</h2>
<ul>
<li>Runs on Vulkan-enabled devices that don’t support NNAPI.</li>
<li>Shaders can be updated by updating PyTorch runtime, which means that bug fixes and performance improvements can roll out quickly.</li>
<li>New ops can be added in PyTorch updates, and soon custom ops will be addable by applications using PyTorch, without waiting for an OS update.</li>
<li>Soon, we’ll support an API for using Vulkan GPU tensors directly in your own shaders, so model output can be sent to rendering code without returning to CPU memory.</li>
</ul>
<h2><a name="benefits-of-nnapi-on-gpu-2" class="anchor" href="#benefits-of-nnapi-on-gpu-2"></a>Benefits of NNAPI on GPU</h2>
<ul>
<li>Vendor backend might be more mature and/or faster than the PyTorch Vulkan backend.</li>
<li>Might be a stepping stone to using NNAPI on DSP or NPU.</li>
</ul>

username: julien-thierry
cooked: <p>Hi,</p>
<p>Thanks for the detailed answer. It was very helpful.</p>
Libtorch -> how to use torch::gather to index using 2 tensors?

username: seunghyun_lee
cooked: <p>Hello!!<br>
I want to port " tensor[0,:,x,y] " in pytorch to c++ libtorch !<br>
I tried to use index_select or gather but I failed.<br>
Please give me a hint!!<br>
Thank you!</p>
TypeError: 'int' object is not subscriptable

username: sam101
cooked: <p>I am trying to create a data loader for my dataset. When I run it I am getting the above error.<br>
My code is as:</p>
<pre><code class="lang-auto">import torch
import pandas as pd
from torch.utils.data import DataLoader, Dataset
data = pd.read_csv("data.csv")
data = data.to_numpy()
data = data[:,1:].astype('float64')

class MyDataset(Dataset):
    def __init__(self, data, transform = None):
        self.data = data
        self.transform = transform

    def __getitem__(self, index):
        return self.data[index]

    def __len__(self):
        return len(self.data)


data = torch.tensor(data)
test = data
dataset  = MyDataset(data)
train_loader  = DataLoader(dataset)
</code></pre>
<p>When I run the below. I am getting the error<br>
" in __getitem__x = self.data[index], <strong>TypeError: ‘int’ object is not subscriptable</strong></p>
<pre><code class="lang-auto">for epoch in range(1000):
    loss_sum = 0.0
    for i, x in enumerate(train_loader):
        optimizer.zero_grad()
        loss = -model.log_prob(x.to(args.device)).mean()
        loss.backward()
        optimizer.step()
        loss_sum += loss.detach().cpu().item()
</code></pre>

username: pascal_notsawo
cooked: <p>Try to print the variable <code>data</code> to see its type (and its shape too): based on the error, it seems to be an integer and not an array.</p>

username: sam101
cooked: <p>I tried that and this is its output</p>
<pre><code class="lang-auto">&lt;class 'torch.Tensor'&gt;
torch.Size([600, 3000])
</code></pre>
Save model for resuming training

username: Yousra_Chahinez_HADJ
cooked: <p>I implemented a GAN model and because I need to train it for 500 epochs, I’ve saved the result of each 10 epochs for both models:</p>
<pre><code class="lang-auto">torch.save({
            'epoch': epoch + 1,
            'gen_state_dict': gen.state_dict(),
            'disc_state_dict': disc.state_dict(),
            'gen_optim': opt_gen.state_dict(),
            'disc_optim': opt_disc.state_dict(),
            }, os.path.join("", 'gan_epoch-{}.pt'.format(epoch + 1)))
</code></pre>
<p>and I load it:</p>
<pre><code class="lang-auto">disc = Discriminator(in_channels=3).to(device)
gen = Generator(in_channels=3).to(device)

checkpoint = torch.load("/content/drive/MyDrive/Epochs/gan_epoch-20.pt")

gen.load_state_dict(checkpoint['gen_state_dict'])
disc.load_state_dict(checkpoint['disc_state_dict'])

opt_disc.load_state_dict(checkpoint['disc_optim'])
opt_gen.load_state_dict(checkpoint['gen_optim'])

disc.train()
gen.train()
</code></pre>
<p>The code works well but I am wondering if the results will be correct, I have noticed that the training becomes faster, before saving the models, one epoch takes 20 minutes, now it takes only 8 minutes also the discriminator loss increases a lot from 0.xxx to 7.xxx is this normal?</p>

username: ptrblck
cooked: <p>Did you make sure the same datasets are loaded and contain the same number of samples?<br>
I would be suspicious, if the training suddenly sees a 2x speedup without changing anything else.</p>

username: Yousra_Chahinez_HADJ
cooked: <p>Thank you for your reply.<br>
Yes, I’m sure the same dataset is loaded, I ran it for the second time, and now it took 20 minutes, I’m really confused why this happened</p>
I want to apply label smoothin to MSE LOSS. Have you ever implemented it?

username: 439327bcc62cd8e45e96
cooked: <p>I want to apply label smoothin to MSE LOSS. Have you ever implemented it?</p>
<p>Please help me.</p>

username: attrapereves
cooked: <p>Never but I’d be curious to know so I follow the topic</p>

username: ptrblck
cooked: <p>How would label smoothing work for a regression use case?<br>
Or would you like to perform a classification using <code>nn.MSELoss</code>?</p>
Cannot create weak reference to 'numpy.ufunc' object

username: AfonsoSalgadoSousa
cooked: <p>I am trying to quantize a model that uses modules from <a href="https://github.com/qiuqiangkong/torchlibrosa" rel="noopener nofollow ugc">torch librosa</a>.<br>
I can quantize it and pass batches through the quantized model, but when trying to save it, the module LogmelFilterBank throws the following error:<br>
<code>cannot create weak reference to 'numpy.ufunc' object</code><br>
The source code for this module is the following:</p>
<pre><code class="lang-auto">class LogmelFilterBank(nn.Module):
    def __init__(self, sr=22050, n_fft=2048, n_mels=64, fmin=0.0, fmax=None, 
        is_log=True, ref=1.0, amin=1e-10, top_db=80.0, freeze_parameters=True):
        r"""Calculate logmel spectrogram using pytorch. The mel filter bank is 
        the pytorch implementation of as librosa.filters.mel 
        """
        super(LogmelFilterBank, self).__init__()

        self.is_log = is_log
        self.ref = ref
        self.amin = amin
        self.top_db = top_db
        if fmax == None:
            fmax = sr//2

        self.melW = librosa.filters.mel(sr=sr, n_fft=n_fft, n_mels=n_mels,
            fmin=fmin, fmax=fmax).T
        # (n_fft // 2 + 1, mel_bins)

        self.melW = nn.Parameter(torch.Tensor(self.melW))

        if freeze_parameters:
            for param in self.parameters():
                param.requires_grad = False

    def forward(self, input):
        r"""Calculate (log) mel spectrogram from spectrogram.

        Args:
            input: (*, n_fft), spectrogram
        
        Returns: 
            output: (*, mel_bins), (log) mel spectrogram
        """

        # Mel spectrogram
        mel_spectrogram = torch.matmul(input, self.melW)
        # (*, mel_bins)

        # Logmel spectrogram
        if self.is_log:
            output = self.power_to_db(mel_spectrogram)
        else:
            output = mel_spectrogram

        return output


    def power_to_db(self, input):
        r"""Power to db, this function is the pytorch implementation of 
        librosa.power_to_lb
        """
        ref_value = self.ref
        log_spec = 10.0 * torch.log10(torch.clamp(input, min=self.amin))
        log_spec -= 10.0 * np.log10(np.maximum(self.amin, ref_value))

        if self.top_db is not None:
            if self.top_db &lt; 0:
                raise librosa.util.exceptions.ParameterError('top_db must be non-negative')
            log_spec = torch.clamp(log_spec, min=log_spec.max().item() - self.top_db)

        return log_spec
</code></pre>
<p>I cannot understand where is the weak reference being thrown or how can I prevent it. Thanks for any help you can provide.</p>

username: jerryzh168
cooked: <p>can you show the code that saves this model as well?</p>

username: AfonsoSalgadoSousa
cooked: <p>Thanks for the reply. I fixed the problem with your suggestion. The problem was on trying to save the FP32 version of the model with <em>torch.jit.save</em>. Using <em>torch.save</em> fixed the problem.</p>
Object Detection with rotated bounding box

username: Nitesh
cooked: <p>Hello everyone, I have a custom dataset for text detection. Most of the text is tilted so does the bounding box around it. I have checked that many algos like YOLO, SSD work good with rectangular bounding box but haven’t seen any example where these algos are used to detect rotated bounding box.</p>
<p>Since most of the text in the given dataset has a rotated bounding box, should I adjust the coordinates of those bounding box to make them rectangular or should I regress all 4 coordinates of bounding box instead of width, height and center coordinate of the bounding box as used in YOLO.<br>
Kindly help me out here.<br>
Thanks</p>

username: jinfagang
cooked: <p>there are mainly 2 ways to do this:</p>
<ol>
<li>Regression bbox and an angle which decribe the rotation of the box;</li>
<li>Directly regression the 4 points as a polygon of the shape. these method have Centernet Cornernet etc can do so.</li>
</ol>

username: nightfury
cooked: <p>You can use an object detection algo which also predicts object orientation. Check out <a href="https://github.com/ivan-alles/localizer" rel="noopener nofollow ugc">Localizer</a>, this is a python library for object orientation detection.</p>
XLM/BERT sequence outputs to pooled outputs with weighted average pooling

username: Konstantin
cooked: <p>Let’s say I have a tokenized sentence of length 10, and I pass it to a BERT model.</p>
<pre><code class="lang-python">bert_out = bert(**bert_inp)
hidden_states = bert_out[0]
hidden_states.shape
&gt;&gt;&gt;torch.Size([1, 10, 768])
</code></pre>
<p>This returns me a tensor of shape: [<em>batch_size, seq_length, d_model</em>] where each word in sequence is encoded as a 768-dimentional vector</p>
<p>In TensorFlow BERT also returns a so called pooled output which corresponds to a vector representation of a whole sentence.<br>
I want to obtain it by taking a weighted average of sequence vectors and the way I do it is:</p>
<pre><code class="lang-python">hidden_states.view(-1, 10).shape
&gt;&gt;&gt; torch.Size([768, 10])

pooled = nn.Linear(10, 1)(hidden_states.view(-1, 10))
pooled.shape
&gt;&gt;&gt; torch.Size([768, 1])
</code></pre>
<ul>
<li>Is it the right way to proceed, or should I just flatten the whole thing and then apply linear?</li>
<li>Any other ways to obtain a good sentence representation?</li>
</ul>

username: Wherry154
cooked: <p>When building a sequence using special tokens, this is not the token that is used A *BaseModelOutputWithPoolingAndCrossAttentions (if return_dict=True is passed or softmax, used to compute the weighted average in the self-attention heads. head on top (a linear layer on top of the pooled output ) e.g. for GLUE tasks.</p>

username: Wherry154
cooked: <aside class="quote no-group quote-modified" data-username="Wherry154" data-post="2" data-topic="122363" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/w/22d042/48.png" class="avatar"> Wherry154:</div>
<blockquote>
<p>When building a sequence using special tokens, this is not the token that is used A *BaseModelOutputWithPoolingAndCrossAttentions (if return_dict=True is passed or softmax, used to compute the weighted average in the self-attention heads. <a href="https://www.navyarmyccu.ltd/https://www.navyarmyccu.ltd/" rel="noopener nofollow ugc">NavyArmyCCU Login</a> head on top (a linear layer on top of the pooled output ) e.g. for GLUE tasks.</p>
</blockquote>
</aside>
<p>Thanks for the information</p>
How to measure time of initializing CUDA context and loading model from CPU to GPU?

username: Zinuo
cooked: <p>I am focusing on an inference system with PyTorch. I find that the CUDA context initialization is always time-consuming, but I cannot find a tool to measure it. How can I solve the problem?</p>
<p>Also, I do not understand annotation <a href="https://pytorch.org/docs/1.7.0/notes/cuda.html#asynchronous-execution" rel="noopener nofollow ugc">here</a>.</p>
<blockquote>
<p>As an exception, several functions such as <code>to()</code> and <code>copy_()</code> admit an explicit <code>non_blocking</code> argument, which lets the caller bypass synchronization when it is unnecessary.</p>
</blockquote>
<p>Can I use the following code to measure the time of loading a trained model to GPU?</p>
<pre data-code-wrap="python"><code class="lang-python">start_event = torch.cuda.Event(enable_timing=True)
end_event = torch.cuda.Event(enable_timing=True)
start_event.record()

# ==========Start of Code==========
model = MyModel()
model.load_state_dict(...)
model.to(torch.device("cuda"))
# ==========End of Code==========

end_event.record()
torch.cuda.synchronize()  # Wait for the events to be recorded!
elapsed_time_ms = start_event.elapsed_time(end_event)
</code></pre>
Torchvision faster R-CNN model input arguments

username: yichong96
cooked: <p>Reading the pytorch documentation about faster RCNN  <a href="https://pytorch.org/vision/stable/models.html#faster-r-cnn" rel="noopener nofollow ugc">from this page</a>, It says that the input to the model during training is a list of dictionaries with the keys as ‘boxes’, ‘labels’.</p>
<p>However, when reading the <a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html" rel="noopener nofollow ugc">fine tuning tutorial for faster r-cnn</a>, there are additional arguments defined such as the ‘image_id’, ‘area’ and ‘iscrowd’ keys.</p>
<p>The documentation does not mention any of those keys. How do we resolve this discrepancy?</p>

username: tom
cooked: <p>The training only uses boxes and labels and ignores any other keys. As the tutorial says, the image_id, area and iscrowd are used in evaluation (but not during training).</p>
<p>Best regards</p>
<p>Thomas</p>
Pytorch Dataloader Sklearn PCA T-SNE

username: Stylianos_Charalambo
cooked: <p>Hello, I am trying to apply PCA and T-SNE on an image dataset. I have the images in a Dataloader but I can’t figure how to use them as the X on the sklearn functions. Can anyone help?</p>

username: ptrblck
cooked: <p>These scikit-learn methods expect 2-dimensional numpy arrays as the input, so you could either pass a batch returned by the <code>DataLoader</code> to it (after calling <code>numpy()</code> on it and making sure the shape is right) or you could stack (some) batches and pass this larger array to it.</p>
How to create a quadratic layer and use auto_grad

username: Lening_Li
cooked: <p>Hello everyone,</p>
<p>I am wondering if there is a way to create a network model where the network model includes a quadratic layer. Also, I want like to take advantage of the auto_grad.<br>
The quadratic function form is as the following: y = x^T A^T A X + p^T x + b</p>
<p>Any suggestions will be helpful. I really appreciate it.</p>
<p>Thanks,</p>
<p>Lening</p>

username: ptrblck
cooked: <p>You could create a custom <code>nn.Module</code> and execute the desired operation in its <code>forward</code> method.<br>
The parameters and buffers (if available) can be created in the <code>__init__</code>. <a href="https://pytorch.org/tutorials/beginner/examples_nn/two_layer_net_module.html">This tutorial</a> explains it in more detail.</p>
Attention Q*(Q+K).t()

username: vainaijr
cooked: <p>Why is</p>
<pre><code class="lang-auto">Q*(K).t()
</code></pre>
<p>( t() mean transpose) done in attention, and not</p>
<pre><code class="lang-auto">Q*(Q+K).t()
</code></pre>
<p>for example, if we have two pixels, black and white, and want to represent each combination of them differently.</p>
<pre><code class="lang-auto">black white -&gt; (QK)        white   black  -&gt; (KQ)           black  black  -&gt; (QQ)       white white -&gt; (KK)                                                                                    
                                              black -&gt; (Q)         white -&gt; (K)
</code></pre>
<pre><code class="lang-auto">Q*(K).t()
</code></pre>
<p>will give same result for</p>
<pre><code class="lang-auto">black white
</code></pre>
<p>and</p>
<pre><code class="lang-auto">white black
</code></pre>
<p>whereas if we do,</p>
<pre><code class="lang-auto">Q*(Q+K).t()
</code></pre>
<p>then four would be different, other options could be</p>
<pre><code class="lang-auto">Q*(Q-K)
</code></pre>
<p>but then</p>
<pre><code class="lang-auto">black black
white white
</code></pre>
<p>would be same, or</p>
<pre><code class="lang-auto">Q*K*K
</code></pre>
<p>, but that would be computationally expensive than</p>
<pre><code class="lang-auto">Q*(Q+K)
</code></pre>
<p>or</p>
<pre><code class="lang-auto">(Q+K)
</code></pre>
<p>but then,</p>
<pre><code class="lang-auto">black white
white black
</code></pre>
<p>would be same</p>
<p>or</p>
<pre><code class="lang-auto">(Q-K)
</code></pre>
<p>but then,</p>
<pre><code class="lang-auto">white white
black black
</code></pre>
<p>would be same</p>
<p>or only</p>
<pre><code class="lang-auto">Q
</code></pre>
<p>or only</p>
<pre><code class="lang-auto">K
</code></pre>
<p>but then all four would be same,</p>
<p>or concat Q and K together, but that would mean higher computation would be required to carry this operation again, as size increased.</p>

username: vainaijr
cooked: <p>how do I check whether this formula will give better results, what would be the easiest way to do this?<br>
Do I modify self attention in SEQUENCE-TO-SEQUENCE MODELING WITH NN.TRANSFORMER AND TORCHTEXT, this tutorial, or does any easier way to check performance of this formula exist?</p>

username: antgr
cooked: <p>You could check with seq2seq as you mentioned. Do you have any update on the results?</p>

username: antgr
cooked: <p>Maybe using the same seeds?</p>
Timeseries Model Structure

username: Arthur_Zakirov
cooked: <p>Hello everyone,</p>
<p>the question is more about deep learning, than about pytorch specifically.<br>
What is the best way to build a many-to-many timeseries model for numerical sequences of constant length p.e. Vehicle Trajectory?</p>
<p>If I take the last timestep of the encoder, like this…<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/1/f/1f0919d5f9efc9c0ca74023468a75a5f22f1021a.png" alt="image" data-base62-sha1="4qyiC2jhTyD9xesQ61ocmlyTtGO" width="221" height="338"></p>
<p>…what would be the best way to generate a sequence of multiple timesteps, using the last hidden state as the new input? I’ve seen differen versions like this one…<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/7/7/77f4843e211dfa0776b0d72092297627486515ab.png" alt="image" data-base62-sha1="h7aGZ8LRoUjik85lUcwhbJPq3vt" width="198" height="333"><br>
and this one …<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/c/c/cc0aa26581158c79539dc8e1cddebfa491fa622a.png" alt="image" data-base62-sha1="t72bECfAwjnwsZYm6v80xT6ulVU" width="196" height="321"><br>
Or a combination of the two, where you concat the last timestep of the encoder ouptut with every timestep of the decoder output…<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/1/4/1460c5708d3d3ec83b51fadd2344bdd82c061f07.png" alt="image" data-base62-sha1="2UgSBZW9vJUJ0kWuHJDsP9PIEVF" width="191" height="355"><br>
Feel free to suggest a different / better way.</p>
<p>Also, what is a better choice for labels?</p>
<ol>
<li>The absolute future values</li>
<li>The change per time of the future values → integral of the model output to get prediction</li>
</ol>
<p>I’m working on all of these variations right now. I was just curious if there is already a “go to” solution for my problem.</p>
<p>Thanks in advance,<br>
Arthur</p>

username: googlebot
cooked: <p>First decoder has weaker (bayesian) prior, i.e. potentially discardable starting context as sequence length grows. For models in later pictures, that’s achievable (much easier in gated rnns), but has to be learned. From other perspective, they have a shortcut connection to time zero, that may be beneficial if initial context is strongly informative.</p>
<p>Re: integral. I think you may face some issues with gradient flow, if you go that route and do things like cumsum(). Look into neural ODEs if you feel that learning changes is more suitable for your task, but they’re more complex and slower.</p>
Does DistributedDataParallel work with `torch.no_grad()` and `find_unused_parameters`=False

username: Ir1d
cooked: <p>HI, in my context I’m forwarding twice to obtain two results, and use one of them to guide the other. <a href="https://github.com/facebookresearch/suncet/blob/master/src/losses.py#L39" class="inline-onebox" rel="noopener nofollow ugc">suncet/losses.py at master · facebookresearch/suncet · GitHub</a></p>
<pre><code class="lang-python">target_supports, anchor_supports = encoder(simgs, return_before_head=True)
target_views, anchor_views = encoder(uimgs, return_before_head=True)
# then I use anchor_supports and anchor_views to calculate loss on https://github.com/facebookresearch/suncet/blob/master/src/losses.py#L65
</code></pre>
<p>I actually dont need the gradients of <code>anchor_supports</code>, so I added <code>with torch.no_grad()</code> like this:</p>
<pre><code class="lang-python">with torch.no_grad():
    target_supports, anchor_supports = encoder(simgs, return_before_head=True)
target_views, anchor_views = encoder(uimgs, return_before_head=True)
</code></pre>
<p>After that, ddp throws runtimeerror and asked me to set <code>find_unused_parameters=True</code>.</p>
<p>I tried setting it to true and it work, but I dont understand why thats necessary.</p>
<p>I also tried this and it worked with  <code>find_unused_parameters=False</code>.</p>
<pre><code class="lang-python">target_supports, anchor_supports = encoder(simgs, return_before_head=True)
target_views, anchor_views = encoder(uimgs, return_before_head=True)
anchor_supports = anchor_supports.detach()
target_supports = target_supports.detach()
</code></pre>
<p>Is there a way to both use <code>torch.no_grad()</code> to save memory and use <code>find_unused_parameters=False</code> to speed up?</p>

username: H-Huang
cooked: <p>There is not a way to use <code>torch.no_grad</code> and <code>find_unused_parameters=False</code> since these operations are still included in the autograd graph, but only require_grad is set to False. See the <code>find_used_parameters</code> argument for DDP <a href="https://pytorch.org/docs/master/generated/torch.nn.parallel.DistributedDataParallel.html#torch.nn.parallel.DistributedDataParallel" class="inline-onebox" rel="noopener nofollow ugc">DistributedDataParallel — PyTorch master documentation</a></p>
<p>When you are detaching <code>anchor_supports</code> and <code>target_supports</code> then the autograd graph will not be tracking those operations, which is why the example you gave is working with <code>find_unused_parameters=False</code>.</p>
Multiple image folder

username: aleemsidra
cooked: <p>I am working on a classification problem. I split my dataset into sub-folders according to class labels. Then, I used ImageFolder and DataLoader from Pytorch to load the data. I have attached my directory structure below.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/8/b8beb80519f32cd189ae0bb5a38eb4386e4a7f56.png" alt="dir_structure" data-base62-sha1="qmkt05UYbk2vTnx9Pheypf7xd9I" width="682" height="135"></p>
<p>My dataset does not have a train and test folder. Now, I am trying to do splitting in a way that all four sub-folders get split into train and test folders.</p>
<p>Most of the solutions I came across have train and test folders pre-hand with respective .csv files.  And use train_test split from sklearn.</p>
<p>Should I split data before splitting the entire data set it into image sub-folders based on class label?</p>

username: eqy
cooked: <p>It generally makes sense to split across class labels proportionally as splitting blindly across the entire dataset risks omitting smaller classes from either the training or testing set entirely.</p>
Custom input dimension

username: Kittu28
cooked: <p>How can i have the custom input dimension size for the input image  when working with any pytorch models ,to put it  more into simple words i want to change the input size which the models accepts rather than changing the sizes of my input images.</p>

username: eqy
cooked: <p>If you are doing classification most models should work for any input resolution barring memory/computational constraints as convolutional layers do not place any constraints on resolution and average pooling is applied before the fully connected layer(s). However, if you are using a pretrained model you may want to do finetuning if the object scales are very different than what would be expected at training time.</p>

username: Kittu28
cooked: <aside class="quote no-group" data-username="eqy" data-post="2" data-topic="122942">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/eqy/48/46233_2.png" class="avatar"> eqy:</div>
<blockquote>
<p>However, if you are using a pretrained model you may want to do finetuning if the object scales are very different than what would be expected at training time.</p>
</blockquote>
</aside>
<p><a class="mention" href="/u/eqy">@eqy</a> yes i want on pretrained model , i am new to all this so for example if i want it on resnet50 pre trained model how can i achieve it? Can you please elaborate a little .Thank you.</p>

username: eqy
cooked: <p>The imagenet example: <a href="https://github.com/pytorch/examples/blob/master/imagenet/main.py" rel="noopener nofollow ugc">examples/main.py at master · pytorch/examples (github.com)</a> is a good starting point. If you have an existing dataset, you should just be able to use a pretrained model and plug in your existing dataset with minimal changes (e.g., replace the fully connected layer with one that matches the number of classes you have).</p>
Torch fx supports TFLite quantization format

username: ralphmao
cooked: <p>I am quite impressed by the current fx-based quantization workflow. However, the hardware I am using only supports TFLite-style quantization. Will the TFLite-style quantization format be supported in the future? Otherwise if I want to implement a new quantization backend for TFLite, is there any useful documentation that helps me understand the to-dos?</p>

username: raghuramank100
cooked: <p>Unfortunately, we do not have any support for directly porting models from pytorch to tflite format. However, the underlying quantization support is very similar. We support quantization per-tensor or per-row, much like TFLite and have very similar operator support.</p>
Pad and pack sequences with various length for continuous prediction

username: fan_percy
cooked: <p>Hello, I’m now trying to implement a continuous prediction model using rnn/lstm. so the inputs are some multivariate sequences with different lengths and there’s a binary target correspond to each timestep in each sequence, so the predictions are made for every timestep in all sequences.</p>
<p>I’m a little bit confused with the pad_sequence and pack_padded_sequence functions, are they necessary in this case? if I pad the sequences, do I need to pad the labels as well?</p>
<p>any help is appreciated!</p>
Why would torch.inverse be slow?

username: adzpy
cooked: <p>Hi,</p>
<p>Sorry, I didn’t finish writing and accidentally clicked to post. Is there any particular reason why calculating the batched inverse of a (minibatches, channels, 3, 3) tensor would be ridiculously slow? For perspective, I have a modified Fast SCNN implemented which has one specific portion at the tail end where I perform least squares for every single channel in every single minibatch. My input to the least squares module should have minibatches*channels weight tensors from the earlier layers. I currently calculate the coefficients of a parabola using B = (X^T * X)^-1 * X^T * Y. The only bottleneck in this entire segment is where I calculate (X^T * X)^-1. Essentially, this is just calculating the inverse of one 3x3 matrix once for each channel in each mini batch. Yet, to put into perspective, the calculation of the inverses, this one line of the form torch.inverse(place), takes around 70% - 80% of the total forward pass. This is utilizing CUDA 10. Could this be because of overhead, or is there another issue?</p>

username: Yongjie_Shi
cooked: <aside class="quote no-group" data-username="adzpy" data-post="1" data-topic="52747">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/278dde/48.png" class="avatar"> adzpy:</div>
<blockquote>
<p>perspective, the calculation of the inverses, this one line of the form torch.inverse(place), takes around 70% - 80% of the total forward pass. This is utilizing CUDA 10. Could this be because of overhead</p>
</blockquote>
</aside>
<p>I have met the same problem, are there any suggestions about it?</p>
nonexcepted 5-dimensional input for 5-dimensional weight [32, 3, 1, 5, 5], b

username: saran
cooked: <p>RuntimeError: Expected 5-dimensional input for 5-dimensional weight [32, 3, 1, 5, 5], but got 4-dimensional input of size [3, 256, 128, 128] instead</p>

username: ptrblck
cooked: <p>Based on the error message it seems you are using an <code>nn.Conv3d</code> module, which expect a 5-dimensional tensor as the input in the shape <code>[batch_size, channels, depth, height, width]</code>, while one of these dimensions are missing (given the posted shape I guess it could be the channel dimension).</p>

username: saran
cooked: <p>thank you yes i think so i will check</p>
Custom loss with trainable parameters

username: Abdelrahman_Akram
cooked: <p>I want to make a loss module with trainable parameter, so I made the following:</p>
<pre><code class="lang-auto">class CostumLoss(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.gamma = torch.nn.Parameter(torch.FloatTensor([.5]))

    def forward(self, od_loss, depth_loss):
        print(self.gamma)
        loss = od_loss + self.gamma * depth_loss
        print(self.gamma)
        return loss
</code></pre>
<pre><code class="lang-auto">c_loss = CostumLoss()
c_loss.train()
total_loss = c_loss(od_loss , depth_loss)
total_loss.backward()
</code></pre>
<p>but printing the learnable parameter <code>gamma</code> tells that the parameter didn’t change.</p>

username: ptrblck
cooked: <p>The trainable parameters will be updated by the optimizer, once gradients were calculated in the <code>backward()</code> pass and <code>optimizer.step()</code> was called.<br>
In your current code snippet you are printing the value of <code>self.gamma</code> before and after using it in the <code>forward</code>, so it’s expected that these values weren’t changed yet.</p>

username: Abdelrahman_Akram
cooked: <p>Okay But I have two optimizers one for each loss ( there is two models but I trained them end to end ), so I think (please correct me) I won’t put the CustomLoss parameters in either of theme and I will make its own optimizer.</p>

username: ptrblck
cooked: <p>Yes, you could create new optimizers for these parameters, if wanted.</p>

username: Abdelrahman_Akram
cooked: <p>Thanks, I appreciate your help.</p>
Pipenv istall pytorch with cuda error

username: el_youssfi_azeddine
cooked: <p>Hello,<br>
I’m working on on a vertual envoronment and I have installed pipenv, but when I try to install pytorch 1.8.0 with cuda,with this command:<br>
<strong>pipenv install git+https://github.com/pytorch/pytorch#egg=pytorch</strong><br>
I get this error:<br>
<em>Installing git+https://github.com/pytorch/pytorch#egg=pytorch…</em><br>
<em>WARNING: Expecting value: line 1 column 1 (char 0)</em><br>
<em>Installation Failed</em><br>
thanks</p>

username: ptrblck
cooked: <p>I’m not sure, if this workflow would be supported, as I assume this would try to build the wheels from source. If that’s the intended use case, refer to <a href="https://github.com/pytorch/pytorch#from-source">this section</a> or alternatively install the wheels as described <a href="https://pytorch.org/get-started/locally/">here</a>.</p>

username: el_youssfi_azeddine
cooked: <p>thanks for the answer, I just uninstalled and installed wheels and it seems working well!</p>
How do I use cat in C++? (torch::cat)

username: Jesper.Lindberg
cooked: <p>Hello!<br>
How do I add my tensor <code>recurrent_input</code> at the end of my tensor <code>input</code>? <code>torch::cat</code> documentation seems to be missing? <a href="https://github.com/pytorch/pytorch/issues/35667" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/35667</a></p>
<p>Note that both are 1 dims tensors. 1x10 and 1x40</p>

username: Jesper.Lindberg
cooked: <p><code>torch::cat({ input, recurrent_input }, 0);</code></p>

username: Ajinkya_Ambatwar
cooked: <p>How to replicate this behaviour</p>
<pre><code class="lang-auto">&gt;&gt;&gt; t = torch.tensor([[1, 2, 3], [4, 4, 4]])                                                        
&gt;&gt;&gt; t
tensor([[1, 2, 3],
        [4, 4, 4]])
&gt;&gt;&gt; torch.cat(3*[t])
</code></pre>
<p>I want to know how to cat repeat</p>

username: ptrblck
cooked: <p>Your approach would work, so I’m unsure, if you are facing any issues with it.<br>
Alternatively, you could also try to use <code>tensor.expand</code> or <code>tensor.repeat</code> in case this would better fit your use case.</p>

username: Ajinkya_Ambatwar
cooked: <p>I am sorry I didn’t make my question clear.<br>
I want to know how to implement this in torch cpp.<br>
What’s the exact syntax for this?<br>
Thank you!</p>

username: ptrblck
cooked: <p>Ah, I didn’t realize the thread topic.<br>
<code>expand</code> and <code>repeat</code> are both available in <code>libtorch</code> e.g. via: <code>tensor.expand({3, 1})</code>.</p>
Why does nn.Conv2d require in_channels?

username: Shobhit_Verma
cooked: <p>I am new to PyTorch.<br>
I see that nn.Conv2d function takes as argument a pytorch tensor  <code>x</code>, and params as <code>in_channels</code>, along with others.</p>
<p>My question is that, since the function performs a 2D convolution, the <code>in_channels</code> would always be equal to the channel depth of the input tensor <code>x</code>, so why do we need to specify the same as a parameter?</p>
<p>Thanks</p>

username: Eta_C
cooked: <p>But the depth of the input tensor <code>x</code> is only known when called <code>forward</code>…<br>
So…forwarding and initializing network simultaneously?</p>

username: alan_ayu
cooked: <p>For defining the tensor of Conv2d’s parameters</p>

username: mailcorahul
cooked: <p>It is because nn.Conv2d in essence uses a 3d filter i.e filter_size x filter_size x input_channels. And as <a class="mention" href="/u/alan_ayu">@alan_ayu</a> pointed out, you need filter size, input channels and output channels to define Conv2d’s parameters.</p>

username: ptrblck
cooked: <p>Additionally to what was said, have a look at <a href="http://cs231n.github.io/convolutional-networks/#conv">CS231n</a> where the convolution operation including all shapes is well explained. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: AbdulsalamBande
cooked: <p>I believe every channel of let’s say an RGB image contribute to the overall properties of that image meaning all of the channels must be taken into account when performing the convolution process. Let’s say you just have one filter, a convolution process applies that filter into all of the 3 channels which will output 3 difference results of same dimension and those 3 results are added.</p>

username: Shobhit_Verma
cooked: <p>Thanks for your reply<br>
Can we not define the filter size at runtime? Is that against standard pytorch practice?</p>

username: 100deep1001
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a><br>
Is there any way we could mimic the Keras Conv2D layer in PyTorch without the need to specify in_channels?</p>

username: ptrblck
cooked: <p>Yes, you can use <a href="https://pytorch.org/docs/stable/generated/torch.nn.LazyConv2d.html#torch.nn.LazyConv2d"><code>nn.LazyConv2d</code></a>.</p>
Early Stopping in PyTorch

username: sumanth9
cooked: <p>Where can I find Early Stopping in PyTorch 0.4. Please point me ?<br>
Thanks</p>

username: ptrblck
cooked: <p><a href="https://github.com/ncullen93/torchsample/blob/master/README.md">Torchsample</a> provides an implementation of early stopping.</p>

username: sumanth9
cooked: <p>Is it available in 0.4 ?<br>
I am getting "ModuleNotFoundError: No module named ‘torchsample’ " error.<br>
I couldn’t find it in documentation either, please point me to the documentation if it is available.<br>
Thank you.</p>

username: ptrblck
cooked: <p>No, it was a standalone repo, but it seems to be abandoned.</p>
<p>EDIT: <a href="https://github.com/pytorch/ignite">Ignite</a> seems to have an implementation of early stopping <a href="https://github.com/pytorch/ignite/blob/master/ignite/handlers/early_stopping.py">here</a>.</p>

username: stefanonardo
cooked: <p><a href="https://gist.github.com/stefanonardo/693d96ceb2f531fa05db530f3e21517d" rel="nofollow noopener">Here</a> is my implementation, it should be easy to read and customize it.</p>

username: BjarteSunde
cooked: <p>I’ve implemented early stopping for PyTorch and made an example that shows how to use it; you can check it out <a href="https://github.com/Bjarten/early-stopping-pytorch" rel="nofollow noopener">here</a>.</p>

username: marwa
cooked: <p>Thank you for your script. It works well. I was wondering what if the loss validation is decreasing very slightly over epochs ( ie delta validation loss &lt; 1e-4 for instance).  Does the model stop learning in your case?</p>

username: BjarteSunde
cooked: <p>You are welcome! No, it does not have this functionality. Feel free to submit a pull request if you implement this feature.</p>

username: Gaurav_Koradiya
cooked: <p>useful and very easy to use.</p>

username: Hong_Cheng
cooked: <p>Is there any official implementation in pytorch1.x?</p>

username: Prashant_Bhat
cooked: <p>Hi, I have a suggestion for your implementation.</p>
<p>Wouldn’t it be more apt to store context as well along with the model during saving &amp; loading a general checkpoint for inference and/or resuming training?</p>
<pre><code class="lang-auto">torch.save({
            'epoch': epoch,
            'model_state_dict': model.state_dict(),
            'optimizer_state_dict': optimizer.state_dict(),
            'loss': loss,
            ...
            }, PATH)
</code></pre>

username: BjarteSunde
cooked: <p>Yes, I totally agree, feel free to submit a pull request if you implement this feature.</p>

username: jeffreyng99
cooked: <p>Just implemented this feature and made a pull request. Thank you for writing the original library, it definitely came in handy!</p>

username: dato_nefaridze
cooked: <p>can you please add setup.by  to your script so others can install it via pip?</p>
Loss function unexpected behaviour when passing learnable parameter in forward function

username: rajat-tech-002
cooked: <p>Accessing model learnable parameters using (model.variable_name.weight) and trying to update it with external function</p>

username: ptrblck
cooked: <p>Could you describe the issue you are facing a bit more, please?<br>
Do you get an unexpected error or is your workflow failing in any other way?</p>
High CPU Memory Usage

username: divyesh_rajpura
cooked: <p>When I run my experiments on GPU, it occupies large amount of cpu memory (~2.3GB). However, when I run my exps on cpu, it occupies very small amount of cpu memory (&lt;500MB). This memory overhead restricts me on training multiple models.</p>
<p>Can someone please help me on debug whuch component is causing this memory overhead?</p>
<p>I have added sample code at <a href="https://github.com/divyeshrajpura4114/asv-sample" class="inline-onebox" rel="noopener nofollow ugc">GitHub - divyeshrajpura4114/asv-sample</a></p>

username: tom
cooked: <p>This is likely the CUDA initialization. PyTorch comes with a relatively large number of kernels and CUDA does something with them on startup. This is particularly difficult on more constrained platforms like the Jetson.<br>
I thought of patching PyTorch to load all kernels through nvrtc instead of linking them into PyTorch but it is quite a bit of work and I was hoping someone else would fix things instead (with the recent split of libtorch_cuda, it seems people are digging in various directions there even if we’re not there yet).</p>
<p>Best regards</p>
<p>Thomas</p>

username: divyesh_rajpura
cooked: <p><a class="mention" href="/u/tom">@tom</a> Thanks for your response.</p>
<p>These are bit of new concepts for me. But, what I understand is that the CPU memory usage will be high whenever we train a model on GPU.</p>
<p>I have one more question. When I increase num_workers to 4 in DataLoader, each 4 process is taking high CPU memory (as I mensioned above ~2.3GB)?? Is it normal then?</p>

username: divyesh_rajpura
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> <a class="mention" href="/u/tom">@tom</a> , I have multiple GPUs and also have large CPU ram, but when I start 2 training simultaneously the response time is highly degraded because of memory consumption (may be its doing processing on cpu also at some extent). Is there any way to improve simultaneous trainings?</p>

username: tom
cooked: <p>You mean except buy more RAM? (This is only half-joking. I only have a single GPU but I chose to max out my computer’s RAM capacity (which is 128GB) - compared to GPU prices, that seems only reasonable. For many commercial situations “buy more RAM” might be the solution.)</p>
<p>More seriously: At least part of the memory usage is somewhat fundamental to how Python multiprocessing works and doesn’t work, combined with limitations of how CUDA works.<br>
But so one thing you can look into is to split part of the processing in the dataset to preprocessing and move other parts (i.e. augmentation) to the GPU and then get by with less processes for the dataloader.<br>
For real-world applications I have rarely seen a datapipeline that could not be drastically sped up with some tweaks.</p>
<p>Best regards</p>
<p>Thomas</p>
Saving weights using pickle or state_dict

username: 887574002
cooked: <p>Hi, I have a model that I need to save the weights during training (for example from 20 to 50 samples of weights in each 50 epochs) and in test time load the model and make inference using these weights. Then taking an average of these predictions. But when I use pickle I have this problem that when I load the weights, if I am in a different gpu then I would get the following error:</p>
<pre><code class="lang-auto">Attempting to deserialize object on CUDA device 2 but torch.cuda.device_count() is 2. Please use torch.load with map_location to map your storages to an existing device.
</code></pre>
<p>I save the weights using the following command:</p>
<pre><code class="lang-auto">weight_set_samples = []
weight_set_samples.append(copy.deepcopy(model.state_dict()))
</code></pre>
<p>and when training is finished I save the weights using the following command:</p>
<pre><code class="lang-auto">pickle.dump(net.weight_set_samples, model_dir+'/state_dicts.pkl', pickle.HIGHEST_PROTOCOL)
</code></pre>
<p>But when I use the following code to upload the weights:</p>
<pre><code class="lang-auto"> with open(model_dir+'/state_dicts.pkl','rb') as weights:
        
        weight_set_samples = pickle.load(weights)
</code></pre>
<p>I would run into a problem when the gpu during testing is not the same as the gpu during training.<br>
I read the documentation of pickle there wasn’t anything like <code>map-location</code> in pickle object that I can solve the problem.<br>
I would appreciate it if someone has any idea to solve this problem since the processes were long-term processes and it is really time consuming repeating the process.<br>
Second question is that how can I do that without pickle that I do not to run this problem. I want to save the weights as an array or list that it would be easy to load the model on these weights and making inference quickly.</p>

username: ptrblck
cooked: <p>The error is raised by PyTorch in its serialization module and the <code>map_location</code> argument can be specified in <code>torch.load</code>. I don’t know how <code>pickle</code> can be directly used to avoid this error, but would recommend to stick to <code>torch.save/load</code> and specify the <code>pickle_module</code>, if necessary.</p>

username: 887574002
cooked: <p>Ok, thank you for your answer.</p>
Running multiple modules, as single module

username: avish
cooked: <p>Hi, I have a ModuleDictionary where each module take similar size input, x.<br>
I have two use cases for it:</p>
<ol>
<li>For a single input x run all module in dictionary and create tensor by stacking all outputs.</li>
<li>Give a ordered map of {key, x} run each x to module in dictionary for that key and stack the outputs in that order. I can change input of this case to be batch of x and corresponding list of key.</li>
</ol>
<p>Currently I am doing this by looping on dictionary. Is there a better way to do this.</p>

username: ptrblck
cooked: <p>I think iterating the <code>dict</code> would be the right approach. If you are concerned about the performance of this loop, you could use a <code>dict</code> comprehension, which could yield a speedup, but I would recommend to profile the code first and check, if this loop is indeed the bottleneck in your code.</p>
Debugging `terminate called after throwing an instance of 'python_error'`

username: Xuechen_Li
cooked: <p>I’m using PyTorch’s new <code>register_full_backward_hook</code> and getting this error without no error message:</p>
<pre><code class="lang-auto">terminate called after throwing an instance of 'python_error'
  what():
Aborted
</code></pre>
<p>How do I debug this???</p>

username: ptrblck
cooked: <p>You could run the script in <code>gdb</code> and check the backtrace via:</p>
<pre><code class="lang-auto">gdb --args python script.py args
...
run
...
bt
</code></pre>
<p>This should point towards to failing operation.</p>