akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Layer Initialisation in multi-subjects multi labels classification problem

msuliman: <p>Hi Everyone,</p>
<p>I am solving a multi-subjects multi labels classification problem with a large number of subjects and labels. To explain my problem, I will use the image classification problem as an example.<br>
Say, within a single image, I have (M) subjects and (N) different labels per each subject. My final output layer is a softmax function that takes a matrix of dimensions (M by N) (where N &gt; M). Now, for the sake of the problem that I am solving, I need the first forward output (only the first!!) of the softmax to be an identity matrix (i.e., subject (i) is given label (i)). The way I am forcing this now is that I am introducing a linear layer before the softmax that gives (M * N) outputs, and I set the wights of this layer to be zero and the bias to be a reshaped version of the identity (in fact 10 * torch.eye(M,N)) which clearly grows very large as M and N increase. For small values of M and N, this idea is working perfectly; however, as M and N increase, I am running out of memory because of this inefficient way. For example, in my case, M =600 to 800, and N=2000 to 3000.</p>
<p>I am wondering if there is another efficient way to do that.</p>
<p>A sample code of the idea is as follows:</p>
<pre><code>class network(nn.Module): 
  def __init__(self, M,N, K, num_features):
    super(network, self).__init__()
    # M subjects
    # N labels per subject
    self.conv1 =  nn.Conv1d(num_features[0], num_features[1], kernel_size=1)  
    self.conv2 =  nn.Conv1d(num_features[1], num_features[2], kernel_size=1)

    self.fc1 = nn.Linear(num_features[2]*K, M)     
    self.fc2 = nn.Linear(M,M*N)
    
    # initialize the weights and bias of fc2 to ensure that the output of the softmax 
    # is the identity matrix at the first forward pass
    self.fc2.weight.data.zero_()
    self.fc2.bias.data.copy_((50)*torch.eye(M,N, dtype=torch.float).reshape(1,-1).squeeze(0))
    
    
    self.activation = nn.ReLU() 
    self.softmax = nn.Softmax(dim=1)

def forward(self, image):
    x = self.activation(self. conv1(image))
    x = self.activation(self.conv2(x))
    x = x.reshape(1,-1)
    x = self.fc1(x)
    x = self.fc2(x).reshape(M,N)
    mult_class = self.softmax(x)

    return mult_class
</code></pre>
<h1>
<a name="main-code-1" class="anchor" href="#main-code-1"></a>Main Code</h1>
<p>M=4<br>
N=3<br>
K=6<br>
num_features = [4,8,16]</p>
<p>image = torch.randn(1,num_features[0],K)<br>
net = network(M,N, K, num_features)<br>
mult_classes = net(image)</p>
Random.choices in pytorch

yegane: <p>Hi<br>
I want to choose random elements from a list with a weighting similar to np.random.choices, but I couldn’t find it in pytorch. Anyways, let’s call it T.<br>
Now, I want to check elements of N=1x256x256 and see any of them is equal to elements of T. If they were the same change them to 0, and if they weren’t change them to 255.<br>
I basically want to make a random mask.</p>
<p>I wondered if you could give me the right lines to do it in pytorch.</p>

ZimoNitrome: <p><a href="https://pytorch.org/docs/stable/generated/torch.multinomial.html#torch.multinomial" rel="noopener nofollow ugc"><code>torch.mulinominal()</code></a></p>
<aside class="quote quote-modified" data-post="14" data-topic="16146">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/phan_phan/48/51960_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/torch-equivalent-of-numpy-random-choice/16146/14">Torch equivalent of numpy.random.choice?</a> 
  </div>
  <blockquote>
    Hi ! 
To clear things up 
If you want to do the equivalent of  <a href="https://numpy.org/doc/stable/reference/random/generated/numpy.random.choice.html" rel="noopener nofollow ugc">numpy.random.choice</a>: 
a = np.array([1, 2, 3, 4])
p = np.array([0.1, 0.1, 0.1, 0.7])
n = 2
replace = True


b = np.random.choice(a, p=p, size=n, replace=replace) 

In pytorch you can use <a href="https://pytorch.org/docs/stable/generated/torch.multinomial.html#torch.multinomial" rel="noopener nofollow ugc">torch.multinomial</a> : 
a = torch.tensor([1, 2, 3, 4])
p = torch.tensor([0.1, 0.1, 0.1, 0.7])
n = 2
replace = True


idx = p.multinomial(num_samples=n, replacement=replace) 
b = a[idx] 

Careful, np.random.choice defaults to replace=True 
But torch.multi…
  </blockquote>
</aside>
What's mean of leading dimensions?

weizhen_song: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/c/ec7d27215e0ab1a1c3f87714784c5ed882dbe95e.png" data-download-href="https://discuss.pytorch.org/uploads/default/ec7d27215e0ab1a1c3f87714784c5ed882dbe95e" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/c/ec7d27215e0ab1a1c3f87714784c5ed882dbe95e.png" alt="image" data-base62-sha1="xK4OWUo4Lud1ugnCjpL1Muidrem" width="690" height="310" data-dominant-color="F6F3F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">873×393 24.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
I can’t know what’s mean of leading dimensions.<br>
who can help me</p>

ZimoNitrome: <p>It means any dimensions preceeding the ones listed. In this context it means that <code>Grayscale()</code> can process tensors such as:<br>
A single image with shape [3, H, W] (no leading dimensions)<br>
A batch of images of shape [B, 3, H, W] (here B is a leading dimension)<br>
A number of batches of shape [N, B, 3, H, W] (here N and B are leading dimensions)<br>
And the list goes on.</p>
Custom GhostBatchNorm with channels_last

Minh_Nguyen: <pre><code class="lang-auto">class GhostBatchNorm(nn.BatchNorm2d):
    def __init__(self, num_features, num_splits, **kw):
        super().__init__(num_features, **kw)

        running_mean = torch.zeros(num_features * num_splits)
        running_var = torch.ones(num_features * num_splits)

        self.weight.requires_grad = False
        self.num_splits = num_splits
        self.register_buffer("running_mean", running_mean)
        self.register_buffer("running_var", running_var)

    def train(self, mode=True, channels_last=False):
        if (self.training is True) and (mode is False):
            # lazily collate stats when we are going to use them
            self.running_mean = torch.mean(
                self.running_mean.view(self.num_splits, self.num_features), dim=0
            ).repeat(self.num_splits)
            self.running_var = torch.mean(
                self.running_var.view(self.num_splits, self.num_features), dim=0
            ).repeat(self.num_splits)
        return super().train(mode)

    def forward(self, input):
        n, c, h, w = input.shape

        if self.training or not self.track_running_stats:
            return F.batch_norm(
                input.view(-1, c * self.num_splits, h, w),
                self.running_mean,
                self.running_var,
                self.weight.repeat(self.num_splits),
                self.bias.repeat(self.num_splits),
                True,
                self.momentum,
                self.eps,
            ).view(n, c, h, w)
        else:
            return F.batch_norm(
                input,
                self.running_mean[: self.num_features],
                self.running_var[: self.num_features],
                self.weight,
                self.bias,
                False,
                self.momentum,
                self.eps,
            )
</code></pre>
<p>I’m trying to implement a GhostBatchNorm class with channels_last enabled; however, I kept experiencing the following error:<br>
<code>RuntimeError: view size is not compatible with input tensor's size and stride (at least one dimension spans across two contiguous subspaces). Use .reshape(...) instead.</code></p>
<p>I have checked the shape of input and all parameters (mean, var, etc…) but all of them seem to be the same with or without channels_last enabled (using <code>model.to(memory_format=torch.channels_last)</code>). Does anyone have a suggestion on what kind of problems I am facing? Thanks in advance.</p>

ptrblck: <p>I guess the <code>view</code> operation on the output of <code>F.batch_norm</code> might be raising the issue. If so, you should call <code>contiguous()</code> on the output before reshaping it to the channels-first shape.</p>

Minh_Nguyen: <p>Thank you very much for your help. Your suggestion was correct; I have to call <code>contiguous</code> for the input used inside <code>F.batch_norm</code> in order for the codes to work.</p>
<p>Do you know why using channels_last would require the use of <code>contiguous</code>? Thank you.</p>
Pytorch Equivalent of tensorflow code for tokenizing a sentence

manish-tripathi: <p>Hi I am new to deep learning…</p>
<p>I am trying to convert some Tensorflow code into Pytorch for practice</p>
<p>can you please help me understand what will be pytorch equivalent for the following code block:</p>
<pre><code class="lang-auto">def tokenize(lang):

  lang_tokenizer = tf.keras.preprocessing.text.Tokenizer(filters='')
  lang_tokenizer.fit_on_texts(lang)

  tensor = lang_tokenizer.texts_to_sequences(lang)

  tensor = tf.keras.preprocessing.sequence.pad_sequences(tensor,padding='post',maxlen=20,dtype='int32')

  return tensor, lang_tokenizer
</code></pre>
<p>the code i am trying to convert is the following:</p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/sunilbelde/neural-machine-translation-english-to-hindi-telugu/blob/main/NeuralMachineTranslation_eng_hindi.ipynb">
  <header class="source">

      <a href="https://github.com/sunilbelde/neural-machine-translation-english-to-hindi-telugu/blob/main/NeuralMachineTranslation_eng_hindi.ipynb" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/sunilbelde/neural-machine-translation-english-to-hindi-telugu/blob/main/NeuralMachineTranslation_eng_hindi.ipynb" target="_blank" rel="noopener nofollow ugc">sunilbelde/neural-machine-translation-english-to-hindi-telugu/blob/main/NeuralMachineTranslation_eng_hindi.ipynb</a></h4>


    <pre><code class="lang-ipynb">{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "NeuralMachineTranslation_eng_hindi.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
</code></pre>


  This file has been truncated. <a href="https://github.com/sunilbelde/neural-machine-translation-english-to-hindi-telugu/blob/main/NeuralMachineTranslation_eng_hindi.ipynb" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
How to fit a model using both function and derivative information?

Eduardo_Hernandez: <p>Hello. I am struggling with implementing a solution to the following problem using pytorch, so I wonder if I can get some feedback in this forum. Let us imagine I want to fit a 2D model M(x,y) (some unspecified NN model) to reproduce the values of a certain function f(x,y). I know how to do this in pytorch no problem; but now let’s imagine that as well as function values f(x,y), I can easily get df/dx and df/dy. Then, naturally one would think that it should be possible to use this additional information to better fit the model M, right?</p>
<p>It would seem that one could set the requires_grad flag to true for both x and y above to obtain the derivatives dM/dx and dM/dy after back propagation, and define the loss function so that the parameters of M reproduce not only the values of f(x,y), but also its derivatives with respect to the variables x and y. I would expect that proceeding in this way made the optimization of parameters probably faster and more robust.</p>
<p>The question is how does one implement this idea in pytorch? I have a clear understanding of the typical fitting cycle when one is using only function [f(x,y)] information, but I struggle to generalise this cycle when one has both function and derivative information. Can someone offer any guidance on how to proceed? Any simple examples?</p>
<p>Thanks!</p>

Eduardo_Hernandez: <p>Since I asked this, I have learned that what I am trying to do is called Sobolev training, and I have indeed found a couple of illustrative implementations using pytorch.</p>
<p>For those interested, see <a href="https://arxiv.org/abs/1706.04859" class="inline-onebox" rel="noopener nofollow ugc">[1706.04859] Sobolev Training for Neural Networks</a>, and two simple applications: <a href="https://github.com/Tony-Y/PotFit" class="inline-onebox" rel="noopener nofollow ugc">GitHub - Tony-Y/PotFit: A PyTorch Example of Potential Fitting</a>, <a href="https://gepgitlab.laas.fr/aparag/sobolev-experiments" class="inline-onebox" rel="noopener nofollow ugc">Amit Parag / Sobolev Experiments · GitLab</a>.</p>
Save tensor to txt file with full data precision

Ardeal: <p>Hi,</p>
<p>I could save tensor to txt file like this:<br>
<code>fp.write('module.weight.data == {}\n'.format(module.weight.data))</code><br>
however, the data in file use scientific value:<br>
<code> [[-1.1359e+00,  1.4675e+00,  2.6742e+00,  ..., -6.3853e-01,            -8.0900e-01, -1.1815e+00],           [ 5.4580e+00,  1.1344e+00, -6.1044e-01,  ..., -1.0242e-01,             6.5639e-01, -4.6325e-01],           [-3.2800e+00, -1.1235e+00, -9.9800e-01,  ..., -7.8122e-01,            -2.4848e-01, -9.8628e-01],</code></p>
<p>I would like to save the data with full precision.<br>
is threre any <code>simple way</code> to save full precision data to txt file?</p>

ptrblck: <p>Store the data in a binary format via <code>torch.save</code> to keep the full precision.<br>
If you really want to write the data to a text file, format the string. Something like <code>'{:.25f}'.format(tensor)</code> might work (assuming you don’t have values really close to zero).</p>

Ardeal: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> ,<br>
Thank you!</p>
<p>code:<br>
<code>fp.write('module.weight.data == {:.9f}\n'.format(module.weight.data))</code></p>
<ul>
<li>module is the layer(sub-class of nn.Module) of network.</li>
</ul>
<p>Error:</p>
<pre><code class="lang-python">Exception has occurred: TypeError       (note: full exception trace is shown but execution is paused at: hook_save_data_student)
unsupported format string passed to Tensor.__format__
</code></pre>

ptrblck: <p>It seems the tensor cannot be passed to the format directly (a scalar can), so a workaround could be:</p>
<pre><code class="lang-python">', '.join(['{:.25f}'.format(x_) for x_ in x])
</code></pre>
<p>However, I think grabbing the numpy array and using <a href="https://numpy.org/doc/stable/reference/generated/numpy.savetxt.html"><code>np.savetxt</code></a> might be the better idea.</p>
nn.TransformerEncoderLayer getting error after translated to pytorch_android_lite

yousanke: <p>Hello,</p>
<p>I am getting an error when running my transformer with pytorch_mobile_lite.</p>
<p>Here’s minimal code to get the error:</p>
<pre><code class="lang-auto">class TestModel(pl.LightningModule):
    def __init__(self, ):
        super(TestModel, self).__init__()
        self.model = TransformerEncoderLayer(128, 2, 512)

    def forward(self, src)-&gt;torch.Tensor:
        src_real = torch.randn(1, 80, 128)
        rtn = self.model(src_real)
        return rtn

model = TestModel()
s = script(model)
m = optimize_for_mobile(s)
m._save_for_lite_interpreter("model.ptl")`
</code></pre>
<p>After the torchscript is generated, replace the model of the <a href="https://github.com/pytorch/android-demo-app/tree/master/HelloWorldApp" rel="noopener nofollow ugc">HelloWorldExample</a>. Running the generated apk, I’ll get the following error from adb:</p>
<blockquote>
<pre><code>--------- beginning of crash
</code></pre>
<p>2021-10-21 01:03:32.470 12338-12338/org.pytorch.helloworld E/AndroidRuntime: FATAL EXCEPTION: main<br>
Process: org.pytorch.helloworld, PID: 12338<br>
java.lang.RuntimeException: Unable to start activity ComponentInfo{org.pytorch.helloworld/org.pytorch.helloworld.MainActivity}: com.facebook.jni.CppException: Following ops cannot be found. Check <a href="http://fburl.com/missing_ops" class="inline-onebox" rel="noopener nofollow ugc">Internal Login</a> for the fix.{aten::sqrt.int, } ()<br>
Exception raised from print_unsupported_ops_and_throw at …/torch/csrc/jit/mobile/import.cpp:206 (most recent call first):<br>
(no backtrace available)<br>
at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:3639)<br>
at android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:3796)</p>
</blockquote>
<p>I am running 1.9.0+cu111 with org.pytorch:pytorch_android_lite:1.9.0</p>
<p>Thank you.</p>

kimishpatel: <p>Are you using selective build? It seems somehow the aten::sqrt op is not built with mobile build so it is not able to find that operator.</p>

yousanke: <p>Hi kimishpatel,<br>
I suppose the pytorch 1.9.0+cu111 and org.pytorch:pytorch_android_lite:1.9.0 should match? I generate the torchscript with the code using the versions without manipulation.<br>
Should the unsupported operation be avoided by the torch.jit.trace itself? Or the users would avoid using them?<br>
Thank You,</p>

Linbin: <p>I can confirm that pytorch_android_lite:1.9.0 does not have this op (without selective build). But it works in the new 1.10 release. Just change the dependency to</p>
<pre><code class="lang-auto">implementation 'org.pytorch:pytorch_android_lite:1.10.0'
implementation 'org.pytorch:pytorch_android_torchvision_lite:1.10.0'
</code></pre>

yousanke: <p>I will be checking this in the following days, thanks for that.</p>

yousanke: <p>This is fixed in the pytorch_android 1.10.0<br>
Thank you guys for helping me.</p>
Hdf5 saved by pytoch can not be seen by h5dump tool

guen_gn: <p>I have one of the most classic 28x28 fashion mnist example which is working fine. Once I save the model (not the model state dict + opt. state dict), rather just by model.save(…, &lt;fileName.h5&gt;.<br>
Now I tried to view the resulting hdf5 but it says can not open:</p>
<pre><code class="lang-auto">root@nonroot-Standard-PC-i440FX-PIIX-1996:~/dev-learn/gpu/pytorch/port-effort-from-tflow-2nd# h5dump p297.h5
h5dump error: unable to open file "p297.h5"
</code></pre>
<p>The code example is in another of my post earlier:<br>
<a href="https://discuss.pytorch.org/t/pytorch-torch-nn-equivalent-of-tensorflow-keras-dense-layers/133518" class="inline-onebox">Pytorch torch nn equivalent of tensorflow (keras) dense layers?</a> 2nd from the bottom which I flagged as solution.</p>
<p>The only change I made was to add one liner:</p>
<pre><code class="lang-auto">...
acc = evaluate_model(valid_dl, model)
print('Accuracy: %.3f' % acc)

torch.save(model, "p297.h5")

# make a single prediction
print("Making prediction...")
...
</code></pre>
<p>By contrast, I have equivalent code in tensorflow which I can save the model as h5 after training and can view it just fine with h5dump tool.</p>

guen_gn: <p>any one can comment?</p>
Can we remove the input layer of a pre-trained model? Does it affect the weights of the model?

ibrahim_khan: <p>I have a pretrained model with input size batch size, 36, 2048, 3. I want to use this model 3 times. In first time I dont need to change the input, however, I need to change the input of the second instance of the model. I want to make it batch size, 36, 2048. Can we delete the input layers of the pre-trained model? Really appreciated for your help yesterday.</p>
How to find average length of audio files?

hema_rathore: <pre><code class="lang-auto">import librosa
import glob
import librosa

path=glob.glob('E:/...*/*.wav') 
for i in range(len(path)) :
    y, sr = librosa.load(path[i], sr=16000)
    z=librosa.get_duration(y)
</code></pre>

eqy: <p>What is the type of <code>z</code> here? Could you simply take an average?</p>

my3bikaht: <p>Your wav file is an array of [n] values. To calculate length of file in seconds you need to divide n by sr (sampling rate, number of measurements per second).</p>
<p>Be sure to provide sr value to librosa.load in case you have wavs from different sources because wavs can be recorded with different sampling rates and librosa will automatically resample them on load. Average len of y across all files, divide by sr.</p>
How can I give high weightage to one class based on training performance

Praveen_Joshi: <p>I am training a Resent-18 model on CIFAR- 10 dataset. But my model is failing to learn over label1 and 5 after certain epochs.</p>
<p>One can see the performance of the model on: <a href="https://wandb.ai/praveenjoshi007/sfl/reports/Shared-panel-21-10-27-22-10-26--VmlldzoxMTYyOTI3?accessToken=6einm83ach3zd2zsl2cowel8o2n7t9k9gg64bqs5coaoryql0a0qtsrzoegqucbk" rel="noopener nofollow ugc">https://wandb.ai/praveenjoshi007/sfl/reports/Shared-panel-21-10-27-22-10-26--VmlldzoxMTYyOTI3?accessToken=6einm83ach3zd2zsl2cowel8o2n7t9k9gg64bqs5coaoryql0a0qtsrzoegqucbk</a></p>
<p>Is there any function that can help make my model learn on the labels showing a high loss or having low classification accuracy?</p>

ptrblck: <p>You could try to estimate the weights for the current epoch based on the unreduced loss and recreate a weighted criterion in the next epoch.<br>
To get the raw, unreduced loss use <code>reduction='none'</code> while creating the criterion and check the losses for each sample in the current batch.</p>
Ongoing effort?: merge script::Module and nn::Module

swilson314: <p>I’m very interested in saving a model trained in c++ to onnx without having to travel through python. It seems like merging script::Module and nn::Module would allow this.</p>
<p>Per <a href="https://github.com/pytorch/pytorch/issues/14811" class="inline-onebox" rel="noopener nofollow ugc">torch::jit::trace C++ equivalent of torch.jit.trace · Issue #14811 · pytorch/pytorch · GitHub</a></p>
<pre><code>    We do plan to merge script::Module and nn::Module in a future release, however!
</code></pre>
<p>Is there a way that I can track the progress of this merge?</p>

JohnSidKim: <p>how about use serialize nn module like following codes.</p>
<p>torch::serialize::InputArchive archive_input;		archive_input.load_from(“D:\module.pt”, device.type());</p>
<p>and then, load archive_input</p>
CUDA out of memory even with DataParallel

Akshay_Goindani: <p>I was using 1 GPU and batch size was 64 and I got cuda out of memory. So I reduced the batch size to 16 to solve it. But when I am using 4 GPUs and batch size 64 with DataParallel then also I am getting the same error:</p>
<p>my code:<br>
device = torch.device(‘cuda’ if torch.cuda.is_available() else ‘cpu’)<br>
device_ids = os.environ[“CUDA_VISIBLE_DEVICES”].split(",")<br>
device_ids = [int(i) for i in device_ids]</p>
<p>encoder = EncoderRNN(input_size, hidden_size, SRC).to(device)<br>
decoder = DecoderRNN(hidden_size, output_size, TRG).to(device)<br>
model = nn.DataParallel(Model(encoder, decoder), device_ids = device_ids).to(device)</p>
<p>With DataParallel we can use multiple GPU and hence increase batch size.</p>

ptrblck: <p><code>nn.DataParallel</code> might use more memory on the default device as described in <a href="https://medium.com/huggingface/training-larger-batches-practical-tips-on-1-gpu-multi-gpu-distributed-setups-ec88c3e51255">this blog post</a>. We generally recommend to use <a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html">DistributedDataParallel</a> to avoid these issues and to get the best performance.</p>

Akshay_Goindani: <p>Should the device be cuda:0 or cuda.<br>
Also, after using multiple gpus and dataparallel, we should be able to increase the batch size as compared to a single gpu. In my case I am still not able to increase the batch size.</p>

ptrblck: <p><code>cuda:0</code> and <code>cuda</code> should refer to the same device.<br>
Could you check that all devices are empty before running the script?<br>
If that’s the case, I would recommend to try DDP.</p>

Olivier-CR: <p>could the PyTorch team fix it then? it’s important to have a single-process multi-device solution, especially as multi-GPU nodes get bigger and bigger, it’s less and less useful to do multi-node training. We need something simpler than DDP please <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

ptrblck: <p>Double post from <a href="https://discuss.pytorch.org/t/cuda-out-of-memory-error-when-using-multi-gpu/72333/9">here</a>.</p>
Is there a library or function to create a manuscript ready schematic of a pytorch model?

AlexSwiderski: <p>As the title suggests, I am wondering if anyone is aware of a library that will create  a schematic of your pytorch model?</p>
<p>This is a snip from google of Alexnet, or something similar:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/8/a8fe9f79db1b8e925990bd72760160cf8d05921b.jpeg" alt="image" data-base62-sha1="o6ZK10dNrnpOBKUE1bNCIxe1ell" width="553" height="221"></p>
Mixed-device models on mobile

beakin: <p>I have a complicated model I’m trying to run on mobile, with one component that is structurally simple but computationally expensive (deep convolutional structure), and the rest structurally complicated enough that there’s no way to run it with the ops supported on the <code>metal</code> backend currently.</p>
<p>Because of our deployment model it would be substantially preferable to have these as part of the same torchscript model object.  I know I can use a torchscript model as an internal component of a larger model that I then script, but when I freeze that outer model presumably the entire thing will be configured to use a common device.  Has anyone figured out how to run different components on different devices for mobile without separating those into different torchscript models?</p>
<p>Thanks!</p>
How should I do resolve this?

Dr_Aniket_Singha: <p>criterion = nn.CrossEntropyLoss()<br>
loss = criterion(output[0],y)</p>
<p>It says like this :<br>
RuntimeError: 1D target tensor expected, multi-target not supported</p>
<p>output[0] = tensor([[0.1302, 0.1725, 0.1690, 0.0921, 0.1245, 0.1362, 0.1755]]<br>
y = tensor([[0, 0, 0, 0, 1, 0, 0]])</p>

tom: <p>Your tensor looks like it is 1-hot encoded. The quickest is to take <code>y.argmax(-1)</code> but you might look into just providing the class indices.<br>
Also, your tensor suspiciously looks like probabilities (summing to one and all). Note that CrossEntropyLoss expects logits, so if you have softmax at the end, you might drop that.</p>
<p>Best regards</p>
<p>Thomas</p>
When to `torch.jit.freeze` for deployment?

Linux-cpp-lisp: <p>Hi all,</p>
<p>In the ideal PyTorch workflow from training to production deployment, where should one freeze the model? In particular, assume you are training a model that you compile to TorchScript and want to keep somewhere to use for a while into the future.</p>
<p>Should I <code>torch.jit.freeze</code> before saving the trained model, and always use the frozen saved model? Or should I save a <em>normal</em> TorchScript model, and <code>torch::jit::freeze</code> it when I load it in C++ for inference, re-freezing and optimizing it every I start an inference process? (Start up time is of no concern for my application, but backward compatibility is.)</p>
<p>Frozen models are clearly less flexible than unfrozen ones (<a href="https://github.com/pytorch/pytorch/issues/57569" class="inline-onebox" rel="noopener nofollow ugc">`torch.jit.freeze`'d models cannot be moved to GPU with `.to()` · Issue #57569 · pytorch/pytorch · GitHub</a>), but it is unclear to me whether:</p>
<ol>
<li>The optimizations they apply are system dependent. (Will I get better performance by freezing on the target system?)</li>
<li>Frozen models are still expected to be fairly future-proof, or if they are more specific to the setup on which they were frozen.</li>
</ol>
<p>Thanks.</p>

Elias_Ellison: <p>Hi!</p>
<p>Freezing optimizations should be system-independent. We introduced <code>optimize_for_inference</code> for system-dependent optimizations (does CUDNN exist and does it’s version work correctly with Conv-Add-Relu fusion, is MKLDNN installed). That one is still a little nascent - for now I’d just recommend using it with vision models.</p>
<ol start="2">
<li>Yes, frozen models are expected to be future-proof. Optimize for inference is not because it bakes in things wrt/the system so saving and loading it isn’t recommended / an intended use case.</li>
</ol>

Linux-cpp-lisp: <p>Hi <a class="mention" href="/u/elias_ellison">@Elias_Ellison</a> ,</p>
<p>Thanks for the quick answer— this is great, was not aware before of <code>optimize_for_inference</code>. Will keep an eye on it as it becomes relevant to a broader class of models.</p>
<p>To flip the question around, are TorchScript models that are <em>not</em> frozen just as well qualified for being archived for later use? (If I want to save them unfrozen and then freeze them “on-demand” before I use them.)</p>

beakin: <p>There shouldn’t be any downside to storing non-frozen torchscript models – they’re intended to be forward compatible as well.</p>

Linux-cpp-lisp: <p>Great, thanks for the quick answers!</p>
Regression for a naive example

MonkeyRn: <p>Hi,<br>
I got a problem when I tried a naive regression example on 4 points. Here is my code.</p>
<pre><code class="lang-auto">import numpy as np
import torch
from torch import nn
device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')

model = nn.Sequential(
    nn.Linear(D, H),
    nn.Sigmoid(),
    nn.Linear(H, C)
)
model.to(device) # Convert to CUDA
criterion = torch.nn.MSELoss()

optimizer = torch.optim.Adam(model.parameters(), lr=1e-3) # built-in L2
# Training
for t in range(10000):
    # Feed forward to get the logits
    y_pred = model(x)
    # Compute the loss (MSE)
    loss = criterion(y_pred, y)
    print("[EPOCH]: %i, [LOSS or MSE]: %.6f" % (t, loss.item()))
    display.clear_output(wait=True)

    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
</code></pre>
<pre><code class="lang-auto">D = 1 # or 2
H = 30 # for example
C = 1
#--------------------
x1 = torch.tensor(np.array([np.linspace(-1,1,4)]).T, dtype=torch.float).to(device)
y = torch.cos(1000*x1)

x2 = torch.tensor(np.array([np.linspace(-1,1,4), np.linspace(-1,1,4)]).T, dtype=torch.float).to(device)
y = torch.cos(1000*x2[:,0])
</code></pre>
<p>When I give X1 (in this case D=1), the loss converges to 0. But when I give X2 (which is just X1 repeated twice and in this case D=2), the solver get stuck and cannot be fitted correctly. I tried with SGD and some other learning_rates also but did not work. What is the issue ?</p>
<p>Thanks</p>

ptrblck: <p>In the second use case you are ignoring the warning:</p>
<pre><code class="lang-python">UserWarning: Using a target size (torch.Size([4])) that is different to the input size (torch.Size([4, 1])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.
</code></pre>
<p>which explains that broadcasting will be used. In your case the wrong loss calculation will thus be used and your training gets stuck. Unsqueeze <code>dim1</code> in <code>y</code> and it should work:</p>
<pre><code class="lang-python">x2 = torch.tensor(np.array([np.linspace(-1,1,4), np.linspace(-1,1,4)]).T, dtype=torch.float).to(device)
y = torch.cos(1000*x2[:,0])
y = y.unsqueeze(1)
</code></pre>

MonkeyRn: <p>It works now ! Thanks a lot !</p>
Constant time algorithm for torch.conj

Anand_Krish: <p>Hi,</p>
<p>So I just read the torch 1.10 release notes and came across the new constant time conjugate of complex tensors.</p>
<p>I would appreciate if you could help me find any information about this constant time algorithm.<br>
(PyTorch`s tweet about this release also mentions constant time transpose. Knowing about that will be great as well)</p>
<p>Thank you.</p>

anjali411: <p>Hi Anand! <code>.conj()</code> is now an O(1) operation because all it does is flips a bit on the tensor to indicate conjugation. However, the “materialization” of this lazy conjugation is still O(n).<br>
Example:</p>
<pre><code class="lang-auto">x = torch.randn(100, dtype=torch.cfloat)
y = x.conj() # O(1)
z = x + y # materialization of conjugation [O(n)] + addition [O(n)]
</code></pre>
<p>However, in some operations like matrix multiplication, we can fuse conjugation with the operation because the underlying library that’s being called provides the option to indicate conjugation, and that’s where you’d see a significant improvement in performance (due to fewer reads and writes) in many cases.</p>
How to create a dataloader for dictionary

Dr_Aniket_Singha: <p>Hi, I have a dictionary as dataset. How to use this as data in training a pytorch model</p>

beakin: <p>In general pytorch doesn’t really care what python structures you use to store your data.  The main caveat here is that when you create the dataloader you may need to specify a <code>collate_fn</code> which takes the individual samples and combines them into a batch.</p>
<p>See <a href="https://pytorch.org/docs/stable/data.html#dataloader-collate-fn" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/data.html#dataloader-collate-fn</a> for more details on how you define that.</p>
<p>Can write up a simple example later if that isn’t clear.</p>
Tensor.long() lose requires_grad

Kevin_Xue: <p>Hi,</p>
<p>i have a question regarding the tensor type conversion.<br>
I have a tensor <code>x</code>, which requires_grad, and it is computed from the leaf tensor, which is created by me.<br>
For some reason, i need the tensor <code>x</code> in type of <code>long</code>. So i applied <code>x = x.long()</code>. However, the resulting x doesn’t has requires_grad=True.</p>
<p>I wonder how i can propagate the requires_grad when i want to convert the tensor to type of <code>long</code> .</p>
<p>Appreciate for your help!</p>

ptrblck: <p>Gradients are only defined for floating point types and you would get an error if you try to enable it on another dtype:</p>
<pre><code class="lang-python">x = torch.randn(1, requires_grad=True)
x.long().requires_grad_()
&gt; RuntimeError: only Tensors of floating point dtype can require gradients
</code></pre>

Kevin_Xue: <p>Thank you for the reply.</p>
<p>Indeed, the <code>long</code> tensor cannot have gradient. Then, i wonder if we can make the differentiable indices for the function <code>image.index_put_(indices, values, accumulate=True)</code> ? The <code>indices</code> are the pixel location, which has requires_grad and is computed from the leaf tensor. <code>image</code> is a zero tensors. I want to write some values for specific pixels into image. Is it possible to allow a differentiable indices in this case?</p>

ptrblck: <p>I don’t think this would be possible, as</p>
<aside class="quote no-group" data-username="Kevin_Xue" data-post="3" data-topic="135218">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kevin_xue/48/43239_2.png" class="avatar"> Kevin_Xue:</div>
<blockquote>
<p>The <code>indices</code> are the pixel location, which has requires_grad and is computed from the leaf tensor.</p>
</blockquote>
</aside>
<p>would be disallowed, wouldn’t it?<br>
You could probably try to come up with a custom <code>autograd.Function</code>, which could define (somehow) the backward and gradients for indices although I wouldn’t have a good idea if and how this could work.</p>

Kevin_Xue: <p>Yeah, i get it. Thank you very much.</p>
Multi-output regression problem using Feed-forward Neural Network

wgu93: <p>Hello guys! I’m training a Feed-forward Neural Network (FFNN) with 11 inputs and 3 outputs for regression problem. The FFNN structure is simple, whose hidden layers consist of linear, ReLU (and BatchNorm). The problem is that three outputs do not have same scale, e.g., output 1 &amp; 2 will be within range [-0.1, 0.1] while output 3 will be [-0.001, 0.001]. Hence, when the MSE loss is backpropagated, the contribution of output 3 error will be neglected (too small) which leads to poor learning performance on output 3 (learning of output 1 &amp; 2 is good). I made some explorations and noticed that FFNN (specifically referred to my model, not to be confused with RBFNN that is also feedforward and mentioned below) does not guarantee good performance on multi-output regression in general, especially in cases where outputs are in different scales.</p>
<p>Well, there are some attempts/ideas I tried/own:</p>
<ol>
<li>I scaled the MSE losses for each output similar to methods mentioned <a href="https://discuss.pytorch.org/t/multi-output-model/49447/3">here</a>. It seems working sometimes, but anyways it requires a lot of tuning.</li>
<li>Some papers indicate that Radial Basis Function Neural Network (RBFNN) works better for such circumstances, I’ll try of course, but I’m more curious about how to solve this problem for my FFNN.</li>
<li>Training several FFNNs with single output instead of one with multiple outputs will solve this problem, but it is time-consuming and complicated in process. So, this will not be considered at present.</li>
</ol>
<p>Hope someone can share some thoughts/experience on multi-output regression using FFNN. Thanks in advance!</p>

googlebot: <p>you can just map your outputs to normalized values, and target these. in more complex cases, maybe do a whitening transformation instead of scalar scaling.</p>
Can help, what's wrong here

Fa_Faj: <p>Traceback (most recent call last):<br>
File “train.py”, line 58, in <br>
model.optimize_parameters()<br>
File “/content/gdrive/My Drive/SNE-RoadSeg-master/models/roadseg_model.py”, line 59, in optimize_parameters<br>
self.forward()<br>
File “/content/gdrive/My Drive/SNE-RoadSeg-master/models/roadseg_model.py”, line 50, in forward<br>
self.output = self.netRoadSeg(self.rgb_image, self.another_image)<br>
File “/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py”, line 493, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/usr/local/lib/python3.7/dist-packages/torch/nn/parallel/data_parallel.py”, line 150, in forward<br>
return self.module(*inputs[0], **kwargs[0])<br>
File “/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py”, line 493, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/content/gdrive/My Drive/SNE-RoadSeg-master/models/networks.py”, line 236, in forward<br>
x3_2 = self.conv3_2(torch.cat([x3_0, x3_1, self.up4_1(x4_1)], dim=1))<br>
File “/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py”, line 493, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/content/gdrive/My Drive/SNE-RoadSeg-master/models/networks.py”, line 94, in forward<br>
x = self.up(x)<br>
File “/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py”, line 493, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/usr/local/lib/python3.7/dist-packages/torch/nn/modules/upsampling.py”, line 131, in forward<br>
return F.interpolate(input, self.size, self.scale_factor, self.mode, self.align_corners)<br>
File “/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py”, line 2563, in interpolate<br>
return torch._C._nn.upsample_bilinear2d(input, _output_size(2), align_corners)<br>
RuntimeError: CUDA out of memory. Tried to allocate 60.00 MiB (GPU 0; 11.17 GiB total capacity; 10.34 GiB already allocated; 30.81 MiB free; 489.75 MiB cached)</p>

my3bikaht: <aside class="quote no-group" data-username="Fa_Faj" data-post="1" data-topic="135234">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/fa_faj/48/43362_2.png" class="avatar"> Fa_Faj:</div>
<blockquote>
<p>RuntimeError: CUDA out of memory. Tried to allocate 60.00 MiB (GPU 0; 11.17 GiB total capacity; 10.34 GiB already allocated; 30.81 MiB free; 489.75 MiB cached)</p>
</blockquote>
</aside>
<p>Your GPU is out of memory. You can decrease batch size to save some.</p>
Torchvision transformation toGrayScale requires Image?

Bagg: <pre><code class="lang-auto">torchvision/transforms/functional.py", line 743, in to_grayscale
    raise TypeError('img should be PIL Image. Got {}'.format(type(img)))
TypeError: img should be PIL Image. Got &lt;class 'torch.Tensor'&gt;
</code></pre>
<p>Hello, I understand the error. However, this should not happen. Transformation toGrayScale accepts both PIL images and Tensors according to the documentation. Also, this error only comes up on my machine and it doesn’t while running my script on Google Colab.</p>

ptrblck: <aside class="quote no-group" data-username="Bagg" data-post="1" data-topic="135143">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/bagg/48/40144_2.png" class="avatar"> Bagg:</div>
<blockquote>
<p>Also, this error only comes up on my machine and it doesn’t while running my script on Google Colab.</p>
</blockquote>
</aside>
<p>I guess your local setup might be using an older <code>torchvision</code> release, so you might want to update it.</p>

Bagg: <p>Thank you. My torchvision version is in fact outdated. However, I can’t update it for lack of permission. I found out that linalg from scipy somehow conflicted with my torchvision version. I swapped the import statements (FIST: scipy, SECOND: torchvision) and now it works.</p>

ptrblck: <p>Did the import switch “fix” the issue in the same (old) torchvision release? If so, which torchvision version are you using?</p>

Bagg: <p>For some reason, I was running torchvision 0.2.2.</p>
<p>Not it is updated to torchvision 0.11.1</p>
<p>The scipy problem raised later. Apparently I cannot do the following.</p>
<pre><code class="lang-auto">import torchvision
form scipy import linalg
</code></pre>
<p>I have to do this instead</p>
<pre><code class="lang-auto">from scipy import linalg
import torchvision
</code></pre>
<p>Apologies for the confusion. And thank you again <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

ptrblck: <p>I can’t reproduce the issue by swapping the import order in this code:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
# from scipy import linalg
import torchvision
from scipy import linalg

img = torch.randn(3, 224, 224)
transform = torchvision.transforms.Grayscale()
out = transform(img)
</code></pre>
<p>However, also note that the functional API op:</p>
<pre data-code-wrap="python"><code class="lang-python">torchvision.transforms.functional.to_grayscale(img, num_output_channels=1)
</code></pre>
<p>is not supporting tensors:</p>
<blockquote>
<p>Convert PIL image of any mode (RGB, HSV, LAB, etc) to grayscale version of image. This transform does not support torch Tensor.</p>
</blockquote>
<p>and you might want to use:</p>
<pre data-code-wrap="python"><code class="lang-python">torchvision.transforms.functional.rgb_to_grayscale(img: torch.Tensor, num_output_channels: int = 1)
</code></pre>
<p>instead</p>
Optimizer.step() on certain variables only

Sohrab_Samimi: <p>Hi everyone,</p>
<p>i have a question about optimizer.step().Let’s say i have a function g depending on a variable t for instance,and i want to parametrize g by a neural network.Let’s say there are only two parameters x and y,just to simplify the problem.Is it possible to use optimizer.step() in such a way that only the parameter y is updated after using loss.backward,and the parameter x is unchanged?</p>
SQLite => Custom Dataset

tchainzzz: <p>I have a massive dataset on disk (far too big for main memory) for which I’m trying to create a custom Dataset class. It is a SQLite database (the Reddit May2015 comments dataset if you’re familiar with that).</p>
<p>Unfortunately, as far as I can tell, the SQLite database lacks a primary key for some reason, making the <strong>getitem</strong> query non-trivial – otherwise I’d just do “SELECT col FROM table WHERE rowid=%s” or something. My current idea is to use Apache Spark (pyspark), and wrap it in a Spark DataFrame via JDBC, but this feels a little roundabout.</p>
<p>Does anyone have experience loading massive SQLite databases in PyTorch?</p>

SWHL: <p>You can refer to the <a href="https://github.com/austinvhuang/pytorch-sqlite" rel="noopener nofollow ugc">pytorch-sqlite</a></p>
Noob Pytorch Classification/Loss function question

Patrick1: <p>Hello guys,<br>
I have some super basic questions about pytorch LSTMs, as I am currently working on my first Machine Learning project. Sorry if my questions are too basic but I am really trying to understand what is happening here. Now to my question:</p>
<p>If I use the LabelEncoder for my targets, how does the Lossfunction compute the Loss? For example if I have a Binary Classification and I only train my model with one label called “Error”. My Model should tell me if the data I put in has this class error or not. Through LabelEncoder() my label “error” is equal to 0. Now my inputs for my Loss functions are (out, targets), which in my case is (0, 0.0645). What I now think to know is that the the output of my model is not a probability if my output function is Linear. I would need a Sigmoid activation  function but this is wrapped in the nn.BCEWithLogitsLoss().<br>
So does my model try to reach 0 as an output and whenever I put data in and the output is bigger then 0.5 it tells me the data is not of the class “error”?<br>
The big Problem I have is understanding how this then works with Multiclass Classification. My LabelEncoder then Encodes my “error temperature” (for example) as a 4. How does my Loss function then computes any loss between the out put of my model ( between 0 and 1) and my target (4).</p>
<p>While looking for answers I found One Hot Encoding, which I feel like would be better fitted but is not used for labelencodiing? Is that true? It is only used for feature encoding of features that are not ordinal related to eachother.</p>
Code works but the model doesn't train

mrswjung: <p>I trained a model</p>
<p>the code works but the model loss doesn’t get better.</p>
<p>the training loss looks like this.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/d/fd4a56d9534ed9f9008c226ae67e2300ceab997f.png" alt="image" data-base62-sha1="A8I4psDKNrNGYVm8d0n7S5Id0w7" width="625" height="432"></p>
<p>What is usually the reason when this type of problem occurs??</p>

my3bikaht: <p>With loss this big you need extremely small learning rate, otherwise your weights will change at crazy amounts. Seems like you are adding loss across samples/batches instead of averaging it.</p>
Questions about the memory function of backward()

lovepytorch: <p>Consider this situation, I combine network a and b’s parametrs into one optimizer, then for loss_a produced by a and loss_b produced by b, I do this:<br>
err = loss_a+loss_b<br>
err.backward()<br>
optimizer.step()</p>
<p>Is it same as:<br>
loss_a.backward()<br>
optimizer_a.step()<br>
loss_b.backward()<br>
optimizer_b.step()</p>
<p>Thanks.</p>

ptrblck: <p>If <code>loss_a</code> and <code>loss_b</code> are not sharing the computation graph (besides the final addition to <code>err</code> in the first approach) and if the optimizers (in the second approach) are also not sharing any parameters, I think the approaches should be equal. At least I don’t see how they could interact.</p>