akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Load a 0.4.0 trained pytorch model in 1.2.0

kraken: <p>Is it possible to load a model trained on the earlier version of PyTorch (0.4.0) in the latest version(let say 1.2.0)?</p>

SimonW: <p>Yes, loading models is backward compatible.</p>

kraken: <p>Thanks for the response, can you guide me to a link which explains such a case</p>

SimonW: <p>You can just use the regular <code>torch.load</code> and <code>load_state_dict</code>. Are there specific modules you are looking for more details with?</p>

kraken: <p>Yes, when I load my model in 1.2.0 and inference it</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "inferencer.py", line 100, in &lt;module&gt;
    main(args)
  File "inferencer.py", line 96, in main
    seg_process(args, myModel)
  File "inferencer.py", line 72, in seg_process
    seg, alpha = net(inputs)
  File "/opt/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/jupyter/mobile_matting/model/segnet.py", line 290, in forward
    seg = self.seg_extract(x)
  File "/opt/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/jupyter/mobile_matting/model/segnet.py", line 191, in forward
    input_cascade1 = self.cascade1(input)
  File "/opt/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/jupyter/mobile_matting/model/segnet.py", line 98, in forward
    input = pool(input)
  File "/opt/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/opt/anaconda3/lib/python3.7/site-packages/torch/nn/modules/pooling.py", line 551, in forward
    self.padding, self.ceil_mode, self.count_include_pad, self.divisor_override)
  File "/opt/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 591, in __getattr__
    type(self).__name__, name))
AttributeError: 'AvgPool2d' object has no attribute 'divisor_override'
</code></pre>
<p>I get the following error, but when i roll back to 0.4.0, the model works fine</p>

SimonW: <p>Are you serializing the state dict or the module object?</p>

akschougule: <p>I got the same error. Model developed on PyTorch 1.0.1 and tested on 1.3.0. Any workaround?</p>
Integrate Kaldi with pytorch

Mohamed_Nabih: <p>Dear All;</p>
<p>I am running Kaldi ASR toolkit and fit MFCC features from speech Dataset and stored it in .ark, .scp and CMVN files, so how I can train my Network based on these files</p>
<p>Thanks</p>

ptrblck: <p>You could use a library like <a href="https://pypi.org/project/kaldiio/">kaldiio</a> to load these samples and create a custom <code>Dataset</code> and pass it to a <code>DataLoader</code> as explained in <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html">this tutorial</a>.</p>
<p>Once you have the <code>Dataset</code> ready, you could continue working with the architecture or your model.<br>
I’m not completely sure how the data is stored, but since you are dealing with MFCC data, I assume you could treat it as “image” data?</p>

vincentqb: <p><a href="https://pytorch.org/audio/" rel="nofollow noopener">torchaudio</a> supports <a href="https://pytorch.org/audio/kaldi_io.html" rel="nofollow noopener">ark and scp</a>, offers <a href="https://pytorch.org/audio/transforms.html#mfcc" rel="nofollow noopener">MFCC</a>, and also has a template for <a href="https://pytorch.org/audio/datasets.html" rel="nofollow noopener">datasets with DataLoader</a> <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Static libtorch (libtorch.a)

Maziar: <p>I am trying to compile the code in <a href="https://pytorch.org/tutorials/advanced/cpp_frontend.html" rel="nofollow noopener">https://pytorch.org/tutorials/advanced/cpp_frontend.html</a>. The code uses dynamic libtorch library but I would like to use the static library.</p>
<p>Does anyone know where to download static libtorch?</p>
Video inference using glow

Udit_Kumar: <p>Hi Guys,<br>
I am new to this world of glow,<br>
I can use ffmpeg or gstreamer to do video decode and infernence.<br>
I<br>
want to use glow for the same. Could you help me, if there is some sample code/docs,  to explain, How video decode and inference could be achieved using glow.</p>
<p>Thanks<br>
Udit</p>

jfix: <p>Hi <a class="mention" href="/u/udit_kumar">@Udit_Kumar</a>,</p>
<p>Thanks for your interest! Do you have an initial starting point, e.g. some ONNX model that you want to run? And do you have a specific architecture you’d like to run on?</p>
<p>We have some documentation on how to run models such as image classification <a href="https://github.com/pytorch/glow/blob/master/docs/Testing.md#image-classification" rel="nofollow noopener">here</a>. However this is not going to satisfy your desire for video specifically. We do intend to improve our video based operators and coverage over the next half year or so.</p>
<p>Thanks,<br>
Jordan</p>
NaN values in torch.where's unselected branch still break gradients

angelolovatto: <p>Hi all,</p>
<p>I’m using <code>torch.where</code> to avoid backpropagating through branches that might yield NaN values. Specifically, I want to exponentiate a number if it’s nonnegative and do some other stuff otherwise (as exponentiating a negative number may yield imaginary numbers).</p>
<p>However, when backpropagating from the loss, the resulting gradient is still NaN, even though the loss is the desired one. Here’s a minimal working example. One can use a <code>nullcontext</code> instead of the autograd anomaly detection to check that <code>a.grad</code> is NaN.</p>
<pre><code class="lang-python">a = torch.as_tensor(-0.1).requires_grad_()
with torch.autograd.detect_anomaly():
    r = torch.where(a &lt; 0, a * 10, a ** 1.8)
    r.backward()
</code></pre>
<pre><code class="lang-python">In [77]: r
Out[77]: tensor(-1., grad_fn=&lt;SWhereBackward&gt;)
</code></pre>
<p>Any ideas on how to circumvent this? Is this considered a bug?</p>
<p>Thanks in advance.</p>

albanD: <p>Hi,</p>
<p>You can see github issues like this one that discuss this: <a href="https://github.com/pytorch/pytorch/issues/15506">https://github.com/pytorch/pytorch/issues/15506</a></p>
<p>Hope this helps.</p>
Detach() followed by requires_grad_()?

pytorchuser96: <p>Recently I have been reading a book called Deep Learning with Pytorch and there is a line of code that I don’t quite get it. Here is the function for training:</p>
<blockquote>
<pre><code>def training_loop(n_epochs, learning_rate, params, t_u, t_c):
    for epoch in range(1, n_epochs + 1):
        if params.grad is not None:
            params.grad.zero_()
        t_p = model(t_u, *params)
        loss = loss_fn(t_p, t_c)
        loss.backward()
        params = (params - learning_rate * params.grad).detach().requires_grad_()

        if epoch % 500 == 0:
            print('Epoch %d, Loss %f' % (epoch, float(loss)))
    return params
</code></pre>
</blockquote>
<p>I was confused by the line params = (params - learning_rate * params.grad).detach().requires_grad_(). Why do we need to first .detach() and .requires_grad_() again?</p>

albanD: <p>This can be used to create a leaf Tensor that requires grad. Which means that if it is used in computations and we then call <code>.backward()</code>, its <code>.grad</code> field will be populated.</p>
BERT classification - same output for every datapoint?

AlexisW: <p>I modified the code <a href="https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/examples/run_classifier.py" rel="nofollow noopener">here</a> for a two-class classification task. However, the raw probability in evaluation stage seems identical for all data points. Is there something I may have a mistake?</p>

Borel: <p>Sorry for the bump, but i’m facing the very same issue with a bert pre-trained model. I’m also using eval() and it’s just when i try to predict.</p>
<p>Here’s my post <a href="https://discuss.pytorch.org/t/bert-huggingface-model-gives-me-constant-predictions/67312/2">link</a>. Any help is welcome.</p>
Two different loss function within a batch

a-parida12: <p>I have concatenated two different dataloaders in pytorch. My aim is to minimize <code>loss1</code> for samples coming from <code>dataloader1</code> and minimize <code>loss2</code> for samples coming from <code>dataloader2</code>.</p>
<p>What would be the ideal/ best methods to do it in pytorch?</p>

richard: <p>What methods have you tried so far? Are you trying to minimize the losses at the same time? If so you might want to define some combined loss, say, <code>loss1 + loss2</code>.</p>

a-parida12: <p>My ultimate aim is to minimise <code>loss1</code> for some samples and <code>loss2</code> for other samples.</p>
<p>I tried minimising <code>a*loss1+b*loss2</code> for samples from dataloader 1 <code>a=1,b=0</code> and samples from dataloader 2 <code>a=0,b=1</code>. I think this is an hacky solution and wanted know if some special provisions in pytorch exist.</p>
Can't get source for site-package

jim_gitonga: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/d/dd3587ff8657205846742a903e9b6583dc906ee5.png" data-download-href="https://discuss.pytorch.org/uploads/default/dd3587ff8657205846742a903e9b6583dc906ee5" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/d/dd3587ff8657205846742a903e9b6583dc906ee5.png" alt="image" data-base62-sha1="vyUd5zGykkwyMQ9tjPxxBZ94H2J" width="690" height="54" data-dominant-color="191919"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">944×74 4.41 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Can I create libtorch c++ project using qt with qmake?

savior: <p>Can I create libtorch c++ project using qt with qmake?</p>

MSA: <p>Hi, did you find an answer?<br>
Thanks,</p>

AlconDivino: <p>I never personally used it but if you link the libtorch.so and libc10.so according to the conventions of qmake and include the header search path it should be possible.</p>

MSA: <p>Yes, that works well.<br>
I tried with Qt and cmake, I didn’t find how to compile the project so I switched to qmake.</p>
<p>Win10:</p>
<pre><code class="lang-auto">QT += core gui concurrent

greaterThan(QT_MAJOR_VERSION, 4): QT += widgets

CONFIG += c++17

QMAKE_CXXFLAGS += -DGLIBCXX_USE_CXX11_ABI=0
CONFIG += no_keywords

DEFINES += QT_DEPRECATED_WARNINGS

#
#DEFINES += QT_DISABLE_DEPRECATED_BEFORE=0x060000    # disables all the APIs deprecated before Qt 6.0.0

SOURCES += \
    main.cpp \
    
HEADERS += \
   

win32:CONFIG(release, debug|release): LIBS += -L/libtorch-win-shared-with-deps-1.4.0/libtorch/lib/ -ltorch -lc10
else:win32:CONFIG(debug, debug|release): LIBS += -L/libtorch-win-shared-with-deps-1.4.0/libtorch/lib/ -ltorchd -lc10d

INCLUDEPATH += libtorch-win-shared-with-deps-1.4.0/libtorch/include
DEPENDPATH += libtorch-win-shared-with-deps-1.4.0/libtorch/include

INCLUDEPATH += $/libtorch-win-shared-with-deps-1.4.0/libtorch/include/torch/csrc/api/include
DEPENDPATH += $/libtorch-win-shared-with-deps-1.4.0/libtorch/include/torch/csrc/api/include
</code></pre>
GANimorph does not learn

merail: <p>Hello.</p>
<p>I am trying to implement GANimorph (<a href="https://arxiv.org/abs/1808.04325" rel="nofollow noopener">https://arxiv.org/abs/1808.04325</a>) on Pytorch. If possible, did everything as indicated in the article. However, I ran into a problem: losses in the learning process change very weakly, noise is generated all the time. What could be the problem? I didn’t train for long, maybe I need to wait? When implementing other GANs, the result was noticeable almost immediately.</p>
<p>Sorry for my bad English, I’m russian student.</p>
Is there any function in pytorch that equivalent to scipy.linalg.cho_solve()?

Kiki_Rizki_Arpiandi: <p>Is there any function in PyTorch equivalent to scipy.linalg.cho_solve() ?</p>

richard: <aside class="quote no-group" data-username="Kiki_Rizki_Arpiandi" data-post="1" data-topic="67753">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kiki_rizki_arpiandi/48/10715_2.png" class="avatar"> Kiki_Rizki_Arpiandi:</div>
<blockquote>
<p>cho_solve</p>
</blockquote>
</aside>
<p><a href="https://pytorch.org/docs/stable/torch.html#torch.cholesky_solve" class="inline-onebox">torch — PyTorch 2.1 documentation</a> should be what you’re looking for.</p>
Penalize generation of vector outside a uniform distribution

chenna: <p>I want to create a loss to penalize when a vector z (generated by network) is generated outside a uniform distribution [-1, 1].</p>
<p>I suggested the following solution:</p>
<p>relu = torch.nn.ReLU()<br>
loss = torch.sum(relu(torch.abs(z)-1))</p>
<p>Is this a proper way? is there any built-in loss function to do this?</p>
Numerous compiler warnings

mboedigh: <p>I didn’t see this posted, but we all must see that there are numerous compiler warnings from c10, aten and others (at least on windows). The c++ api tests work, so I guess these are harmless, but it makes finding warnings and errors in my code challenging. Since I’m not a pro, can I ask in this forum about some possible edits that I would like to make to get rid of these?<br>
For example:<br>
all the Half operators have the form</p>
<pre><code class="lang-auto">inline C10_HOST_DEVICE Half operator op(Half a, int b)
{
 return a op static_cast&lt;Half&gt;(b);
}
</code></pre>
<p>This seems to construct a Half object by calling <code>Half(b).</code> But, there is no Half constructor taking an int. As a result b is implicitly cast to float: <code>Half((float)b)</code>. This causes a C4244 warning conversion from ‘int’ to ‘float’, possible loss of data</p>
<p>I’m not sure the authors intent, but would this cause any problems to other parts of the library (or is there a better fix)? For example can I cast b to an unsigned short directly without causing problems?</p>
<pre><code class="lang-auto">inline C10_HOST_DEVICE Half operator op(Half a, int b)
{
 return a op static_cast&lt;Half&gt;( static_cast&lt;float&gt;b );
}
</code></pre>

peterjc123: <p>Please read this post.<br>
<aside class="onebox stackexchange">
  <header class="source">
      <a href="https://stackoverflow.com/questions/2541984/how-to-suppress-warnings-in-external-headers-in-visual-c" target="_blank" rel="nofollow noopener">stackoverflow.com</a>
  </header>
  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/129976/bob-whiteman" target="_blank" rel="nofollow noopener">
    <img alt="Bob Whiteman" src="https://www.gravatar.com/avatar/504ccb9786fccf04b4bb877147913474?s=128&amp;d=identicon&amp;r=PG&amp;f=1" class="thumbnail onebox-avatar" width="60" height="60">
  </a>
<h4>
  <a href="https://stackoverflow.com/questions/2541984/how-to-suppress-warnings-in-external-headers-in-visual-c" target="_blank" rel="nofollow noopener">How to suppress warnings in external headers in Visual C++</a>
</h4>

<div class="tags">
  <strong>visual-c++, suppress-warnings, brew-framework, brewmp</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stackoverflow.com/users/129976/bob-whiteman" target="_blank" rel="nofollow noopener">
    Bob Whiteman
  </a>
  on <a href="https://stackoverflow.com/questions/2541984/how-to-suppress-warnings-in-external-headers-in-visual-c" target="_blank" rel="nofollow noopener">11:43PM - 29 Mar 10 UTC</a>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>

mboedigh: <p>Thanks. Based on the that post, this worked to suppress all warnings</p>
<pre><code class="lang-auto">#pragma warning(push, 0)
#include &lt;torch/torch.h&gt;
#pragma warning(pop)
</code></pre>
Face detection in real time with MTCNN ? 'NoneType' object has no attribute 'size'

hunar: <p><a href="https://stackoverflow.com/questions/59935966/face-detection-in-real-time-with-mtcnn-nonetype-object-has-no-attribute-siz" rel="nofollow noopener">face detection in real time with MTCNN ? ‘NoneType’ object has no attribute ‘size’</a></p>
Custom torchtext.data problem

Hou_ZeYu: <p>I already read the tutorial about <a href="https://pytorch.org/tutorials/beginner/text_sentiment_ngrams_tutorial.html#load-data-with-ngrams" rel="noopener nofollow ugc">TEXT CLASSIFICATION WITH TORCHTEXT</a>.<br>
I want to do my custom text classification task, but there are some different between <strong>torch.data</strong> and <strong>torchtext.data</strong>.<br>
take a look in the tutorial.</p>
<blockquote>
<p>import torch<br>
import torchtext<br>
from torchtext.datasets import text_classification<br>
NGRAMS = 2<br>
import os<br>
if not os.path.isdir(‘./.data’):<br>
os.mkdir(‘./.data’)<br>
train_dataset, test_dataset = text_classification.DATASETS[‘AG_NEWS’](<br>
root=‘./.data’, ngrams=NGRAMS, vocab=None)<br>
BATCH_SIZE = 16<br>
device = torch.device(“cuda” if torch.cuda.is_available() else “cpu”)</p>
</blockquote>
<p>In these example, datasets are load after download.<br>
If I have a custom data look like “train.csv”.<br>
This “train.csv” is a table, have two columns.<br>
One of column name is “text”, containing text.<br>
One of column name is “label”, containing my target.</p>
<div class="md-table">
<table>
<thead>
<tr>
<th>text</th>
<th>label</th>
</tr>
</thead>
<tbody>
<tr>
<td>this is an apple</td>
<td>0</td>
</tr>
<tr>
<td>NBA is started today</td>
<td>1</td>
</tr>
<tr>
<td>We are family</td>
<td>2</td>
</tr>
</tbody>
</table>
</div><p>How to write a standard torchtext.data class in these case?</p>

sai_m: <p>We have to start with declaring fields<br>
from torchtext.data import Field, LabelField<br>
TEXT = Field(tokenize=word_tokenize) # converting strings to int<br>
LABEL = LabelField(dtype=torch.float)</p>
<p>Then construct dataset, this step will tell fields what data to work on</p>
<p>datafields = [(“text”, TEXT), (“labels”, LABEL)]<br>
trn, tst = torchtext.data.TabularDataset.splits(path = ‘Data’,train = ‘train.csv’,<br>
test = ‘test.csv’,<br>
format = ‘csv’,<br>
skip_header = True,<br>
fields = datafields)</p>
<p>After you follow these steps, you have your datasets ready, next steps would be building vocab using TEXT.build_vocab(trn)</p>
<p>Then you will define your batch size and construct iterators.</p>
<p>Hope this will help!!</p>

zhangguanheng66: <p>I think it’s very similar to the text classification datasets in torchtext. You should be able to use most of the building blocks there. The only part to change is the <code>_csv_iterator</code> function <a href="https://github.com/zhangguanheng66/text/blob/a5880a3da7928dd7dd529507eec943a307204de7/torchtext/datasets/text_classification.py#L36-L39" rel="nofollow noopener">here</a>. In the original datasets, labels come before the text. But in your case, the text comes before the labels.</p>
<p>We suggest to use the new abstraction (text classification datasets follow the new abstraction) because it’s more compatible with <code>torch.data.utils</code>. Feel free to open issues on Github if you have any questions.</p>
How to forward input through some layers of a pretrained model and not the others?

Ilkin: <p>Hi,</p>
<p>So, I am wondering how I can pass input through select layers of a model and not the others. For example, I have a pretrained UNet, but I am only interested in the output of each layer of the encoder before downsampling. To get the intermediate layer outputs, I use hooks, but how do I make sure that after the input passes through all encoder layers, the network stops does not waste time decoding the encoded output. The solution I have right now is to create a separate model called UNet_Encoder with the same layer names as the original UNet and then load the weights in. I wonder if there is a less hackish way to overcome this issue.</p>

ptrblck: <p>I think deriving another class from your original <code>UNet</code> and changing the <code>forward</code> method accordingly is a valid approach.<br>
If you know about this use case before writing the initial model, you could also separate the encoder and decoder in submodules and use a passed flag in the <code>forward</code> method to use different branches.</p>

Ilkin: <p>Thanks for such a quick response! It’s clear now, I just wanted to make sure there isn’t a more efficient way of doing this.</p>

Shisho_Sama: <p>There are many ways to do it. ptrblk kindly suggested one of them.<br>
You can also do use your networks children() method , get the modules you want, add them to a nn.ModuleList(), and then send them to a nn.Sequential() and thats it.<br>
you have your model.</p>
Data loader that gives two batches per iteration?

f3ba: <p>I have a model where the training logic requires looking at two different batches in each iteration. How can I use <code>DataLoader</code> for this?</p>

G.M: <p>simply take 2 batches:</p>
<pre><code class="lang-auto">batch_1=next(data_loader)
batch_2=next(data_loader)
</code></pre>

f3ba: <p>Here is the code snippet I ended up using:</p>
<pre><code class="lang-auto">train_dl_cp = copy.deepcopy(train_dl)
for epoch in range(epochs):
    it = iter(train_dl_cp)
    for b, x1 in enumerate(train_dl):
        x2 = next(it)
        .... # do something with x1 and x2
</code></pre>
<p>I think this should be fine.</p>
Will Pytorch shift to Swift from Python or make it support Swift?

JonathanSum: <p>I have watched Fastai’s videos. It looks like Swift is a great language, and it can really speed up like 3 times faster or even more than Pytorch. Swift can use Python code and C++.</p>

ptrblck: <p>The speedup sounds interesting!<br>
Could you post the comparison you are referring to?<br>
Is this “plain Swift code” vs. “plain Python” or are other libraries used e.g. numpy?</p>

Shisho_Sama: <p>I believe its mostly advertisement and early hype for swift4TF! nothing new.<br>
its still in its infancy it seems and has a long way to go, in order to get to the level in which Python or languages such as C/C++ are in deep learning.<br>
But I guess having it wouldnt hurt. early adoption could be a good thing in case it really makes life easier without much compromises!</p>

JonathanSum: <p>Right, when I have time. I will do a VGG19 pre-trained in style transfer in Swift vs Pytorch.<br>
I think I have mistaken that it was plain Python that was not even vectorized.</p>
Dataset/Dataloader to Load Batches per Class

RevoGen: <p>My dataset is in the format</p>
<pre><code class="lang-auto">root/
    1/
        1.png
        2.png
        3.png
        ...
    2/
        1.png
        2.png
        3.png
        ...
    ...
</code></pre>
<p>I need to be able to load batches containing images only from a specific class, for each class.<br>
Eg. (1st batch are 8 images from root/1, 2nd batch are 8 images from root/2, so on)</p>
<p>Is there any way to do so without having multiple datasets, each for a particular class?</p>
<p>Any help is appreciated.</p>

Idodo: <p>If I understand you correctly, the ImageFolder class does just that:</p>
<p><a href="https://pytorch.org/docs/stable/torchvision/datasets.html#imagefolder" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/docs/stable/torchvision/datasets.html#imagefolder</a></p>

RevoGen: <p>But is it possible to specify which class to sample from?</p>

Idodo: <p>Not with the ImageFolder object. But you could perhaps create an ImageFolder object for each class to be able to sample independently from each class.</p>
Kivy error: sdl2 - ImportError: DLL load failed: The specified module can not be found

Saike13: <p>Code:<br>
import kivy<br>
from kivy.app import App<br>
from kivy.uix.label import Label</p>
<p>class MyApp(App):<br>
def build(self):<br>
return Label(text=“Hello world!”)</p>
<p>if <strong>name</strong> == “<strong>main</strong>”:<br>
MyApp().run()</p>
<p>Error:<br>
sdl2 - ImportError: DLL load failed: The specified module can not be found.<br>
File “C:\Users\Usuario\AppData\Local\Programs\Python\Python37-32\lib\site-packages\kivy\core_<em>init</em>_.py”, line 63, in core_select_lib<br>
fromlist=[modulename], level=0)<br>
File “C:\Users\Usuario\AppData\Local\Programs\Python\Python37-32\lib\site-packages\kivy\core\window\window_sdl2.py”, line 27, in <br>
from kivy.core.window._window_sdl2 import _WindowSDL2Storage</p>

ptrblck: <p>Hi Saike,</p>
<p>you’ve landed in the PyTorch discussion board, while your question seems to focus on Kivy.<br>
I would recommend to create a topic in their forum or on StackOverflow to get support. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
Dataset / Dataloader "best practices" with DataParallel

filippobistaffa: <p>I’m training a simple LSTM model using two GPUs (2 x GeForce RTX 2080 Ti) via <code>DataParallel</code>:</p>
<pre><code class="lang-auto">class MyLSTM(nn.Module):
    def __init__(self, input_dim, hidden_dim):
        super(MyLSTM,self).__init__()
        self.input_dim = input_dim
        self.hidden_dim = hidden_dim
        self.LSTM = nn.LSTM(input_dim, hidden_dim, batch_first=True)
        self.LNN = nn.Linear(hidden_dim, input_dim)

    def forward(self, i):
        self.LSTM.flatten_parameters()
        o, _ = self.LSTM(i)
        return self.LNN(o)
</code></pre>
<pre><code class="lang-auto">model = nn.DataParallel(model)
model.cuda()
</code></pre>
<p>Given the size of my input and the amount of available memory, I found that I can use a maximum batch size of 28. Each sequence is a tensor of 1440 * 3969, hence each batch should be a tensor of 28 * 1440 * 3969.</p>
<p>What are the best practices about loading the input data to the GPUs? By reading around a bit, it seems that one should use <code>Dataset</code> and return CPU tensors and <code>DataLoader</code> with <code>'pin_memory' = True</code>, then during training copy the batch to the GPUs with <code>batch.to('cuda', non_blocking=True)</code>.<br>
With this approach I get a processing rate of ~0.352 batches per second.</p>
<p>On the other hand, by returning CUDA tensors directly from the <code>Dataset</code> (hence avoiding pinned memory at all), I get a higher processing rate of 0.427 batches per second.</p>
<p>How is that a apparently “sub-optimal” approach yields a higher processing rate?<br>
Am I doing something wrong, or is it related to the (relatively) small batch size?</p>

ptrblck: <aside class="quote no-group" data-username="filippobistaffa" data-post="1" data-topic="67708">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/filippobistaffa/48/7875_2.png" class="avatar"> filippobistaffa:</div>
<blockquote>
<p>How is that a apparently “sub-optimal” approach yields a higher processing rate?<br>
Am I doing something wrong, or is it related to the (relatively) small batch size?</p>
</blockquote>
</aside>
<p>Your approach might have a slower initialization time, but could be faster during the training.<br>
The main drawback is that you most likely load the complete dataset onto the device, which will use (some) GPU memory which will thus not be available for your model.<br>
Also, preprocessing each sample might not be that easy, e.g. since a lot of image processing transformations from <code>torchvision</code> rely on <code>PIL.Image</code>, which uses numpy arrays under the hood and thus needs CPU arrays.</p>
<p>If that’s not the case and your dataset is “quite” small, your approach is valid.</p>
<p>PS: We also recommend to use <a href="https://pytorch.org/docs/stable/nn.html#distributeddataparallel"><code>DistributedDataParallel</code></a> (as Multi-Process Single-GPU), which should be the fastest approach.</p>
What does it mean to 'Ensure that modules are not reused ' in quantization?

Shisho_Sama: <p>In Quantization Workflows explained in the <a href="https://pytorch.org/docs/stable/quantization.html#quantization-workflows" rel="noopener nofollow ugc">documentations here</a>, there is a listing describing how to go about quantizing a network in which the 2nd item says:</p>
<blockquote>
<p>Post Training Static Quantization: This is the most commonly used form of quantization where the weights are quantized ahead of time and the scale factor and bias for the activation tensors is pre-computed based on observing the behavior of the model during a calibration process. Post Training Quantization is typically when both memory bandwidth and compute savings are important with CNNs being a typical use case. The general process for doing post training quantization is:</p>
<pre><code>1.Prepare the model:
         a. Specify where the activations are quantized and dequantized explicitly by adding QuantStub and DeQuantStub modules. 
         b. Ensure that modules are not reused.
         c. Convert any operations that require requantization into modules
 2. ...
 3. etc
</code></pre>
</blockquote>
<p>What I’m refering to here, is the b. part which says :</p>
<pre><code class="lang-auto">b. Ensure that modules are not reused.
</code></pre>
<p>What does this mean exactly? is this refering to the QuantStub and DeQuantStub attributes? or not?<br>
I mean, Can I simply do :</p>
<pre><code class="lang-auto">input1 = self.quant(input1)
input2 = self.quant(input2)
input3 = self.quant(input3)
</code></pre>
<p>and later on do :</p>
<pre><code class="lang-auto">output1 = self.dequant(output1)
output2 = self.dequant(output2)
output3 = self.dequant(output3)
</code></pre>
<p>if it is, then how are we supposed to incorporate multiple input/multiple output for quantization/dequantization?</p>
<p>if not! then I dont understand exactly what this means and would greatly appreciate if anyone can shed some l ight on this .</p>
<p>By the way, I would also appreciate it, if anyone could clarify what this '97 mean in the following section (which is the item 3 of the same list I just asked about):</p>
<blockquote>
<ol start="3">
<li>Specify the configuration of the quantization methods ‘97 such as selecting symmetric or asymmetric quantization and MinMax or L2Norm calibration techniques.</li>
</ol>
</blockquote>
<p>Thanks a lot in advance</p>
Any best pratice for debuging distributed pytorch training script?

lepton: <p>Is there any best practice that you could debug a distributed training script <code>train.py</code> step by step, for example,</p>
<pre><code class="lang-auto">python -m torch.distributed.launch --nproc_per_node=2 train.py --input1 --input 2
</code></pre>
<p>I was using the Pycharm as IDE. Your help will be appreciated.</p>
Backward pass is extremely slow

sayan_ghosal: <p>Hi,<br>
I am running a pytorch module that involves multiple cond1d layers, contranspose1d layers, and module_list of linear layers. And the number of subjects in each batch is of the order of 10 (total samples = 100).<br>
It is taking an incredible amount of time for each epoch. When I ran the code with torch.util.profiler I got the following result.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/c/2c2d8c472fc901ecc1307b44bcb193313d50b75f.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/2c2d8c472fc901ecc1307b44bcb193313d50b75f" title="Screen Shot 2020-01-26 at 11.14.23 AM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/c/2c2d8c472fc901ecc1307b44bcb193313d50b75f_2_690x461.jpeg" alt="Screen Shot 2020-01-26 at 11.14.23 AM" data-base62-sha1="6iOASf9zYn9bin1O5wGGQF7IPDN" width="690" height="461" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/c/2c2d8c472fc901ecc1307b44bcb193313d50b75f_2_690x461.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/2/c/2c2d8c472fc901ecc1307b44bcb193313d50b75f_2_1035x691.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/2/c/2c2d8c472fc901ecc1307b44bcb193313d50b75f_2_1380x922.jpeg 2x" data-dominant-color="37192F"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2020-01-26 at 11.14.23 AM</span><span class="informations">2034×1360 965 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I’m not sure what’s going wrong. Also, I don’t really understand the function named “to” in the autograd profiler. Any help is really appreciated.<br>
Thank you.</p>

ptrblck: <p>It seems you are using the CPU for the processing?<br>
How many cores and which CPU are you using?<br>
You could try to use MKL-DNN as described <a href="https://discuss.pytorch.org/t/use-mkldnn-in-pytorch/54943/4">here</a> for a potential speedup.</p>
Image Transformation inside the forward function of our network

Ankur_Raj: <p>Hello,</p>
<p>Can we apply image transformation inside the forward function of our network?<br>
I want to apply this transformation in the forward definition of my network.</p>
<pre><code class="lang-auto">self.transforms = transforms.Compose([transforms.ToPILImage().convert('RGB'),transforms.Resize(90),
                                              transforms.CenterCrop(70),transforms.ToTensor()])

</code></pre>
<p>I am using a pre-trained model(resnet50) and changing the last fully connected to my needs.</p>
<p>In the case when I apply transformation outside the network(transform PIL ie resize, center-crop, then Tensor) my network works.<br>
But, for the case when I covert my input image to tensor and then inside my network try to convert it back to PIL and apply the operations. It doesn’t work.<br>
I receive this error, even if I have applied to unsqueeze operation.</p>
<pre><code class="lang-auto">Expected 4-dimensional input for 4-dimensional weight 64 3 7 7, but got 3-dimensional input of size [3, 640, 960] instead
</code></pre>
<p>If I try to unsqueeze my tensor before entering the network, I get “CUDA memory full” error.</p>
<p>My Network Definition :-</p>
<pre><code class="lang-auto">class Network(nn.Module):
    
    def __init__(self):
        super().__init__()
        self.model = models.resnet50(pretrained=True)
        for param in self.model.parameters():
            param.requires_grad = False
        self.model_in_fetures = self.model.fc.in_features
        self.model.fc = nn.Sequential(nn.Linear(self.model_in_fetures,1000),nn.ReLU(),nn.Dropout(0.5),
              nn.Linear(1000,250),nn.ReLU(),nn.Dropout(0.5),
              nn.Linear(250,50),nn.ReLU(),nn.Dropout(0.5),
              nn.Linear(50,8))
        self.transforms = transforms.Compose([transforms.ToPILImage().convert('RGB'),transforms.Resize(90),
                                              transforms.CenterCrop(70),transforms.ToTensor()])

        
    def forward(self, x):
        
        x = self.transforms(x)
        x = x.unsqueeze(0)
        return self.model(x)
</code></pre>

ptrblck: <p>Could you try to reduce the batch size and rerun the code again?<br>
Your current transformation will most likely not work for these issues:</p>
<ul>
<li>
<code>transforms.ToPILImage()</code> does not have a <code>convert</code> method, which is a method of <code>PIL.Image</code>
</li>
<li>Transformations for <code>PIL.Images</code> work on single images, so you would need to loop your input tensor as:</li>
</ul>
<pre><code class="lang-python">x = torch.randn(2, 3, 224, 224)
out = []
for x_ in x:
    out.append(transform(x_))
out = torch.stack(out)
</code></pre>
What are the equations it uses model(x) manually?

rerere_L: <p>I have extracted bias and weight from trained model.</p>
<pre><code class="lang-auto">print('X = ',X[0,:])
print(type(pinv(Wy)),pinv(Wy).shape)
Wy.detach().numpy()
h1 = torch.matmul(Wy.T,y2_pred-by)
k1 = torch.tensor([math.tanh(h1[i]) for i in range(len(h1))] ) ; 
x1 = torch.matmul(Wx.T,k1-bx) ; print('X1 = ',x1)
</code></pre>
<p>but give this result:</p>
<pre><code class="lang-auto">X =  tensor([-0.7951, -0.0736,  0.8450,  0.8444,  0.8469, -0.9261, -0.9261, 12.0541])
X1 =  tensor([ 0.4630, -0.3566,  0.8700,  1.3055,  0.0930, -0.2514, -0.1384,  0.0268]
</code></pre>
<p>Where is the problem and what are all equations for reproduce the model(x) command.??<br>
Im using:</p>
<ul>
<li>One hidden layer</li>
<li>activation function = tanh</li>
<li>x=3   h= 6   y = 8</li>
</ul>

albanD: <p>Your code tries to use <code>atanh</code> instead of <code>tanh</code> no?</p>

rerere_L: <p>i change with correction</p>

albanD: <p>Do you have the code that defines the model? In particular the <code>__init__</code> and <code>forward</code> functions.</p>
Setting learner parameters with metalearner doesn't backprop to metalearner

sabetai: <p>I’d like to set the parameters() of a model to the output of another model without breaking the autograd graph. A naive approach like</p>
<p><code>for p in model.parameters():     p = nn.Parameter(t) </code></p>
<p>doesn’t work. How do I go about this?</p>

G.M: <p>First, set the parameters before the training. You can set it by accessing the <code>.data</code> attribute(which is a tensor) of the parameter. For example, <code>a.data.copy_(b)</code>. You can also set it with <code>torch.no_grad():</code>.</p>

sabetai: <p>Thanks <a class="mention" href="/u/g.m">@G.M</a>, but that is the opposite of what I’m asking. I want to repeatedly reset the parameters of a ‘learner’ model each iteration during training by a ‘metalearner’. Using <code>data</code> or <code>no_grad</code> doesn’t allow gradient to pass back to the metalearner parameters, which is what I’m looking to do.</p>

G.M: <p>so r u trying to let the gradient of the “learner” to pass to the “meta learner”?</p>

sabetai: <p>Yes. Do you know how to set parameter variables while still being traceable by the autograd library?</p>

G.M: <p>I guess parameters work the same as tensors. U can try <code>a.copy_()</code> without disabling  autograd.</p>
Could not find any instance of Visual Studio

Redcxx: <p>Hi, I am trying to build PyTorch from source but it does not seem to work.<br>
I have installed these dependencies: <code>numpy ninja pyyaml mkl mkl-include setuptools cmake cffi typing</code><br>
I also have visual studio 2019 installed and set the CMake generator flag.</p>
<pre><code class="lang-auto">PS D:\University_of_Manchester\Projects\readocs\pytorch&gt; cmd                                                                                                                                                       Microsoft Windows [版本 10.0.18363.592]
(c) 2019 Microsoft Corporation。保留所有权利。

| D:\University_of_Manchester\Projects\readocs\pytorch =&gt;set CMAKE_GENERATOR=Visual Studio 16 2019

| D:\University_of_Manchester\Projects\readocs\pytorch =&gt;py setup.py install
Building wheel torch-1.5.0a0+6ad9e5c
-- Building version 1.5.0a0+6ad9e5c
cmake --build . --target install --config Release -- /maxcpucount:8
CMake Error:
  Generator

    Visual Studio 15 2017

  could not find any instance of Visual Studio.



Traceback (most recent call last):
  File "setup.py", line 737, in &lt;module&gt;
    build_deps()
  File "setup.py", line 316, in build_deps
    cmake=cmake)
  File "D:\University_of_Manchester\Projects\readocs\pytorch\tools\build_pytorch_libs.py", line 62, in build_caffe2
    cmake.build(my_env)
  File "D:\University_of_Manchester\Projects\readocs\pytorch\tools\setup_helpers\cmake.py", line 339, in build
    self.run(build_args, my_env)
  File "D:\University_of_Manchester\Projects\readocs\pytorch\tools\setup_helpers\cmake.py", line 141, in run
    check_call(command, cwd=self.build_dir, env=env)
  File "C:\Users\wweilue\AppData\Local\Programs\Python\Python37\lib\subprocess.py", line 347, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['cmake', '--build', '.', '--target', 'install', '--config', 'Release', '--', '/maxcpucount:8']' returned non-zero exit status 1.

| D:\University_of_Manchester\Projects\readocs\pytorch =&gt; 
</code></pre>

peterjc123: <p>Could you please try <code>py setup.py install --cmake</code>?</p>
Creating Tensor on the CPU and then transfer to GPU

Gaurav_Jain: <p>Hi,</p>
<p>This might be a fundamentally absurd question, however, I just wanted to understand what am I not understanding right.</p>
<p>So, to use the GPU to train the model, I did the following obvious things:</p>
<p><code>inputs,labels = data[0].to(torch_device),data[1].to(torch_device)</code></p>
<p>and this worked well.</p>
<p>However, before this, I had tried the following:</p>
<pre><code class="lang-auto">inputs,labels = data
inputs.to(torch_device)
labels.to(torch_device)
</code></pre>
<p>but this did not work as when I checked</p>
<p><code>input.is_cuda</code></p>
<p>it returned false.</p>
<p>Now, one more look into it and I understood the difference - <code>input</code> was already a tensor and I was trying to transfer that to the gpu, which fails, whereas when I transfer <code>data</code>, a list, first to the gpu and then create the tensor there, that works perfectly.</p>
<p>Is there a specific reason why a <code>tensor</code> created on the cpu can’t be transferred as is to the GPU?</p>
<p>Thanks</p>

G.M: <p><code>.to()</code> is not in-place operation. U have to use <code>inputs=inputs.to(torch_device)</code>.</p>

Gaurav_Jain: <p>Oh right! Thanks. I assumed the ‘Unified Memory’ paradigm and thought that it would be taken care of.</p>
<p>Now all makes sense. It has got nothing to do with tensors or data.</p>
<p>Thanks,<br>
Gaurav</p>
How to draw the normal distribution parameter distribution chart of VAE model?

shirui-japina: <p>At present, referring to the code <a href="https://github.com/pytorch/examples/tree/master/vae" rel="noopener nofollow ugc">here</a>, the VAE model is implemented. Next I want to try to draw a distribution map of the normal distribution parameters in the VAE model. According to the current understanding and reference <a href="https://github.com/timbmg/VAE-CVAE-MNIST" rel="noopener nofollow ugc">here</a>, it should be possible to draw the following coordinate chart.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/7/f7af81b40dbbcfc5edb15dd7c5ca4699ce53a458.png" data-download-href="https://discuss.pytorch.org/uploads/default/f7af81b40dbbcfc5edb15dd7c5ca4699ce53a458" title="E9-Dist"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/7/f7af81b40dbbcfc5edb15dd7c5ca4699ce53a458_2_281x250.png" alt="E9-Dist" data-base62-sha1="zl7XiG6aLkVJdVsjEP2m31ayFDi" width="281" height="250" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/7/f7af81b40dbbcfc5edb15dd7c5ca4699ce53a458_2_281x250.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/7/f7af81b40dbbcfc5edb15dd7c5ca4699ce53a458_2_421x375.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/f/7/f7af81b40dbbcfc5edb15dd7c5ca4699ce53a458_2_562x500.png 2x" data-dominant-color="EBEAE8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">E9-Dist</span><span class="informations">1648×1463 509 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
My question is,</p>
<ol>
<li>What is the abscissa and ordinate coordinates of this graph that can be drawn through the VAE model?</li>
<li>What changes are needed / should be made for the code referenced above?</li>
</ol>

ptrblck: <ol>
<li>
<p>I guess the image shows the mapping of some inputs (and their classes) using some dimensionality reduction technique such as PCA or TSNE. The axes could be the two highest principle components.</p>
</li>
<li>
<p>You could return the latent output of your VAE and use e.g. <a href="https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.PCA.html"><code>sklearn.decomposition.PCA</code></a> or <a href="https://scikit-learn.org/stable/modules/generated/sklearn.manifold.TSNE.html"><code>sklearn.manifold.TSNE</code></a> to reduce the dimensions.</p>
</li>
</ol>

Shisho_Sama: <p>There is a pull request for VAE example, which I guess has what you need, its not yet been merged , but it should work just fine : <a href="https://github.com/pytorch/examples/pull/632" rel="nofollow noopener">https://github.com/pytorch/examples/pull/632</a><br>
have a look</p>
Cross-Compiling just C++

idurugkar24: <p>I want to compile the source, but just for C++.<br>
I also want to cross-compile for an Intel Atom E3845.<br>
I tried running cmake on the source but it’s erroring out with</p>
<pre><code class="lang-auto">CMake Error at cmake/Codegen.cmake:172 (message):
  Failed to get generated_cpp list
Call Stack (most recent call first):
  caffe2/CMakeLists.txt:2 (include)


-- Configuring incomplete, errors occurred!
</code></pre>
<p>Any help or direction appreciated.</p>
Accessing the loss from any module at backward time

sebastienwood: <p>Hello <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>I’m trying to implement a function which operates on the raw model loss at backward time due to badly defined gradients. One naive approach is to define a global variable which contains the loss and access it as is, however playing around with global variables is a bit edgy.</p>
<p>As such, is there a mecanism that allows to get the model loss from the <code>ctx</code> in a few lines ? (i.e. not modifying the whole model)</p>

ptrblck: <p>I’m not sure I understand the question completely, but if you would like to grab the gradient at specific parameters, you could use hooks as:</p>
<pre><code class="lang-python">model.layer.param.register_hook(my_hook)
</code></pre>
<p>and manipulate or print out the gradient.</p>
<p>Would this work or could you explain, what loss you would like to see?</p>

sebastienwood: <p>Thanks for the reply, I’ll explain a bit more : let’s say I have a function such as</p>
<pre><code class="lang-auto">class MyRound(torch.autograd.Function):
    @staticmethod
    def forward(ctx,input):
        return torch.round(input)
    
    @staticmethod
    def backward(ctx,grad):
        return grad.clone()
</code></pre>
<p>In my use case the <code>grad</code> at <code>backward</code> is not well defined. Rather than operating on this badly defined <code>grad</code> I can take a look at the model loss (e.g. cross entropy) and redefine a <code>grad</code> that is useful. Can I access the cross-entropy loss (or whatever “root” loss from which the backpropagation started) from within <code>backward</code> ? (e.g. through the <code>ctx</code> object) Result would look like :</p>
<pre><code class="lang-auto">@staticmethod
    def backward(ctx,grad):
        good_grad = ctx.root_loss 
        # and some operations to have the correct grad
        return good_grad
</code></pre>
Multiple labels classification problem based on pytorch

yong_xu: <p>Is there a complete code example based on Pytorch about multiple labels classification with unbalanced dataset?</p>

ptrblck: <p>Do you mean “multi-label classification”, where each sample might belong to zero, one, or multiple classes or “multi-class classification”, where each sample belongs to a specific class only?</p>
<p>We have a few code snippets in this forum showing how to use a weighted loss or weighted sampling to counter the effect of an imbalanced dataset.</p>

yong_xu: <p>I mean multi-label classification, i.e., multiple classes can be assigned to each sample. The code snippets you mentioned i have seen. Actually , i am learning Pytorch framework, i just need a more complete code , that is, given one example  dataset , the code has shown how to train the classifier and output the accuracy.</p>