akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Libtorch can be used on the arm development board？

xifanlover: <p>hello everyone,I am using a arm development board.its MCU is i.mx6dl(cortex-A9 ARMV7 32).Now A QT project running on the board.I want my QT project Call the libtorch to load pytorch module.my question is:Can torch and libtorch be compiled on the ARM development board?<br>
I search on the net,someone compiled the pytorch to raspberry pie or android.But no one can compiled the pytorch to other arm board.Can I need to compiled the libtorch to arm board too？<br>
who can help me？</p>

xifanlover: <p>I really hope to cross compile libtorch in the ubantu system and transplant it to the development board。who can help me？</p>

tom: <p>You could use the Android build as inspiration.<br>
That said, you might also be able to use <a class="mention" href="/u/leviviana">@LeviViana</a> 's  build for the Raspberry and take the libtorch included in PyTorch:</p><aside class="quote" data-post="1" data-topic="58580">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/leviviana/48/9548_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/pytorch-1-3-wheels-for-raspberry-pi-python-3-7/58580">PyTorch 1.3 wheels for Raspberry Pi (Python 3.7)</a> <a class="badge-category__wrapper " href="/c/quantization/17"><span data-category-id="17" data-drop-close="true" class="badge-category " title="This category is for questions, discussion and issues related to PyTorch’s quantization feature."><span class="badge-category__name">quantization</span></span></a>
  </div>
  <blockquote>
    I’ve made it available here: <a href="https://wintics-opensource.s3.eu-west-3.amazonaws.com/torch-1.3.0a0%2Bdeadc27-cp37-cp37m-linux_armv7l.whl" rel="noopener nofollow ugc">https://wintics-opensource.s3.eu-west-3.amazonaws.com/torch-1.3.0a0%2Bdeadc27-cp37-cp37m-linux_armv7l.whl</a>. 
Have fun !
  </blockquote>
</aside>

<p>Best regards</p>
<p>Thomas</p>

xifanlover: <p>thank you very mush.I will try.</p>
Resnet50 Pytorch Model Accuracy Loss

mai202: <p>Hello,<br>
I took the resnet50 PyTorch model from torchvision and exported to ONNX. When I ran it using image-classifier on first 1000 images of imagenet data set, i am seeing almost 20% accuracy loss from the resnet50 caffe2 model (on same 1000 images). It makes me wonder if the options i am using for running pytorch model is not correct. I am using “-use-imagenet-normalization” “-compute-softmax” (pytorch model does not softmax in the end) and “-image-mode=0to1”. Does input to pytorch model gets normalized in the same way as caffe2 model?</p>
<p>What can I check to debug this further?</p>

ptrblck: <aside class="quote no-group" data-username="mai202" data-post="1" data-topic="89074">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/278dde/48.png" class="avatar"> mai202:</div>
<blockquote>
<p>Does input to pytorch model gets normalized in the same way as caffe2 model?</p>
</blockquote>
</aside>
<p>Where do you have the caffe2 model from?</p>
<p>As a debugging step I would recommend to pass an input with constant values to both models and compare the outputs (e.g. <code>torch.ones()</code>).<br>
If the outputs are equal, the issue is most likely created in the preprocessing of the data.</p>

mai202: <p>I got caffe model from “<a href="http://fb-glow-assets.s3.amazonaws.com/models" rel="nofollow noopener">http://fb-glow-assets.s3.amazonaws.com/models</a>” which is the url in glow to download models.<br>
<a href="https://github.com/pytorch/glow/blob/master/utils/download_datasets_and_models.py" rel="nofollow noopener">https://github.com/pytorch/glow/blob/master/utils/download_datasets_and_models.py</a></p>
<p>I have also tried models from onnx zoo, <a href="https://github.com/onnx/models/tree/master/vision/classification/resnet/model" rel="nofollow noopener">https://github.com/onnx/models/tree/master/vision/classification/resnet/model</a><br>
“resnet50-v2-7.onnx” is showing same accuracy loss.</p>
<p>For same input, models does show different outputs. I have not tried with torch.ones but for same image.</p>

ptrblck: <p>Thanks for the update. Unfortunately, I cannot be of much help, as I don’t know how these models (in Glow and ONNX) were created.<br>
Just by reading through the ONNX link, it seems that the preprocessing is <a href="https://github.com/onnx/models/tree/master/vision/classification/resnet#preprocessing">identical</a> to the <code>torchvision.models</code>.</p>

mai202: <p>Thanks for getting back to me.</p>
Extremely Slow Network

ihammond: <p>Hello, I am using a convolutional decoder like network to recreate 500x500 point images (continuous data between 0,1) from 3 points of data. My network is not too complex and looks like the image shown below, yet with a little under 3000 training samples and a batch size of 300 it takes over six hours for a single epoch. This is rediculous! Any idea why?<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/f/0fcedb480fcc71924ecacd1b21f7031e24b889b2.png" data-download-href="https://discuss.pytorch.org/uploads/default/0fcedb480fcc71924ecacd1b21f7031e24b889b2" title="Screen Shot 2020-07-13 at 12.41.03 PM"><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/f/0fcedb480fcc71924ecacd1b21f7031e24b889b2.png" alt="Screen Shot 2020-07-13 at 12.41.03 PM" data-base62-sha1="2fQlIcWFWICf54GGIQpM0yFjKtY" width="690" height="441" data-dominant-color="E2E2E2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2020-07-13 at 12.41.03 PM</span><span class="informations">1021×653 11.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
The Torch layers and activations i’m using are ConvTransposed2d and 3d, PReLU, Linear, and Tanh.</p>
The size of tensor a (2) must match the size of tensor b (39) at non-singleton dimension 1

Nawel: <p>Hello,</p>
<p>I’m trying to train a CamemBERT Base Cased model in Google Colab.  I’m using fast-bert library.</p>
<p>Sometimes the code runs fine first time without error.</p>
<p>Other times, the same code, using the same data, results in a “CUDA out of memory” error.</p>
<p>Previously, restarting the runtime or exiting the notebook, going back into the notebook, doing a factory runtime restart, and re-running the code runs successfully without error. Just now though, I’ve tried a restart and re-try many times and got the error every time, even when reducing the batch size to 1.</p>
<p>Does anyone know why this is happening, why it is intermittent, and/or what I can do about it?</p>
<p>When it does work, when reducing the batch size per gpu from 16 to 8. I got this error:</p>
<p>RuntimeError: The size of tensor a (2) must match the size of tensor b (39) at non-singleton dimension 1</p>
<p>I know it’s due to the batch size changing, but have no idea how to fix it.</p>
PyTORCH and NPU

Peter_Peter: <p>Hello all,</p>
<p>I have a question regarding the inner-working of PyTorch and trying to understand how can PyTorch leverage from NPU and basically interface custom NPU for PyTorch?<br>
In other words how to access the computing capability of NPU to Pytorch to have benefited from?<br>
Is there any interface in PyTorch to be developed on custom NPU and provide compute performance?</p>
<p>Thank you.</p>
Python with ILP64 openBLAS

Rohit_Srivastava: <p>How does one build pytorch with int64_t indexing version of openBLAS to support LAPACK operations on inputs of sizes &gt; 2^32</p>

albanD: <p>Hi,</p>
<p>Once you have your OpenBLAS with int64_t indexing built, you can either put it in a known OpenBLAS install folder like <code>/opt/OpenBLAS/lib</code> or specify its path in <code>LD_LIBRARY_PATH</code>. Then you can run <code>setup.py</code> script with <code>BLAS=OpenBLAS</code> to ensure that it uses OpenBLAS and not another BLAS library.</p>

Rohit_Srivastava: <p>How do one built openBLAS with int64_t support ?</p>

albanD: <p>I don’t know <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"> But I’m sure you can find instructions in their repo or online.</p>
I want to multiply the input to forward by a large constant. Would it be ok to save the constant in the __init__ function?

torchlight: <p>Would this break the autograd? And what is the difference between doing this and using self.register_buffer</p>

albanD: <p>Hi,</p>
<p>In the init of the <code>nn.Module</code>? No that won’t change anything from the point of view of the autograd.<br>
The benefit of having it being a buffer is that it will be moved around when you do things like <code>model.cuda()</code>.</p>
Summing a tensor according to a different tensor provided

melody_marks: <p><code>torch.tensor</code> sums up the tensor along any given dimension. For example, if we have a tensor of size <code>[1000, 300]</code>, <code>torch.sum(T, axis=0)</code> will return a tensor of shape <code>[300]</code>. I don’t want to sum the whole tensor. I have another tensor <code>[200, 200, 600]</code>. So, what I want is, tensor of shape <code>[3, 300]</code> in which, the first row is sum of first 200 rows of tensor T, second row is sum of next 200 rows and the last row is sum of remaining 600 rows of tensor T.  Is it possible to achieve this? It is guaranteed that sum of values in different tensor <code>(200 + 200 + 600)</code> is always equal to number of rows in main tensor T. Thanks in advance <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>Edit : I am aware about the splitting in pytorch, however, it will require me to use a for loop. Turns out that number of elements in different tensor are really large, and in every iteration, using for loop is too much costly. I’m looking for an efficient way of doing this operation.</p>
<p>I had one idea to write a separate function to do this operation. This function will take the boundaries and the main tensor as input and return the required sum. However, I’m not sure how do I parallelise this function. This function basically does the same thing on different parts of the input tensor. Does pytorch provide any functionality to parallelise this operation? Parallelising this is really efficient and cool since the operations are independent of each other, are mutually exclusive and exhaustive. If this is not possible in current scope of pytorch, are there any other external tools/libraries that can do this?</p>
Semantic segmentation, how does it work?

Hwarang_Kim: <p>I<code>m trying to run U-Net. My data set has 3 classes (2 classes + background) so I made the mask which contains class indices for each pixel like [[0,1,2,1,2,0,...,1],[0,1,1,0,0,...,1]...[0,1,1,0,1,...,2]]. The mask shape is (512,512), and class indices are 0, 1, 2 (0 for background). But I wonder how the model works! My model(U-Net)</code>s input channel is 1 (gray image), and output is 3 channels.<br>
I have no idea which loss function do i need to use? why?<br>
And if the answer is CrossEntropyLoss, how does it work despite output channel is 3, mask channel is 1.<br>
Please god help me.<br>
(Or did i do something wrong?)</p>

ptrblck: <p>You should use <code>nn.CrossEntropyLoss</code> and can have a look at the <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.CrossEntropyLoss">docs</a> to see how the loss formula is implemented.<br>
I would recommend to skip the spatial dimensions for now and just take a look at the standard multi-class classification use case, where the output has the shape <code>[batch_size, nb_classes]</code> and the target contains class indices in the range <code>[0, nb_classes-1]</code> with the shape <code>[batch_size]</code>.<br>
As you can see in the docs, the output logits can be directly indexed using the target.</p>
<p>For the multi-class segmentation use case, the same formula yould be applied but for each pixel location, since the model output and the target now have additional <code>[..., height, width]</code> dimensions.</p>
Numerical differences between numpy and pytorch?

satubus: <p>Hello all,</p>
<p>When computing mean and std on numpy or pytorch tensors, they yield different results. How come ?</p>
<p>rng = np.random.RandomState(1)<br>
dataset1 = rng.uniform(low=-0.01, high=0.01, size=(1000, 20))<br>
dataset2 = torch.from_numpy(data_numpy)<br>
print(dataset1.mean(), dataset1.std())<br>
print(dataset2.mean(), dataset2.std())</p>
<p>returns<br>
2.5537095782416174e-05 0.005769608507668796<br>
tensor(-3.14198780749532207562380037302319e-06, dtype=torch.float64) tensor(0.00576444647196127767790896356814, dtype=torch.float64)</p>

albanD: <p>Hi,</p>
<p>This comes from the limited precision of floating point numbers.<br>
Each operation on float32 has a precision of ~1e-6. And accumulating a large number of them can lead to big differences.<br>
Similar for float64 where it starts around 1e-12 and goes up from there.</p>

satubus: <p>Hello AlbanD,<br>
Thank you for your quick response and time.<br>
Maybe I do not understand fully. Shouldn’t pytorch and numpy operations yield same values when initialized with same floating point precision assuming the rounding rules are equal?</p>

albanD: <p>Hi,</p>
<p>Unfortunately no <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"><br>
The reason is that floating point operations are not associative: <code>(a + b) + c != a + (b + c)</code>. And so any difference in the order where things are accumulated will lead to such discrepancies.<br>
For these ops, both pytorch and numpy use multithreading. But because they do this in a slightly different way, you see these differences.</p>
<p>Note that some operations are not even deterministic (usually on the GPU) and running it twice in a row won’t give you bit-perfect same results. See the note in the doc about determinism if you want to learn more.</p>
Broadcasting element wise multiplication in pytorch

nowyouseeme: <p>I have a tensor in pytorch with size <code>torch.Size([1443747, 128])</code>. Let’s name it tensor <code>A</code>. In this tensor, 128 represents a batch size. I have another 1D tensor with size <code>torch.Size([1443747])</code>. Let’s call it <code>B</code>. I want to do element wise multiplication of B with A, such that B is multiplied with all 128 columns of tensor <code>A</code> (obviously in an element wise manner). In other words, I want to broadcast the element wise multiplication along <code>dimension=1</code>.<br>
How can I achieve this in pytorch? <br></p>
<p>It I didn’t have a batch size involved in the tensor A (<code>batch size = 1</code>), then normal <code>*</code> operator would do the multiplication easily. <code>A*B</code> then would have generated resultant tensor of size <code>torch.Size([1443747])</code>. However, I don’t understand why pytorch is not broadcasting the tensor multiplication along dimension 1? Is there any way to do this?</p>
<p>What I want is, <code>B</code> should be multiplied with all 128 columns of <code>A</code> in an element wise manner. So, the resultant tensors’ size would be <code>torch.Size([1443747, 128])</code>.</p>

KFrank: <p>Hello Mr. Knight!</p>
<aside class="quote no-group quote-modified" data-username="nowyouseeme" data-post="1" data-topic="89608" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/n/f0a364/48.png" class="avatar"> nowyouseeme:</div>
<blockquote>
<p>I want to do element wise multiplication of B with A, such that B is multiplied with all 128 columns of tensor <code>A</code>.…<br>
I want to broadcast the element wise multiplication along <code>dimension=1</code>.</p>
</blockquote>
</aside>
<p>Give <code>B</code> a dimension of size 1 using <code>unsqueeze()</code> so that it has a<br>
dimension from which to broadcast:</p>
<pre data-code-wrap="python"><code class="lang-python">B.unsqueeze (1) * A
</code></pre>
<p>Best.</p>
<p>K. Frank</p>
Torchscript back to eager

tmain: <p>Hi everyone,</p>
<p>is there an easy way to convert a torchscripted model back to an eager model ?</p>
<p>Example:</p>
<pre><code class="lang-python">eager_model = MyModel()
scripted_model = torch.jit.script(eager_model)
recovered_eager_model = some_function(scripted_model)  ### could not find anything about it in the docs
</code></pre>

tom: <p>No, and it is strongly advised that you keep your source code around when doing stuff with JITed models.</p>
<p>That said, you can probably get a reduced model by walking the traced module, setting the children in init and using the <code>.forward</code> of each module.<br>
This would get you something you can run and re-trace, but you loose how it was built (i.e. the inits).</p>
<p>Fun fact: In building a <a href="https://lernapparat.de/transformers-pytorch-tvm/" rel="nofollow noopener">PyTorch-TVM bridge</a> I built a PyTorch autograd function (but with TVM as backend instead of calling PyTorch) corresponding to the traced model.</p>
<p>Best regards</p>
<p>Thomas</p>
Predicting a function with PyTorch

Capo_Mestre: <p>Hello. There are many examples of neural networks for MNIST hand-written digits classification problem, where the output is a 10-element softmax-vector with one maximum value corresponding to the prediction. This is the case where a label for a particular data-sample is just a number (one-element label, that can take values from 0 to 9). What would the last layer be and a what would the loss function be if I want to be able to predict a multi-element label, where the label for a particular data-sample consists of many elements (say, 128)? In other words, how to predict a function?</p>
<p>For the sake of a dummy example, consider the following problem. We are given an arbitrary greyscale image I(x,y) as an input. We want to integrate the image along the horizontal axis (x-axis) to get the result as a series of numbers f(y)=∑I(x,y). We want our neural network to output f(y).</p>

ptrblck: <p>The model output wouldn’t necessarily need to change for the mentioned use cases, but the loss functions and targets.</p>
<p>E.g. if you would like to predict N floating point values, your model could output a tensor in the shape <code>[batch_size, N]</code> and you could use e.g. <code>nn.MSELoss</code> for this regression task.<br>
On the other hand, if your would like to output N labels, where zero, one, or multiple classes could use active, you could stick to the output shape and use <code>nn.BCEWithLogitsLoss</code> instead as the criterion.</p>

Capo_Mestre: <p>Thanks for the clarification! It make sense.</p>
_inference within create_supervised_trainer?

pytorchnewbie: <p>Hi,</p>
<p>In one of ignite’s issues (<a href="https://github.com/pytorch/ignite/pull/48/files" rel="nofollow noopener">https://github.com/pytorch/ignite/pull/48/files</a>) I saw that the _inference was passed in to the trainer engine. Why is this option not in the docs? Is it not recommended to do this? Why not?</p>
<p>Thanks</p>

ptrblck: <p>Moving to <code>Ignite</code> category for better visibility and CC’ing <a class="mention" href="/u/vfdev-5">@vfdev-5</a> <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

vfdev-5: <p>Seems like it is a very old PR and things changed since than.</p>
<p>Currently, we do not provide such abstractions like <code>Trainer</code> etc as there is no unique definition of it as it can be task/config dependent etc. We left that to the user to compose its trainer using one or more <code>Engine</code>s for training/validation etc.</p>
How to generate result of desirable size?

bolt25: <p>I have a model which was trained on images of (512,512) size.<br>
Now I have an image which is of dimension (x,y) where x,y are not equal to 512,512. Which means I’ll either need to pad them or crop them for making them (512,512).<br>
Now what I want to do is, I need to generate image of the size (x,y) at the end.<br>
Any Ideas on how to do that?</p>
<p>PS- I’m talking about taking crops from the main image and passing them from the model and finally stitching them back together.</p>
<p>Thank you in advance!</p>

Usama_Hasan: <p>I guess what you’re asking for is generative adversarial models (GAN’s) for image reconstruction. You can find a tutorial of face generation here.<br>
<a href="https://pytorch.org/tutorials/beginner/dcgan_faces_tutorial.html" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/dcgan_faces_tutorial.html</a><br>
As for crops, you can use torchvision.transfroms to do that</p>

bolt25: <p>No no! I’m not looking for GANs!<br>
Suppose I have 2 images of (40,40) and I want to stitch them together.<br>
That’s what I’m looking for!</p>

Usama_Hasan: <p>Oh I see, I guess this helps you.</p>
<pre><code class="lang-auto">import torch
import matplotlib.pyplot as plt

image1 = torch.randn(size= (40,40))
image2 = torch.randn(size = (40,40))

width = image1.size(0) + image2.size(0)
height = image1.size(1)

image3 =torch.zeros(size = (height,width))

image3[:height,:int(width/2)] = image1
image3[:height,int(width/2):width] = image2

plt.imshow(image3)
plt.show()
</code></pre>
<p>Or you can check make_grid function in torchvision.utils</p>
'Tensor' object has no attribute 'meter'

tianle-BigRice: <p>When I use meter.AverageValueMeter() times wrong,<br>
My code looks like this:</p>
<pre><code class="lang-auto">    loss_meter = meter.AverageValueMeter()
    previous_loss = 1e10
    # train
    for epoch in range(opt.max_epoch):
        
        loss_meter.reset()
        # confusion_matrix.reset()

        for ii,(data,label) in tqdm(enumerate(train_dataloader),total=len(train_data)):

            # train model 
            input = Variable(data)
            input = input.float()
            target = Variable(label)
            if opt.use_gpu:
                input = input.cuda()
                target = target.cuda()

            optimizer.zero_grad()
            score = model(input)
            loss = criterion(score,target) / (128 * 128)
            loss.backward()
            optimizer.step()  

  
            loss.meter.add(loss.item())
</code></pre>

Nikronic: <p>Hi,<br>
Actually, I don’t know what <code>meter</code> is here but don’t you have typo?</p>
<aside class="quote no-group" data-username="tianle-BigRice" data-post="1" data-topic="89548">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/t/5daacb/48.png" class="avatar"> tianle-BigRice:</div>
<blockquote>
<p><code>loss_meter = meter.AverageValueMeter()</code></p>
</blockquote>
</aside>
<p>You defined <code>loss_meter</code> but you are calling <code>loss.meter</code>. <code>loss</code> is a tensor and it does not have <code>meter</code>.</p>
<aside class="quote no-group" data-username="tianle-BigRice" data-post="1" data-topic="89548">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/t/5daacb/48.png" class="avatar"> tianle-BigRice:</div>
<blockquote>
<p><code>loss.meter.add(loss.item())</code></p>
</blockquote>
</aside>
<p>So, I think last line should be <code>loss_meter.meter.add(loss.item())</code></p>
<p>Bests</p>

tianle-BigRice: <p>thank you,i did typo . <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=9" title=":sweat_smile:" class="emoji" alt=":sweat_smile:"><br>
the code should be loss_meter.add(loss.item) <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=9" title=":sweat_smile:" class="emoji" alt=":sweat_smile:"></p>
Loading data in custom dataloader based on condition

harshj94: <p>I have created my own CustomDataloader and in my <strong>getitem</strong>(self, idx) method, I want to make sure that the data is returned only if a certain condition is met. I want to know what should be returned from Dataloader in such cases?</p>
<pre><code class="lang-auto">def __getitem__(self, idx):

	if condition is True:
		...
		...
		...
		return Image, Label
	else
		...
		...
		do nothing
</code></pre>
<p>But we cannot return None. How can I handle this condition that my batch does not include this data, which may result in a smaller batch size?</p>

tux: <p>If you don’t use these data, why don’t you preprocess your training set before calling the dataloader ? You can do it with a <code>for</code> loop for instance.</p>

neeek2303: <aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="16" height="16">
      <a href="https://github.com/msamogh/nonechucks" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars3.githubusercontent.com/u/1230386?s=400&amp;v=4" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://github.com/msamogh/nonechucks" target="_blank" rel="nofollow noopener">msamogh/nonechucks</a></h3>

<p>Deal with bad samples in your dataset dynamically, use Transforms as Filters, and more! - msamogh/nonechucks</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
Can anybody help me out?

samr1104: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/6/06422e73326e2e4f9febcc58ecdc1342a8e998e3.png" data-download-href="https://discuss.pytorch.org/uploads/default/06422e73326e2e4f9febcc58ecdc1342a8e998e3" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/6/06422e73326e2e4f9febcc58ecdc1342a8e998e3_2_690x384.png" alt="image" data-base62-sha1="TmEN1CzLLpsBTmFzxmrLUfG6EX" width="690" height="384" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/6/06422e73326e2e4f9febcc58ecdc1342a8e998e3_2_690x384.png, https://discuss.pytorch.org/uploads/default/optimized/3X/0/6/06422e73326e2e4f9febcc58ecdc1342a8e998e3_2_1035x576.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/0/6/06422e73326e2e4f9febcc58ecdc1342a8e998e3.png 2x" data-dominant-color="F8F8F9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1193×664 28.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Implement a MLP classifier network with hyper-parameters obtained from sklearn

Pronerd: <p>I am trying to implement an MLP network with the LBFGS optimizer to Solve a classification problem. I have implemented a network in the sklearn with following configuration:</p>
<pre><code class="lang-auto">MLPClassifier(activation='relu', alpha=1e-5,
             hidden_layer_sizes=(40,40,40), 
             max_fun=15000, max_iter=1500,
             solver='lbfgs', tol=0.0001,
             validation_fraction=0.1, verbose=False, warm_start=False)
</code></pre>
<p>But I don’t know how these hyperparameters should be configured in PyTorch. For example,  ‘max_iters’ stands for number of gradient steps, not for the number of epochs for “LBFGS” optimizer and I don’t know how should I interpret that in PyTorch!</p>

Nikronic: <p>Hi,</p>
<p>For MLP part and activation function, you can use <code>torch.nn</code> modules such as <code>nn.Linear</code> and <code>nn.ReLU</code>.<br>
For <code>L-BFGS</code> solver, PyTorch supports it in <code>torch.optim</code>. You can see possible arguments <a href="https://pytorch.org/docs/stable/optim.html#torch.optim.LBFGS" rel="nofollow noopener">here</a>.<br>
For <code>validation_fraction</code> argument, you can either create validation set using <code>sklearn.model_selection</code> or manually using list comprehensions or even data loaders of pytorch.</p>
<p>Bests</p>

Pronerd: <p>Thank you very much, but my main question is about the <code>max_iter</code> parameter. As I have noticed, this parameter in <code>sklearn </code> (when the solver is “LBFGS”) doesn’t indicate the number of epochs but the number of gradient steps. Moreover, the parameter <code>max_fun</code> indicates the maximum number of loss function calls. How can I set these two parameters in <code>torch.optim</code>?</p>

Nikronic: <p>If you visited the link I provided you could see the definition of <code>optim.LBFGS</code> which is:</p>
<pre><code class="lang-auto">torch.optim.LBFGS(params, lr=1, max_iter=20, max_eval=None, tolerance_grad=1e-07, tolerance_change=1e-09, history_size=100, line_search_fn=None)
</code></pre>
<p>Where <code>max_iter</code> is same as sklearn and <code>max_fun</code> is <code>max_eval</code>.</p>
<pre><code class="lang-auto">lr (float) – learning rate (default: 1)
max_iter (int) – maximal number of iterations per optimization step (default: 20)
max_eval (int) – maximal number of function evaluations per optimization step (default: max_iter * 1.25).
tolerance_grad (float) – termination tolerance on first order optimality (default: 1e-5).
tolerance_change (float) – termination tolerance on function value/parameter changes (default: 1e-9).
history_size (int) – update history size (default: 100).
line_search_fn (str) – either ‘strong_wolfe’ or None (default: None).
</code></pre>
Grid_sample one to many

marioviti: <p>Hello!</p>
<p>I was wondering if is there a smart way to achieve what I call (for lack of better definition) one to many grid_sample.<br>
What do I mean by that? instead of sampling  N(many) (C x IH x IW) images using N(many) (OH x OW x 2) grids is it possible to have  <a href="https://pytorch.org/docs/0.3.1/nn.html#torch.nn.functional.grid_sample" rel="nofollow noopener">grid_sample</a> to have a single image  1 (C x IH x IW) sampled with N (OH x OW x 2) grids? and why not also Many to One in wich many images are sampled at the same locations?</p>
<p>Is there a smart/fast way to achieve this (instead of looping over N calls to grid_sample)? It could be really interesting especially for sparse object detection in very big images or volumes.</p>

marioviti: <p>The solution is simple:</p>
<p>reshape the grid as (1 X NOH x OW x2)<br>
call grid_sample and reshape the output to (NxCxOHxOw)!</p>
<p>The downside is that you may have border issues due to the interpolation of coordinates in very different places…</p>
Error when exporting ScriptFuntion to ONNX

Zhiqiang_Fong: <p>Hello everyone, I’m new to ONNX and I’m trying to convert a model where I need do some for-loop assignmens like the code below,</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

@torch.jit.script
def create_alignment_v2():
    base_mat = torch.zeros(2, 2, 2)

    for i in range(base_mat.size(0)):
        base_mat[i][0][0] = 1

    return base_mat

class ToyModule(nn.Module):

    def __int__(self):
        super().__init__()


    def forward(self, duration_predictor_output):

        alignment = create_alignment_v2()
        # output = alignment @ x
        return alignment

def test():
    module = ToyModule()
    module.eval()
    x = torch.rand(2, 28, 384)
    alignment = module(x)
    torch.onnx.export(module, x, 'toy.onnx',
                      export_params=True,
                      opset_version=10,
                      do_constant_folding=True,
                      verbose=True,
                      input_names=['seq'],
                      output_names=['alignment'],
                      dynamic_axes={'seq': {0: 'batch', 1: 'sequence'},}
                      )

test()
</code></pre>
<p>And the error message is:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "/mfs/fangzhiqiang/workspace/tts/FastSpeech/jit_script.py", line 83, in &lt;module&gt;
    test()
  File "/mfs/fangzhiqiang/workspace/tts/FastSpeech/jit_script.py", line 78, in test
    dynamic_axes={'seq': {0: 'batch', 1: 'sequence'},}
  File "/home/fangzhiqiang/miniconda3/envs/tts/lib/python3.6/site-packages/torch/onnx/__init__.py", line 168, in export
    custom_opsets, enable_onnx_checker, use_external_data_format)
  File "/home/fangzhiqiang/miniconda3/envs/tts/lib/python3.6/site-packages/torch/onnx/utils.py", line 69, in export
    use_external_data_format=use_external_data_format)
  File "/home/fangzhiqiang/miniconda3/envs/tts/lib/python3.6/site-packages/torch/onnx/utils.py", line 488, in _export
    fixed_batch_size=fixed_batch_size)
  File "/home/fangzhiqiang/miniconda3/envs/tts/lib/python3.6/site-packages/torch/onnx/utils.py", line 351, in _model_to_graph
    fixed_batch_size=fixed_batch_size, params_dict=params_dict)
  File "/home/fangzhiqiang/miniconda3/envs/tts/lib/python3.6/site-packages/torch/onnx/utils.py", line 120, in _optimize_graph
    torch._C._jit_pass_onnx_prepare_inplace_ops_for_onnx(graph)
IndexError: vector::_M_range_check: __n (which is 2) &gt;= this-&gt;size() (which is 2)
</code></pre>
<p>In fact, if I remove the assignment operation, the graph can be built succesfully. I wonder whether this is a bug and how to convert such model to ONNX. Thanks for any reply~</p>
Meaning of transform_input parameter in inceptionV3

jugo: <p>Hi, I’m trying to modify the output of inceptionV3 by creating  new class that derives from InceptionV3.</p>
<p>My Question is regarding the parameter transform_input in:<br>
<a href="http://torchvision/models/inception.py" rel="nofollow noopener">torchvision/models/inception.py</a></p>
<p>If I’m using the pretrained version and not passing this parameter this is set to True, does this mean that I should not do any preprocesing to the images that I feed the network?<br>
Currently I’m using a composed Transform that I apply to the images, most of it is for data augmentation but I also have normalization there:</p>
<pre><code class="lang-auto">deep_drive_training_trafo = transforms.Compose([
    transforms.Resize((299, 299)),
    transforms.RandomResizedCrop(size=299, scale=(0.333, 1.0), ratio=(0.75, 1.3333333333333333)),
    transforms.RandomHorizontalFlip(),
    transforms.ColorJitter(brightness=0.6, contrast=0.5, saturation=0.8),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406],  # here we need 3 values, 1 for each chanel
                         std=[0.229, 0.224, 0.225]),
    transforms.RandomErasing(p=0.3, scale=(0.033, 0.2), ratio=(0.3, 3.3), value='random', inplace=False)
])
</code></pre>
<p>Is this correct or by not passing transform_input to InceptionV3 the model is applying another transform on top of that? If it is true which should I use?</p>

ptrblck: <p>If you set <code>transform_input=True</code>, <a href="https://github.com/pytorch/vision/blob/3c254fb7af5f8af252c24e89949c54a3461ff0be/torchvision/models/inception.py#L191">this line</a> will call into <a href="https://github.com/pytorch/vision/blob/3c254fb7af5f8af252c24e89949c54a3461ff0be/torchvision/models/inception.py#L119-L125">this method</a>.</p>
<p>However, by default this argument is set to <a href="https://github.com/pytorch/vision/blob/3c254fb7af5f8af252c24e89949c54a3461ff0be/torchvision/models/inception.py#L43"><code>False</code></a>.</p>

Janani_Sundara_gandh: <p>Pretrained model has transform_input as True by default. I encountered it when there was mismatch in output when loading the statedict separately. You may need to specify in documentations clearly.</p>
Output of torch.argsort is not right

coincheung: <p>Hi,</p>
<p>In brief, my code is like this:</p>
<pre><code class="lang-auto">a = torch.randn(3,4).view(-1)
print(a)
print(a.argsort)
</code></pre>
<p>my pytorch version is 1.5.1, and the output on my platform is:</p>
<pre><code class="lang-auto">tensor([-0.3868, -1.0274,  2.1992,  0.2730, -0.7430, -1.2569, -0.9533,  0.4771,
        -1.5247, -0.0411, -0.9043,  0.2035])
tensor([ 8,  5,  1,  6, 10,  4,  0,  9, 11,  3,  7,  2])
</code></pre>
<p>How could I make it work, or is there any problem with my understanding of this function ?</p>

Nikronic: <p>Hi,</p>
<p>Could you explain what the issue is? Based on your printed output, it is working fine.</p>
<p><a href="https://pytorch.org/docs/master/generated/torch.argsort.html" rel="nofollow noopener">torch.argsort docs</a><br>
Bests</p>

coincheung: <p>Hi,</p>
<p>If I have not misunderstood the function definition, <code>a.argsort()</code> would give the index of the ordered array.</p>
<p>In this case, the tensor is <code>[-0.3868, -1.0274,  2.1992,  0.2730, -0.7430, -1.2569, -0.9533,  0.4771,         -1.5247, -0.0411, -0.9043,  0.2035]</code>, and the smallest element of it is <code>-1.5247</code> with index <code>7</code>. However, the first element of the output index array of <code>a.argsort()</code> is 8, the corresponding element of which in the array is <code>-0.0411</code>. I thinks this is not correct, since the smallest element is the 7th, and the output of <code>a.argsort()</code> should be <code>[7, ...]</code>.</p>

Nikronic: <p>I think you are using Matlab logic, in python we count from zero, so 7th element is <code>0.4771</code> and 8th element is <code>-1.5247</code>.</p>

coincheung: <p>yes, the 7th is 0.4771 which is the neither the largest nor the smallest value in the given array.since the largest is the one with index 2(count from 0) with value <code>2.1992</code> and the smallest is the one of index 8 with value <code>-1.5247</code>.</p>

coincheung: <p>Ah, it is my bad, I will check my logic carefully. Thanks for replying !!</p>
Libtorch ubuntu runtime error

21fa417e3fb06e56040c: <p>hi all.<br>
I have a question.</p>
<p>same code runs fine on Windows, but a runtime error occurs on Linux.<br>
Can you look at me together?</p>
<p>this is my repo <a href="https://github.com/kerry-Cho/transfer-learning-Libtorch" rel="nofollow noopener">https://github.com/kerry-Cho/transfer-learning-Libtorch</a></p>
<p>How to start building</p>
<pre><code class="lang-auto">1. run bash build.sh 
2. cd build/
3. make 
4. cd bin/
5. run ./transfer-learning

</code></pre>
<p>below crash code</p>
<pre><code class="lang-auto">torch::Tensor read_data(std::string location) {
	/*
	 Function to return image read at location given as type torch::Tensor
	 Resizes image to (224, 224, 3)
	 Parameters
	 ===========
	 1. location (std::string type) - required to load image from the location
	 Returns
	 ===========
	 torch::Tensor type - image read as tensor
	*/
	std::cout &lt;&lt; location &lt;&lt; std::endl;
	cv::Mat img = cv::imread(location, 1);
	cv::resize(img, img, cv::Size(224, 224), cv::INTER_CUBIC);
	torch::Tensor img_tensor = torch::from_blob(img.data, { img.rows, img.cols, 3 }, torch::kByte); &lt;&lt; here crash point
	img_tensor = img_tensor.permute({ 2, 0, 1 });
	return img_tensor.clone();
}
</code></pre>

21fa417e3fb06e56040c: <p><a class="mention" href="/u/yf225">@yf225</a> help me <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=9" title=":smile:" class="emoji" alt=":smile:"> <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=9" title=":smile:" class="emoji" alt=":smile:"> <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=9" title=":smile:" class="emoji" alt=":smile:"></p>

21fa417e3fb06e56040c: <p><a class="mention" href="/u/yf225">@yf225</a> sorry.<br>
It was my mistake.<br>
It happened because of a file path error</p>
Missunderstanding of Torchvision FasterRCNN arguments

Mdhvince: <p>Hello,</p>
<p>I just look at the FasterRCNN code on PyTorch repo and I think I misunderstood some arguments. FasterRCNN have <code>box_nms_thresh</code> as argument, but when testing different inference by setting up the nms with different values, I notice that it actually refer to an IOU threshold, because it will remove boxes that overlap more than the threshold passed.</p>
<p>In my understanding, I think nms threshold have to refer to the task of removing boxes lower than a confidence score (like the <code>box_score_thresh</code> do).</p>
<p>I have a doubt on this one, I would like someone confirm my understanding of the arguments here or explain me how should I understand it.</p>
<p>Thank you</p>
<p>Medhy</p>

Mdhvince: <p>I also look at the torchvision.ops implementation but here my thought is confirmed by passing iou_thresh as argument name <code>torchvision.ops.nms( *boxes* ,  *scores* ,  *iou_threshold* )</code>.</p>

Arka_Mitra: <p>As per the source code, box_score_thresh (float): during inference, only return proposals with a classification score greater than box_score_thresh.<br>
So it actually talks about classification scores. Not IoU scores.</p>
Extending Quantization-Aware Training

mg1371: <p>Hi,</p>
<p>I would like to extend QAT to support two below cases. Does anyone know how I can achieve these?</p>
<ol>
<li>
<p>mixed-precision: being able to set precision for each layer separately (manually)</p>
</li>
<li>
<p>lower precisions: being able to fake-quantize to lower than 8-bit (using a QConfig?)</p>
</li>
</ol>
<p>Thanks!</p>

jerryzh168: <p>Raghu is adding support for sub 8 bit qat right now cc <a class="mention" href="/u/raghuramank100">@raghuramank100</a><br>
I think mixed precision is supported as long as you can have sub 8 bit observers, in eager mode quantization you’ll need to set the qconfig manually for each child module.</p>

mg1371: <p>Thanks <a class="mention" href="/u/jerryzh168">@jerryzh168</a> and <a class="mention" href="/u/raghuramank100">@raghuramank100</a>.</p>
<p>Is there a way to have fake 8-bit observers (scale factors and zero point) in current implementation?</p>

jerryzh168: <p>you’ll need to implement your own observer module (<a href="https://github.com/pytorch/pytorch/blob/master/torch/quantization/observer.py" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/master/torch/quantization/observer.py</a>) and fake quantize module(<a href="https://github.com/pytorch/pytorch/blob/master/torch/quantization/fake_quantize.py" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/master/torch/quantization/fake_quantize.py</a>) to support 8-bit scale and zero_point</p>

111179: <p>hi, have you implement your question? Now, I am learning QAT…</p>
Pytorch mobile deployment

zeus: <p>Hi. I am using pytorch 1.5.0. I have put gradle dependency as</p>
<pre><code class="lang-auto">dependencies {
    implementation fileTree(dir: "libs", include: ["*.jar"])
    implementation 'androidx.appcompat:appcompat:1.1.0'
    implementation 'androidx.constraintlayout:constraintlayout:1.1.3'
    testImplementation 'junit:junit:4.12'
    androidTestImplementation 'androidx.test.ext:junit:1.1.1'
    androidTestImplementation 'androidx.test.espresso:espresso-core:3.2.0'
    implementation 'org.pytorch:pytorch_android:1.4.0'
    implementation 'org.pytorch:pytorch_android_torchvision:1.4.0'

}
</code></pre>
<p>I trained my model on CPU and used this script to export</p>
<pre><code class="lang-auto">model_ft.eval()
example = torch.rand(1, 3, 224, 224)
traced_script_module = torch.jit.trace(model_ft, example)
traced_script_module.save("model_res.pt")
</code></pre>
<p>When I run the detection in android, it shows</p>
<pre><code class="lang-auto">Process: com.example.pytorchmobile, PID: 7756
    com.facebook.jni.CppException: empty_strided not implemented for DispatchKeySet(BackendSelect, CUDATensorId) (empty_strided at aten/src/ATen/Functions.h:4051)
    (no backtrace available)
</code></pre>
<p>Please HELP ME.</p>
Doubt regarding linking based system

vainaijr: <p>Is there a work done on creating a linking based system, for example, linking words with each other, like,</p>
<pre><code class="lang-auto">'red'
^
|
'apple' -&gt; 'tree'
|
V
'taste'
</code></pre>
<p>and instead of updating word embedding, our neural network task is to update these links, that is create new links, or break existing links, what would be the parameters in this case?</p>

vdw: <p>You might want to look into <a href="https://arxiv.org/abs/2002.00819" rel="nofollow noopener">link prediction for knowledge bases</a>.</p>
<p>The basic goal is that, given an existing knowledge base, what new facts can be predicted. Not sure if this matches your use case.</p>
How Do I Make a Custom Dataset for Image Segmentation?

Max_Smith: <p>Hi, spent the last couple hours researching this to no avail.  I’m trying to import both my testing images and my overlay masks into pytorch as one dataset.</p>
<p>My folders look like this</p>
<p>All testing images-<br>
-Video1<br>
-image1.png<br>
-image2.png<br>
-image3.png<br>
-Video2<br>
-image1.png<br>
-image2.png<br>
-image3.png<br>
-Video3<br>
-image1.png<br>
-image2.png<br>
-image3.png<br>
-Video4<br>
-image1.png<br>
-image2.png<br>
-image3.png</p>
<p>All labels-<br>
-Video1<br>
-image1.png<br>
-image2.png<br>
-image3.png<br>
-Video2<br>
-image1.png<br>
-image2.png<br>
-image3.png<br>
-Video3<br>
-image1.png<br>
-image2.png<br>
-image3.png<br>
-Video4<br>
-image1.png<br>
-image2.png<br>
-image3.png</p>
<p>How can I make a dataset? I already read the offical pytorch tutorial but that one uses a csv file as the labels but this time I have actual images as labels. Please help thanks.</p>

Alex_Dima: <p>Hello!I am quite new in pytorch and recently had to deal with sth like that. I am not sure if I understand your issue very well, but this is how I made my dataset:</p>
<pre><code class="lang-auto">#get all image and mask paths
image_paths = glob.glob("C:\\Users\\...\\data\\img\\*.jpg")
mask_paths = glob.glob("C:\\Users\\...\\data\\msk\\*.jpg")

#split paths
len_images = len(image_paths)
print(len_images)
train_size = 0.6

train_image_paths = image_paths [:int(len_images*train_size)]
test_image_paths = image_paths[int(len_images*train_size):]

train_mask_paths = mask_paths [:int(len_images*train_size)]
test_mask_paths = mask_paths[int(len_images*train_size):]

#custom dataset class

class image_dataset(Dataset):
    def __init__(self, images, masks, train=True):
        self.images = images
        self.masks = masks
      
    def __getitem__(self, idx):
        image = Image.open(self.images[idx])
        mask = Image.open(self.masks[idx])        
        return image,mask
        
    def __len__(self):
        return len(self.images)

</code></pre>
<p>In my case I only have the pictures and masks in one folder each, but I suppose there is a workaround for the subfolders. I hope this will be of help and not more confusing!Good luck!</p>
Efficient Data api

Selcuk_Caglar: <p>Hello. Dataloader can be extremely important for training speed.<br>
I was looking for a efficient data api and i have seen tensorflow data api that can load data into memory in different ways.<br>
<a href="https://www.tensorflow.org/guide/data_performance" class="onebox" target="_blank" rel="nofollow noopener">https://www.tensorflow.org/guide/data_performance</a></p>
<p>There isn’t any ready to use project for Pytorch.</p>

ptrblck: <aside class="quote no-group" data-username="Selcuk_Caglar" data-post="1" data-topic="88426">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/selcuk_caglar/48/17689_2.png" class="avatar"> Selcuk_Caglar:</div>
<blockquote>
<p>There isn’t any ready to use project for Pytorch.</p>
</blockquote>
</aside>
<p>I’m not sure to understand the claim.<br>
Are the <code>Dataset</code> and <code>DataLoader</code> utilities not working for your use case or which project is missing?</p>

Selcuk_Caglar: <p>Pytorch dataloader is working well. I am talking about more efficient dataloader. Dataloader is a bottleneck for training.<br>
(new)tensorflow data api can load data while optimizasyon process going on. Dataloading process doesn’t depend on training process. It look like asynchronously streaming.</p>

ptrblck: <p>The <code>DataLoader</code> implementation uses multiprocessing (specified by <code>num_workers</code>) to load the next batches in the background, while the training is executed, so the functionality should be similar.</p>
<p>However, if you have other suggestions, please add them to <a href="https://github.com/pytorch/pytorch/issues/41292">this feature request</a>.</p>