akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to view torch transformed images?

ooodragon: <p>Hi<br>
I’m using three torchvision transforms</p>
<pre><code class="lang-auto">transform_list = [transforms.RandomHorizontalFlip(p=1),
     transforms.ToTensor(),
     transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))]
</code></pre>
<p>and I want to view that horizontal flip IS indeed done, so in the training loop I did this</p>
<pre><code class="lang-auto">    for i, (images, _) in tqdm(enumerate(trainloader)):
        imshow(torchvision.utils.make_grid(images))
</code></pre>
<p>but it would only show only original images<br>
how can I view those augmented images?</p>
<p>one more question… (if I may…)<br>
I want to receive a ‘flag’ variable when a certain transformation (or augmentation) is done to that data…<br>
Any possible ideas? I will re-code &lt;torch.utils.data&gt; if I have to</p>

aradhyamathur: <p>Did you apply the transform in in the trainloader ?</p>

ooodragon: <p>yes, as you see in the code above<br>
I did apply random horizontal flip with probability of 1</p>

aradhyamathur: <p>I meant if you passed the transformation to the Dataset object as shown in the Transforms section for eg.  <code>transformed_dataset = FaceLandmarksDataset(csv_file='data/faces/face_landmarks.csv', root_dir='data/faces/', transform=transforms.Compose([ Rescale(256), RandomCrop(224), ToTensor() ]))</code><br>
as shown in more detail <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html" rel="nofollow noopener">here</a></p>

ooodragon: <p>Yes it was<br>
The list was made only for my convenience but it was wrapped woth compose and given to dataloader<br>
There no problem with it</p>

ooodragon: <p>OK… I thought horizontal flip was flipping on horizontal axis… hence the name horizontal flip<br>
should have noticed though, since horizontal flip was used ubiquitously… nobody wants his/her network to learn from upside down images…<br>
Random horizontal flip is flipping <em>horizontally</em>… (so in my sense, it is flipping on vertical axis)<br>
many peop might know… but I just figured out by comparing original images… haha</p>
Localization Information from a Convolutional Neural Network

collinbrake: <p>I am attempting to extract the pixel location of an object (a row of plants in my application) from an image. From this <a href="https://discuss.pytorch.org/t/if-input-does-not-have-any-locality-will-naive-dnn-better-than-cnn/47301">related topic</a> I have gathered that CNN’s are not designed to provide locality information about objects in an image, but rather to detect an object anywhere in an image by constructing a translation-invariant model.</p>
<p>Is there any way to tease this positional information out of a CNN, or is a fully-connected model the route to take? Could a CNN be trained to recognize an image with a vertical row of plants at, say column <span class="hashtag">#16</span>, as an entirely different object than an image with a row of plants at column <span class="hashtag">#244</span>? In this case I envision having a discrete number of possible classes that would be equal to the number of columns in the image, and the network would return 16 for the image with the row at column <span class="hashtag">#16</span>.</p>

ptrblck: <aside class="quote no-group" data-username="collinbrake" data-post="1" data-topic="85807">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/50afbb/48.png" class="avatar"> collinbrake:</div>
<blockquote>
<p>I am attempting to extract the pixel location of an object (a row of plants in my application) from an image.</p>
</blockquote>
</aside>
<p>This sounds rather like a segmentation use case, where each pixel will get a class prediction.</p>
<p><code>torchvision</code> provides a few segmentation models, which you could try out for your use case.<br>
Let me know, if I misunderstood your question.</p>

collinbrake: <p>Thanks ptrblck! Segmentation looks like the right method for my use case.</p>
Backward twice i am desperate

wongcheukwai: <p>Hi, my code has this kind of problem,RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time. But i just have one backward function. Here is my code. keys and values are initialised outside this loop as keys = torch.zeros(10, 512).cuda()<br>
values = (torch.ones(10, 10) / 10).cuda().</p>
<pre><code>for batch_idx, (noise_x, noise_y) in enumerate(noise_loader):
    try:
        clean_x, clean_y = clean_iter.next()
    except:
        clean_iter = iter(clean_loader)
        clean_x, clean_y = clean_iter.next()


    clean_x, clean_y = clean_x.cuda(), clean_y.cuda()
    noise_x, noise_y = noise_x.cuda(), noise_y.cuda()
    x_all = torch.cat((clean_x, noise_x),0)
    optimizer_n.zero_grad()

    features_clean, logits_clean = network(clean_x, get_feat=True)
    features_noise, logits_noise = network(noise_x, get_feat=True)

    print('check')
    #keys_temp1, values_temp1 = Mea.module(clean_y, features_clean, F.softmax(logits_clean, dim=1))

    keys, values = memory.module(clean_y, features_clean, F.softmax(logits_clean, dim=1), keys ,values)

    mem_pred = memory.assimilation(keys, values, features_noise)

    #loss_clean = CEloss(logits_clean, clean_y)
    loss_ME = CEloss(mem_pred, noise_y)
    #Loss_total = loss_clean + loss_ME
    #Loss_total.backward()

    loss_ME.backward()
    optimizer_n.step()</code></pre>

mayool: <p>You can set</p>
<p><code>loss_ME.backward(retain_graph=True)</code></p>
<p>but this might lead to memory issues, depending on your data.</p>

wongcheukwai: <p>i tried this one, it says run out of memory after a few iterations</p>

mayool: <p>I had the same error a couple of days ago, for me this was fixed using .detach().<br>
But I am not sure what you have to .detach()</p>
<p><a href="https://discuss.pytorch.org/t/convolutional-lstm-retain-graph-error/85200">this</a> is the post…Maybe it helps <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=9" title=":sweat_smile:" class="emoji" alt=":sweat_smile:"></p>
Accessor failed

leigaoxiang1: <p>auto srcdata = score.accessor&lt;float,3&gt;();</p>
<pre><code class="lang-auto">cout &lt;&lt; srcdata[0][0][0] &lt;&lt; endl;
for (int i = 0; i &lt; srcdata.size(0); i++) {
	int max_index = 0;
	float max_value = -1000;
	for (int j = 0; j &lt; srcdata.size(1); j++) {
		
		for (int k =0;k&lt; srcdata.size(2);k++)
		{
			cout &lt;&lt; "maxvalue:" &lt;&lt; srcdata[i][j][k] &lt;&lt; endl;
			if (srcdata[i][j][k] &gt; max_value) {
				max_value = srcdata[i][j][k];
				max_index = k;
			}

		}
</code></pre>
<p>but i get nothing by srcdata[i][j][k]</p>

ptrblck: <p>What do you mean by “nothing”? Is the <code>cout</code> not printing anything?<br>
This code snippet works for me:</p>
<pre><code class="lang-python">from torch.utils import cpp_extension

cpp_source = """
torch::Tensor my_fun(torch::Tensor x){
    auto srcdata = x.accessor&lt;float, 3&gt;();
    for (int i = 0; i &lt; srcdata.size(0); i++) {
        int max_index = 0;
        float max_value = -1000;
        for (int j = 0; j &lt; srcdata.size(1); j++) {
            for (int k =0;k&lt; srcdata.size(2);k++){
                std::cout &lt;&lt; "srcdata[" &lt;&lt; i &lt;&lt; "][" &lt;&lt; j &lt;&lt; "][" &lt;&lt; k &lt;&lt; "]: " &lt;&lt; srcdata[i][j][k] &lt;&lt; std::endl;
                if (srcdata[i][j][k] &gt; max_value) {
                    max_value = srcdata[i][j][k];
                    max_index = k;
                }
            }
        }
    }
    return x;
}
"""

module = torch.utils.cpp_extension.load_inline(
    name="test_extension",
    cpp_sources=cpp_source,
    functions="my_fun",
    verbose=True,
)

x = torch.randn(3, 3, 3)
out = module.my_fun(x)
print(out)
print('done')
</code></pre>

leigaoxiang1: <p>when i execute ```<br>
std::cout&lt;&lt;srcdata[i][j][k]&lt;&lt;endl:</p>
<pre><code class="lang-auto">i can not print nothing, why can you get the data in ```
srcdata and success get the value, is libtorch version?
</code></pre>

ptrblck: <p>Could you try to run my code and check, if you get valid outputs, please?<br>
The code should be executable in your Python IDE.<br>
I’m using the latest nightly binary.</p>

leigaoxiang1: <p>t have tried,your code is right, it can get the value in ```<br>
srcdata</p>
<pre><code class="lang-auto"></code></pre>

leigaoxiang1: <p>std::vector&lt;std::vector&gt; crnn_deocdeMat(torch::Tensor score, std::vector<a>std::string</a> alphabetChinese) {</p>
<pre><code>score = score.permute({1,0,2});
c10::IntList score_size = score.sizes();
cout &lt;&lt; "score_size:" &lt;&lt; score_size &lt;&lt; endl;
//const float* srcdata = reinterpret_cast&lt;const float *&gt;(score.data_ptr());
auto srcdata = score.accessor&lt;float,3&gt;();
cout &lt;&lt; ";;;;" &lt;&lt; srcdata[0][0][0] &lt;&lt; endl;
</code></pre>
<p>}<br>
cout &lt;&lt; “;;;;” &lt;&lt; srcdata[0][0][0] &lt;&lt; endl;this line print nothing,i dont know why,it si so strange</p>
Parallelism of a custom loop on GPU in python

a_d: <p>Hi all,<br>
I am trying to travel an image with a 50x50 kernel, infer on that 50x50 sub-image image and draw a bounding box corresponding to the class.<br>
essentially something like -</p>
<pre><code class="lang-auto">for i in range(image.height):
    for j in range (image.width):
        out = model(image[:, : , i-25 : i+25, i-25 : i+25])
</code></pre>
<p>where inference would be on the GPU, is there any way I could parallelize this process in python or use the PyTorch jit, something similar to the parallel_for in C++…</p>
<p>TIA</p>

ptrblck: <p>I think your best bet is to create the patches beforehand (e.g. using <code>unfold</code>), push these patches into the batch dimension, and try to perform the forward pass once.<br>
Afterwards, you could reshape the output and process each patch using your bounding box approach.</p>
How to add at specific but possibly repeating indices

dhananjayraut: <p>I want to do something like</p>
<pre><code class="lang-python">a = torch.tensor([0.0, 0.0, 0.0, 0.0, 0.0])
b = torch.tensor([1, 2, 2, 4, 0, 1, 3])
c = torch.tensor([0.3, 0.2, 0.4, 0.1, 1.0, 2.0, 0.4])

a[b] += c
</code></pre>
<p>expected result : tensor([1.0000, 2.0000, 0.6000, 0.4000, 0.1000])<br>
actual result:       tensor([1.0000, 2.0000, 0.4000, 0.4000, 0.1000])</p>
<p>at index 2 it is retaining last value 0.4 instead of adding 0.2 and 0.4<br>
I want to do this without creating onehot vectors of b and then using scatter_ as it is taking too much memory.</p>

ptrblck: <p><code>a.put_(b, c, accumulate=True)</code> should work.<br>
I assume <code>a[1]</code> should have the value 2.3 after the addition.</p>
Labels unevenness problems--data preprocessing

samr1104: <p>In many datasets, I got a problem like there are different numbers of data in different labels, how do I reinforce the data to reduce this kind of unevenness so that model predictions won’t bias to the labels of more numbers??</p>

ptrblck: <p>You could use a weighted loss as described e.g. in the <code>nn.CrossEntropyLoss</code> <a href="https://pytorch.org/docs/stable/nn.html?highlight=crossentropyloss#torch.nn.CrossEntropyLoss">docs</a> or alternatively you could use over/undersampling using a <code>WeightedRandomSampler</code>.<br>
<a href="https://discuss.pytorch.org/t/how-to-handle-imbalanced-classes/11264/2">Here</a> is an example of its usage.</p>
Hi how can i interpolate an image by scipy

Golam_Moctader_Ronok: <p>Hi how can i interpolate an image by scipy</p>

ptrblck: <p><a href="https://docs.scipy.org/doc/scipy/reference/generated/scipy.interpolate.interp2d.html"><code>scipy.interpolate.interp2d</code></a> might work.<br>
Note that this question is not related to PyTorch so you might get faster and better answers on StackOverflow or a dedicated forum for other libraries. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
How to implement permanent weight dropout in a layer?

mrpatekful: <p>Hi! I am trying to implement a weight dropout mechanism, that drops weights from a feed forward layer and keeps them zero during training. I am wondering, what is the best way to do this?</p>
<pre><code class="lang-python">class SparseLinear(torch.nn.Module):
    """
    Linear layer that uses sparse dense weights.
    """

    def __init__(self, in_features, out_features, sparsity):
        super().__init__()

        w = torch.empty(out_features, in_features)
        torch.nn.init.sparse_(w, sparsity=sparsity)
        self.w = torch.nn.Parameter(w)

        b = torch.empty(1, out_features)
        torch.nn.init.sparse_(b, sparsity=sparsity)
        self.b = torch.nn.Parameter(b)

        self.w_mask = self.w == 0
        self.b_mask = self.b == 0

    def forward(self, inputs):
        return torch.nn.functional.linear(
            inputs,
            torch.masked_fill(self.w, self.w_mask, 0.0),
            bias=torch.masked_fill(self.b, self.b_mask, 0.0))

</code></pre>
<p>This is my current solution, however I don’t like that it performs masking on every forward call, I would rather prefer a solution that prevent the optimizer from modifying them in the first place.</p>

ptrblck: <p>I think you would have to mask these parameters at some point (either directly or their gradients to prevent an update).<br>
The optimizer cannot take parts of tensors, so I’m not aware of any other clean approach. <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"></p>
RuntimeError: unexpected EOF, expected 10700742 more bytes. The file might be corrupted

sjt524: <p>The full information</p>
<p>Traceback (most recent call last):<br>
File “main.py”, line 17, in <br>
trainset = datasets.MNIST(‘data’, train=True, download=False, transform=transform)<br>
File “/home/sjt524/anaconda3/lib/python3.7/site-packages/torchvision/datasets/mnist.py”, line 80, in <strong>init</strong><br>
self.data, self.targets = torch.load(os.path.join(self.processed_folder, data_file))<br>
File “/home/sjt524/anaconda3/lib/python3.7/site-packages/torch/serialization.py”, line 593, in load<br>
return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)<br>
File “/home/sjt524/anaconda3/lib/python3.7/site-packages/torch/serialization.py”, line 780, in _legacy_load<br>
deserialized_objects[key]._set_from_file(f, offset, f_should_read_directly)<br>
RuntimeError: unexpected EOF, expected 10700742 more bytes. The file might be corrupted.</p>
<p>Ubuntu 16.04.5 LTS (GNU/Linux 4.15.0-43-generic x86_64)</p>
<p>Python 3.7.0 (default, Jun 28 2018, 13:15:42)<br>
Type “copyright”, “credits” or “license” for more information.<br>
IPython 6.5.0 – An enhanced Interactive Python.</p>

ptrblck: <p>Probably the download of the MNIST dataset failed.<br>
Could you delete the MNIST files in <code>data</code> and download the dataset again, please?</p>
What happens to precision of Pytorch tensors from GPU when converted to Numpy array CPU?

maxcompression: <p>I am not sure when I convert a Pytorch tensor into a numpy array, whether the precision of the Pytorch tensor is maintained in the Numpy array. What precision is a standard Pytorch nn layer at? When I use the code below, do I keep the same number of decimals? Even when I set the print options of both Pytorch and Numpy to as high as possible, it seems that the Numpy arrays have lower precision. I then tried to convert the Pytorch tensors to np.float128, which makes it more precise, but still not sure whether I am capturing the full precision of the Pytorch tensors.</p>
<p>Below are the two options I tried. I want to make sure my Numpy CPU array captures at least the same precision as my Pytorch GPU tensor that corresponds to the activation values of neurons in a layer of a FFNN.</p>
<p>First option, that gives Numpy float 32:</p>
<pre><code class="lang-auto">activationvector.detach().cpu().numpy()
</code></pre>
<p>Second option, forcing the numpy array to be float 128. It seemed to make a difference whether I used float 64 or float 128, which probably means numpy float 64 didn’t capture full precision, maybe float 128 doesn’t do that either…</p>
<pre><code class="lang-auto">np.array(activationvector.detach().cpu(),dtype=np.float128)
</code></pre>
<p>I hope someone can tell me how to make sure my Numpy CPU array captures at least the same precision as my Pytorch GPU tensor</p>
<p>Thanks in advance!</p>

ptrblck: <p>The numpy array and PyTorch tensor should be bitwise accurate, as they are sharing the same underlying data as seen here:</p>
<pre data-code-wrap="python"><code class="lang-python">x = torch.tensor(1/3)

print('{:.20f}'.format(x))
print('{:.20f}'.format(x.numpy()))
print('{:.20f}'.format(x.numpy().astype(np.float64)))

y = x.numpy()
y += 0.1

print('{:.20f}'.format(x))
print('{:.20f}'.format(x.numpy()))
print('{:.20f}'.format(x.numpy().astype(np.float64)))
</code></pre>
<aside class="quote no-group" data-username="maxcompression" data-post="1" data-topic="84739">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/85e7bf/48.png" class="avatar"> maxcompression:</div>
<blockquote>
<p>I then tried to convert the Pytorch tensors to np.float128, which makes it more precise, but still not sure whether I am capturing the full precision of the Pytorch tensors.</p>
</blockquote>
</aside>
<p>Upcasting the data should not make the result more precise, as the original value was already represented in the lower precision.</p>

maxcompression: <p>Thanks a lot for the clarification!</p>
Does Layer normalization delay NN's learning?

shwe87: <p>Hello,</p>
<p>First, I tested two simple models for ASR in Spanish:</p>
<p><strong>Model 1:</strong></p>
<pre><code>- Layer Normalization 
- Bi-directional GRU
- Dropout
- Fully Connected layer
- Dropout
- Fully Connected layer as a classifier (classifies one of the alphabet chars)
</code></pre>
<p><strong>Model 2:</strong></p>
<pre><code>  - Conv Layer 1
  - Conv Layer 2
  - Fully Connected
  - Dropout
  - Bidirectional GRU
  - Fully connected layer as a classifier
</code></pre>
<p>I tried with 30 epochs because I have limited resources of GPU.<br>
The validation and training loss for these two models:<br>
<a href="https://i.stack.imgur.com/oaKEj.png" rel="nofollow noopener"><img src="https://i.stack.imgur.com/oaKEj.png" alt="enter image description here" width="631" height="310"></a><br>
Model 1 performed not so good as expected.<br>
Model 2 worked too well, after 20 epochs, it started overfitting (please see the graph in the notebook results) and in the output, I could actually see some words creating which seems like the labels. Although it is overfitting, it still needs training because it doesn’t predict the total outcome. For start, I am happy with this model.</p>
<p>I tested a third complex Model.</p>
<p><strong>Model 3:</strong></p>
<pre><code>  - Layer Normalization
  - RELU
  - Bidirectional GRU
  - Dropout 
  - Stack this 10 times more.
</code></pre>
<p>The valid loss and training loss for this model:<br>
<a href="https://i.stack.imgur.com/fvpHC.png" rel="nofollow noopener"><img src="https://i.stack.imgur.com/fvpHC.png" alt="enter image description here" width="625" height="310"></a><br>
I tested this on 30 epochs and there were no good results, actually, all the predictions were blank…</p>
<p>Is this because this complex model needs more epochs for training?</p>
<p><strong>Update:</strong></p>
<p>I modified the model by adding 2 convolutional layer before the stacked GRU and the model seems to have improved but the validation loss and training loss just doesn’t seem to improve at all. I tried even with 100 epochs and didn’t work.</p>
<p>I see that in the first model and the third model I applied layer normalization and both’s prediction seems to be very bad…Does layer normalization makes the learning delay? But according to papers like:<br>
<a href="https://www.arxiv-vanity.com/papers/1607.06450/" rel="nofollow noopener">https://www.arxiv-vanity.com/papers/1607.06450/</a> layer normalization speeds up the training and also helps in speeding the training loss. So, I am really confused. I have limited resources of GPU and I am not sure if I should go for another try without layer normalization…</p>
How to implement squared hinge loss

xolotl18: <p>Hi everyone,<br>
I need to implement the squred hinge loss in order to train a neural network using a svm-like classifier on the last layer.<br>
It is an image classification problem on cifar dataset, so it is a multi class classification. The idea is to have multiple “scores” for each output neuron of the network and the network should be trained minimizing the sum of the losses for each class.<br>
Is there an already implemented version of this loss? If not, what is the easiest way to implement it?<br>
Thank you!</p>

xolotl18: <p>Hi, in the end I implemented a version of the multi label squared hinge loss.<br>
It makes sense in my head, but it goes to zero after the first epoch. And I can’t figure out why.<br>
I used batches of size 64 from the dataloader.<br>
the labels are one hot encoded<br>
here is the code</p>
<p>class MultiClassSquaredHingeLoss(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(MultiClassSquaredHingeLoss, self).<strong>init</strong>()</p>
<pre><code>def forward(self, output, y): #output: batchsize*n_class
    n_class = y.size()[1]
    #margin = 1 
    margin = 1
    #isolate the score for the true class
    y_out = torch.sum(torch.mul(output, y)).cuda()
    output_y = torch.mul(torch.ones(n_class).cuda(), y_out).cuda()
    #create an opposite to the one hot encoded tensor
    anti_y = torch.ones(n_class).cuda() - y.cuda()
    
    loss = output.cuda() - output_y.cuda() + margin
    loss = loss.cuda()
    #remove the element of the loss corresponding to the true class
    loss = torch.mul(loss.cuda(), anti_y.cuda()).cuda()
    #max(0,_)
    loss = torch.max(loss.cuda(), torch.zeros(n_class).cuda())
    #squared hinge loss
    loss = torch.pow(loss, 2).cuda()
    #sum up
    loss = torch.sum(loss).cuda()
    loss = loss / n_class        
    
    return loss
</code></pre>
<p>I would really appreciate your help</p>

ptrblck: <p>How does the model output look compared to the target, if you reach the zero loss?<br>
I haven’t checked the implementation carefully, but could you use a sample output and target to check the validity of the loss?</p>

xolotl18: <p>hi, thank you for answering.<br>
I think I messed up something concerning the fact that outputs come in batches, because changing some stuff it gives me problems with output and target dimensions.</p>
The order betweent import torch and import pandas does matter

jianjunwu: <p>hi, I encounter following error if I import torch first and then import pandas:<br>
<em>ImportError  GLIBCXX_3.4.21  not found</em><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/0/3087c3d42964181388f4ed49a4f3989977756be9.png" data-download-href="https://discuss.pytorch.org/uploads/default/3087c3d42964181388f4ed49a4f3989977756be9" title="1"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/0/3087c3d42964181388f4ed49a4f3989977756be9_2_690x188.png" alt="1" data-base62-sha1="6VjNcP20huCVeG9nZdjvoGHitFD" width="690" height="188" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/0/3087c3d42964181388f4ed49a4f3989977756be9_2_690x188.png, https://discuss.pytorch.org/uploads/default/original/3X/3/0/3087c3d42964181388f4ed49a4f3989977756be9.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/3/0/3087c3d42964181388f4ed49a4f3989977756be9.png 2x" data-dominant-color="0E0C0C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">1</span><span class="informations">1030×282 123 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
however, if I import pandas first and then import torch, everything is ok.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/b/fbadc40b917691ba40236ae412211f67985eccab.png" data-download-href="https://discuss.pytorch.org/uploads/default/fbadc40b917691ba40236ae412211f67985eccab" title="2"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/b/fbadc40b917691ba40236ae412211f67985eccab_2_690x138.png" alt="2" data-base62-sha1="zUs8t4wATGHEt8nfSzgccnxYtRh" width="690" height="138" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/b/fbadc40b917691ba40236ae412211f67985eccab_2_690x138.png, https://discuss.pytorch.org/uploads/default/original/3X/f/b/fbadc40b917691ba40236ae412211f67985eccab.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/f/b/fbadc40b917691ba40236ae412211f67985eccab.png 2x" data-dominant-color="160E0E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">2</span><span class="informations">720×144 41.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
can someone give some suggestions for fix the error thoroughly?<br>
my os is <em>CentOS7.4.1708</em><br>
the version of pytorch is 1.5.0+cpu and anaconda is Anaconda3-2019.03-Linux-x86_64.</p>
Super slow training in ResNet18 in V100 GPU

farhanrahman: <p>Hello,</p>
<p>I have been training a Resnet18 network on synthetic dataset of about 350,000 images. Each of the image has a label, all the labels are stored in a text file. I have created a simple custom Dataset class that loads one image at a time inside the getitem function. The label file is just read once in the init function and converted to a tensor (since its small).</p>
<p>The trouble is one epoch would take me about 45+ minutes in a system with a V100 GPU. As far as I am concerned, is it not a bit too much? Can it be due to a slow dataloading? Please share your opinions on this.</p>
<p>Thanks</p>

dhananjayraut: <p>your speed is 129 images/sec which is lower than I expect see <a href="https://developer.nvidia.com/deep-learning-performance-training-inference" rel="nofollow noopener">https://developer.nvidia.com/deep-learning-performance-training-inference</a> the pytorch resnet50 model does inference at 905 images/sec on a single V100. I will try increasing batch size as resnet18 is fairly small mode for a V100 and number of workers for dataloader.</p>

farhanrahman: <p>Thanks Dhananjay for your time to reply! Those were my intuitions as well. I tried playing with the num_workers parameters as well, however playing around with it does not seem to change much so I have kept it between 8/16 or 32. But as I said, it did not seem to matter much.</p>
<p>My batch size is already 64, I do not want to make it larger than that in case that makes it harder for the network to generalize (As far as I understand). Is there anything else that can solve this issue for me? Please note that the images are loaded inside the get_item function inside the Dataset class.</p>
ONNX export for the TUNIT model

Sayak_Paul: <p>I am trying to export the [<code>tunit</code>] model to ONNX format.</p>
<p>The model consists of two subnetworks for inference - an encoder and a generator. The encoder spits out the style representation for a reference image and the generator takes in that style representation and a source image to produce an image that is transformed into the domain of the reference image. Following is an example:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/b/4bfbb0b644665405b32e44137fd45d2a7c300072.png" alt="image" data-base62-sha1="aQb3mr80Rb0ci72PbRdgABJLIVY" width="384" height="145"></p>
<p>However, the generator does not implement <code>forward</code> and which is I am probably not able to export it using <code>onnx</code>. Here’s a <a href="https://colab.research.google.com/gist/sayakpaul/d79acb411b0d68a27711db069d49f4cc/scratchpad.ipynb" rel="noopener nofollow ugc">Colab Gist</a> that demos the inference process. Please note that I used the <code>animalFaces10_0_00</code> pre-trained checkpoints. I first copied the files from [here] (<a href="https://drive.google.com/drive/folders/1rU1B9OLQjYMBzU6VQX7UwLxod2WzOfNz?usp=sharing" rel="noopener nofollow ugc">https://drive.google.com/drive/folders/1rU1B9OLQjYMBzU6VQX7UwLxod2WzOfNz?usp=sharing</a>) to my personal Drive, created a folder called <code>animalFaces10_0_00</code>, and copied the files to that folder.</p>

Sayak_Paul: <p>As I am a new user, I wasn’t able to supply more than two links. Here’s the GitHub repo of the <code>tunit</code> model: <a href="https://github.com/clovaai/tunit" rel="nofollow noopener">https://github.com/clovaai/tunit</a>.</p>
A little code problem

caokx: <p>a = torch.tensor([4.], requires_grad=True)<br>
print(a)<br>
b = a.pow(3)<br>
print(b)<br>
b.backward()<br>
print(a.grad)</p>
<p>tensor([4.], requires_grad=True)<br>
tensor([64.], grad_fn=)<br>
tensor([48.])</p>
<p>is the a.grad=3?</p>

harsha_g: <p>Not sure what exactly the question is. But, read <code>a.grad</code> as the derivative of <code>b</code> with respect to <code>a</code> which in this case is <code>3*(4^2) = 48</code>.</p>

ziqipang: <p>I was just wanting to reply the same. Agree with you!</p>
In DataLoader num_workers > 1 shuffles data even when shuffle=False

qwwq215: <p>I am observing this phenomena in PyTorch 0.4.0. Setting num_workers &gt; 1 and shuffle = False, is shuffling the data. I think this is counterintuitive to the users of PyTorch and changes should be made so that num_workers does not shuffle the data.</p>

SimonW: <p>i don’t think it shuffles data. where did you see this?</p>

qwwq215: <p>Code snippet to reproduce the issue.</p>
<pre><code class="lang-auto">import torch
import torchvision
import torchvision.transforms as transforms

def load_data():
    
    transform = transforms.Compose([
        transforms.ToTensor(),
        transforms.Lambda(lambda z: print(z.min(), z.max()))
    ])

    dataset = torchvision.datasets.ImageFolder('/home/ssd0/images/', transform=transform)
    dataloader = torch.utils.data.DataLoader(dataset, batch_size=16, shuffle=False, num_workers=4)
    
    return dataloader

for data in load_data():
    pass
</code></pre>
<p>The above code ends in error because <code>transforms.Lambda(lambda z: print(z.min(), z.max()))</code> returns <code>NoneType</code>. But if you carefully see the prints, it displays 16 torch tensors of min and max value of each image of the first batch. And this keeps changing every time I run the code. Setting num_workers=1 resolves this problem. I suspect 4 parallel workers are racing against each other. Please tell me if I am doing something wrong.</p>

yz3007: <p>I faced the exactly same problem with you and did the same experiment as you did. It showed that workers cowork in the same batch. Shuffle=False means that the batch order (the big data chunk) is stable, howerver, the datapoints in batch is randomized by several subprocess. You can set num_workers = 0 which means that only main process deals it. By this way, the batch order and the data order in batch are both stable.</p>
How to solve this error?

MisBah_Awan: <p>kindly anyone help me how to solve this problem.i have  try many solutions but no one works…</p>
<p><code>IndexError: invalid index of a 0-dim tensor. Use tensor.item() to convert a 0-dim tensor to a Python number</code></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/8/8fae4e260f47d2766b8e8119280d005ab08eed9b.png" data-download-href="https://discuss.pytorch.org/uploads/default/8fae4e260f47d2766b8e8119280d005ab08eed9b" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/2X/8/8fae4e260f47d2766b8e8119280d005ab08eed9b.png" alt="image" data-base62-sha1="kv3IZacsNHn6RbbtH8rYoEQVj7B" width="690" height="99" data-dominant-color="F4F3F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1339×194 13.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<pre><code class="lang-auto">for i in range(4):
            label = labels[i]
            class_correct[label] += c[i].item()
            class_total[label] += 1
</code></pre>

Xiaoyu_Song: <p>Hi, could you provide more details about your <code>class_correct</code>, and  how is <code>c</code> defined</p>

MisBah_Awan: <p>here is my code.</p>
<pre><code class="lang-auto">class_correct = list(0. for i in range(3))
class_total = list(0. for i in range(3))
with torch.no_grad():
    for data in testloader:
        images, labels = data
        images = Variable(images)
        labels = Variable(labels)
        outputs = model(images)
        _, predicted = torch.max(outputs, 1)
        c = (predicted == labels).squeeze()
        for i in range(10):
            label = labels[i]
            class_correct[label] += c[i].item()
            class_total[label] += 1
            #print('Accuracy of %5s : %2f %%' % (classes[i], 100 * class_correct[i] / class_total[i]))


# =============================================================================
# for i in range(10):  
#     print('Accuracy of %5s : %2f %%' % (
#         classes[i], 100 * class_correct[i] / class_total[i]))
</code></pre>

Yang_TIAN: <p>check the batch size of testdata loader？<br>
testloader = torch.utils.data.DataLoader(dataset=testset,<br>
batch_size=4,<br>
shuffle=False)<br>
in case you set it in batch_size=1</p>
'aten::slow_conv_transpose2d' not support in 'QuantizedCPUTensorID'

zhijian_li: <p>OS: Win7 64bit<br>
Pytorch 1.5.0_CPU<br>
when I try to quantize an unet model, meet the error below:</p>
<pre><code class="lang-auto">RuntimeError: Could not run 'aten::slow_conv_transpose2d' with arguments from the 'QuantizedCPUTensorId' backend. 'aten::slow_conv_transpose2d' is only available for these backends: [CPUTensorId, VariableTensorId].
</code></pre>
<p>Is there any way to workaround this?</p>

khizar-anjum: <p>Had the exact same problem. I worked around this by inserting <code>torch.quantization.DeQuantStub</code> and <code>torch.quantization.QuantStub</code> before and after the <code>ConvTranspose2d</code> layer. I don’t know if this affects performance or anything.</p>
<p>Just take a look at the following class I converted:</p>
<pre><code class="lang-auto">from torch.quantization import DeQuantStub, QuantStub
class UpsamplerBlock (nn.Module):
    def __init__(self, ninput, noutput):
        super(UpsamplerBlock, self).__init__()
        self.conv = nn.ConvTranspose2d(ninput, noutput, 3, stride=2, padding=1, output_padding=1, bias=True)
        self.bn = nn.BatchNorm2d(noutput, eps=1e-3)
        self.quant = QuantStub()
        self.dequant = DeQuantStub()

    def forward(self, input):
        output = self.conv(self.dequant(input))
        output = self.bn(self.quant(output))
        return F.relu(output)
</code></pre>

jerryzh168: <p>yeah, quantized conv transpose 2d is not supported yet, but <a class="mention" href="/u/zafar">@Zafar</a> is working on it right now</p>