akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Adding Native Pruning Support

berserkr: <p>Hi All,</p>
<p>I want to add support for pruning whereby we can add hints to a layer, say some flag called ‘pruned’, so that instead of doing the convolution, we just format the output and then pass a zero tensor. This is to basically skip doing unnecessary ops on pruned weights.</p>
<p>Is there some type of architecture document that covers call graphs or how python invokes the native C++ functions for different architectures. I am interested in understanding what are the entry points for the different calls. For example, if I want to edit the C++ code for the convolution operation, which is the entry point? There are a bunch of Conv implementations but without guidance it is hard to keep track.</p>
<p>Thank you!</p>

tom: <p>A PyTorchy way of doing this would be to work on a module basis, e.g. using the brand new FX facility or a JIT representation and mutate the module / JIT graph.</p>
<p>That said, the outmost C++ entry points are <a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/Convolution.cpp" rel="noopener nofollow ugc">convXd</a> found via native_functions.yaml. <a href="https://lernapparat.de/selective-excursion-into-pytorch-internals/" rel="noopener nofollow ugc">I once blogged about some details of how Python maps to C++</a>, it’s old, but save the location of derivatives it’s still mostly accurate.</p>
<p>Best regards</p>
<p>Thomas</p>

berserkr: <p>Thank you for your response. I’ve looked at that entry point, however, for some reason I am not hitting the C++ methods I expect when running the model on my build. So I was wondering if there was some detailed doc on Pytorch architecture. Will take a look at our blog. Cheers!</p>
What's the largest model you train with PyTorch?

imaginary: <p>Hello, can you folks please let me know what’s the largest model you train with PyTorch?<br>
I want to measure power consumption with &amp; without a few tweaks in the PyTorch source code.<br>
I’m motivated by the ACL 2019 paper <a href="https://arxiv.org/abs/1906.02243" rel="noopener nofollow ugc">Energy and Policy Considerations for Deep Learning in NLP</a>.<br>
Thank you!</p>
Feed network from intermediate layers (and train these layers)

AlexAlonso: <p>Hi,</p>
<p>I am using a detection network. I would like to use a model that calculates 3 features vectors and then use them in the model head. I want to split the model in features and head so I can run them separately. Right now, I can extract the features vector using hooks, but I am not sure how can I feed those features in the head to compute the final output. In tensorflow you can just pass a dict with the input variables to the model and compute the prediction… An extra complication is the features are not in consecutive layer, so each goes to different modules in the head.</p>
<p>Thanks a lot,<br>
Alex</p>
Understand pytorch optimization, weight decay

zezo: <p>I want to understand how weight decay (L2 penalty) is working:<br>
`torch.optim.Adam(<em>params</em> , <em>lr=0.001</em> , <em>betas=(0.9</em> , <em>0.999)</em> , <em>eps=1e-08</em> , <em>weight_decay=0</em> , <em>amsgrad=False</em> ).<br>
How L2 penalty actually work during optimization?</p>

googlebot: <p>It is more obvious in older (1.6) sources, weight_decay only affects one line:</p>
<blockquote>
<p>d_p = d_p.add(p, alpha=weight_decay)</p>
</blockquote>
<p>i.e. gradient = gradient + param_value * weight_decay<br>
so, as param_value deviates from zero, taken steps increase</p>

zezo: <p>So, it is changing constraining the current gradient with the current parameters?</p>

googlebot: <p>Um, of course “current” values are used. Mathematically, effect of this is the same as of usual L2 decay, it is just applied via gradient.</p>
Memory friendly transformers over (too) long time series

bluesk: <p>I am doing time series classification and I want to experiment with transformers. Given that transformers’ memory usage grows as square of the input length, the whole sequence would be much grater than the maximum input size of the classifier. So I am trying to come up with ways to encode disjoint sections of the sequence first, save these encodings into a tensor and then feed the tensor to the classifier.</p>
<p>This is a very naive way to do it:</p>
<pre><code class="lang-python">encoder = MyEncoder() # encodes a sequence of size window size into a scalar (for simplicity)
classifier = MyClassifier() # classifies a sequence of size m &gt;= n_windows

seq = torch.tensor([..my pretty long batch sequence ..])

window_size = ... some integer
n_windows = len(seq) // window_size
encoded_seq = torch.zeros(n_window)
for i in range(n_windows):
      window = seq[i*window_size : (i+1)*window_size)]
      encoded_seq[i] = encoder(window)

output = classifier(encoded_seq)
</code></pre>
<p>Is there a more efficient way to do this?</p>
<p>I thought of using <code>tensor.apply_</code> but that too seems slow.</p>
com.facebook.jni.CppException: [enforce fail at inline_container.cc:222] . file not found: archive/constants.pkl

Dosapati_Nikhil: <p>Hello Everyone. I was able to load the .pt file in python. But when I tried to load the same model in Java Android I was facing this issue.<br>
Android Java Version:<br>
‘org.pytorch:pytorch_android:1.6.0’<br>
‘org.pytorch:pytorch_android_torchvision:1.7.0’<br>
Python Version:<br>
pytorch:1.6</p>
<pre><code> Caused by: com.facebook.jni.CppException: [enforce fail at inline_container.cc:222] . file not found: archive/constants.pkl
(no backtrace available)
    at org.pytorch.NativePeer.initHybrid(Native Method)
    at org.pytorch.NativePeer.&lt;init&gt;(NativePeer.java:23)
    at org.pytorch.Module.load(Module.java:24)
    at org.pytorch.Module.load(Module.java:34)
    at org.pytorch.helloworld.MainActivity.onCreate(MainActivity.java:39)
    at android.app.Activity.performCreate(Activity.java:7136)
    at android.app.Activity.performCreate(Activity.java:7127)
    at android.app.Instrumentation.callActivityOnCreate(Instrumentation.java:1271)
</code></pre>
<p>So please let me know,  do I need to make any changes in Java Code while loading the model because I was able to load the sample model provided by the PyTorch Android team(version 1.4.0).</p>
<p>Is it related to versioning of pytorch? And Is pytorch java and pytorch versioning similar?</p>

IvanKobzarev: <p>The version of gradle ‘org.pytorch:pytorch_android:1.6.0’ should be the same for pytorch binaries that you used to generate pt file.</p>
<p>Did you use 1.6.0 version for pt file?</p>

windmaple: <p>I’m facing the exact same issue. However, I suspect mine is related to quantized model. I’m using PyTorch 1.7.1 to generate the model:</p>
<blockquote>
<p>import torchvision, torch<br>
model = torchvision.models.quantization.mobilenet_v2(pretrained=True, quantize=True)<br>
torch.save(model.state_dict(), “model.pt”)</p>
</blockquote>
<p>In Android app, I use:</p>
<blockquote>
<p>dependencies {<br>
implementation ‘androidx.appcompat:appcompat:1.1.0’<br>
implementation ‘org.pytorch:pytorch_android:1.7.1’<br>
implementation ‘org.pytorch:pytorch_android_torchvision:1.7.1’<br>
}</p>
</blockquote>
<p>I got:</p>
<blockquote>
<p>Process: org.pytorch.helloworld, PID: 8644<br>
java.lang.RuntimeException: Unable to start activity ComponentInfo{org.pytorch.helloworld/org.pytorch.helloworld.MainActivity}: com.facebook.jni.CppException: [enforce fail at inline_container.cc:222] . file not found: archive/constants.pkl<br>
(no backtrace available)</p>
</blockquote>
<p>Is there something to be done to run quantized model?<br>
I also see the same error w/ Android benchmark binary and iOS benchmark app.</p>
<p><a class="mention" href="/u/ivankobzarev">@IvanKobzarev</a> any advice on this?</p>

windmaple: <p>Never mind. I figured out my mistake.</p>

zerotosingularity: <p>Facing the same issue, could you share what fixed it for you?</p>
<p>Thanks!</p>
<p>Update: I’ve used PyTorch 1.6 for both save and load but the error remains.</p>
Still the solutions on converting Torch::tensor to cv::Mat won't work

siavash.one: <p>Hi everyone,</p>
<p>I have been following the topics on converting Torch::tensor to cv::Mat but they don’t yield the exact result I am expecting.</p>
<p>So I have trained a network with a number of batches (16) and now I want to evaluate it with single image. The network is supposed to output another image and I think it has been trained well for that. The image I’m trying to feed in is a gray-scale 256x256:</p>
<pre><code class="lang-auto">cv::Mat img = cv::imread("\path\to\img", cv::IMREAD_UNCHANGED);
    cv::normalize(img, img, 0.0, 1.0, cv::NORM_MINMAX, CV_32F);

    torch::Tensor myTensor = torch::from_blob(img.data, {img.rows, img.cols, 1}, torch::kFloat32);
    myTensor = myTensor.permute({2, 0, 1});
    myTensor = myTensor.view({1, 1, 256, 256});
    myTensor = myTensor.to(device);
</code></pre>
<p>And then I feed it to my network,get the output tensor, and try to save the image as CV_8UC1 as follows:</p>
<pre><code class="lang-auto">torch::Tensor out_tensor = MyCNN-&gt;forward(myTensor);
    out_tensor = out_tensor.permute({ 0, 2, 3, 1 });
    out_tensor = out_tensor.squeeze(0).detach();
    out_tensor = out_tensor.detach().cpu().mul(255).clamp(0,255).to(torch::kByte);

    cv::Mat resultImg(256, 256, CV_8UC1);
    std::memcpy((void*)resultImg.data, out_tensor.data_ptr(), sizeof(torch::kByte)*out_tensor.numel());

    imwrite("resultImg.tiff", resultImg);
</code></pre>
<p>It does produce and save an image but it’s nowhere close to what I expect, looks more like a white woven fabric!</p>
<p>One thing I should mention is that the lines for reading in an image from HDD is exactly the same as the way I wrote the custom dataloader for training except the following line:</p>
<pre><code class="lang-auto">myTensor = myTensor.view({1, 1, 256, 256});
</code></pre>
<p>which is obviously is not used for training since we train with batches.</p>
<p>Any input or help will be greatly appreciated.</p>
<p>Thanks!</p>

jbschlosser: <p><code>out_tensor</code> is not guaranteed to be contiguous. I’d try using <code>ravel()</code> to get a contiguous tensor for the call to <code>memcpy()</code>.</p>
PyTorch gather 3D source with 2D index

dvirginz: <p>Given a <strong>source</strong> tensor of shape <code>[B,N,F]</code> and an index tensor of shape <code>[B,k]</code>, where <code>index[i][j]</code> is an index to a specific feature inside <code>source[i][j]</code> is there a way to extract an output tensor such that:</p>
<pre><code class="lang-auto">output[i][j] = source[i][j][index[i][j]]
</code></pre>
<p><code>torch.gather</code> specifies that <code>index.shape == source.shape</code>, while here the shape of the source is one dimension bigger.</p>
<pre><code class="lang-auto">source = [
[[0.1,0.2],[0.2,0.3]],
[[0.4,0.5],[0.6,0.7]],
[[0.7,0.6],[0.8,0.9]]
]
index = [
[1,0],
[0,0],
[1,1]
]

desired_output = [
[0.2,0.2],
[0.4,0.6],
[0.6,0.9]
]
</code></pre>

dvirginz: <p>For future references - The solution is</p>
<pre><code class="lang-auto">source.gather(2,index.unsqueeze(2)).squeeze(2)
</code></pre>
Improving performance neuroevolution

witl: <p>I’m doing neuroevolution. Since I would only be doing inference with the model, and manipulating the models’ weight “manually”, do I need to do something else but declaring <a href="https://pytorch.org/docs/stable/autograd.html" rel="noopener nofollow ugc">torch.autograd.set_grad_enabled(False)</a> to avoid unnecessary computations? Is there anything else that I can do to improve performance?</p>
<p>Another point is that currently, I’m using Python, for convenience. I’m also a C++ developer so believe that in this situation (reinforcement learning scenarios) C++ would provide much better performance since it is required much computation out of the Pytorch framework. Is there anyone out there with experience (or material) to share with me? I never used Pytorch API with C++ btw.</p>
torch.trapz gives slightly different answer from coding naively

LintonVision: <p>If I need to integrate over 100,000 samples, I get a slightly different answer whether I code naively or using torch.trapz:</p>
<pre><code class="lang-auto">import torch  # Import PyTorch

samples = 100000
x = torch.linspace(-5, 5, samples)
y = x**2

integral = 0  # Initialise integral = 0
for i in range(0, samples-1):
    integral = integral + (0.5*(y[i]+y[i+1])*(x[i+1]-x[i]))

integral2 = torch.trapz(y,x)

print(integral-integral2)
print(torch.isclose(integral, integral2))
</code></pre>
<p>Does anyone know why this is, and how to fix it? Thanks!</p>

tom: <p>One thing I always recommend when doing this sort of experiment is to check whether it is more likely to be numerical accuracy or an algorithmic difference.<br>
So if you add the <code>dtype=torch.double</code> argument to the <code>torch.linspace</code> call, you’ll see that the difference goes down significantly (for me from 9e-4ish to 7e-14ish).<br>
This points to the algorithms computing the same result but perhaps having a different order for the additions, which typically leads to some discrepancy due to numerical precision.</p>
<p>Best regards</p>
<p>Thomas</p>

LintonVision: <p>That’s awesome Thomas! Thanks so much for the reply. It’s a really helpful tip I didn’t know about, and really appreciate it <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Train Accuracy decreses after some epochs

Kapil_Rana: <p>After running my model multiple time,</p>
<p>the train accuracy and test accuracy both decreasing after 4th epoch.<br>
how it is possible, what could be reasons?</p>
<p>Train Accuracy<br>
0,==Acc==,45.29<br>
1,==Acc==,67.1<br>
2,==Acc==,74.84<br>
3,==Acc==,80.25<br>
4,==Acc==,83.77<br>
5,==Acc==,56.99<br>
6,==Acc==,46.62<br>
7,==Acc==,46.06<br>
8,==Acc==,46.92<br>
9,==Acc==,46.66<br>
10,==Acc==,47.72<br>
11,==Acc==,48.04<br>
12,==Acc==,46.24<br>
13,==Acc==,48.41<br>
0,==Acc==,6.37<br>
14,==Acc==,29.57<br>
15,==Acc==,47.54<br>
16,==Acc==,47.54<br>
17,==Acc==,46.96<br>
0,==Acc==,10.82<br>
18,==Acc==,47.54</p>
<p>Test Accuracy<br>
0,==Acc==,45.28<br>
1,==Acc==,71.76<br>
2,==Acc==,79.92<br>
3,==Acc==,80.61<br>
4,==Acc==,94.31<br>
5,==Acc==,46.17<br>
6,==Acc==,47.49<br>
7,==Acc==,47.49<br>
8,==Acc==,46.17<br>
9,==Acc==,48.86<br>
10,==Acc==,47.49<br>
11,==Acc==,47.47<br>
12,==Acc==,47.48<br>
13,==Acc==,43.71<br>
14,==Acc==,47.47<br>
15,==Acc==,47.49<br>
16,==Acc==,47.49<br>
17,==Acc==,47.49<br>
18,==Acc==,47.49.</p>
<p>I am using a simple custom crossentropyloss<br>
def custom_categorical_cross_entropy(y_pred, y_true):<br>
y_pred = torch.clamp(y_pred, 1e-9, 1 - 1e-9)<br>
return -(y_true * torch.log(y_pred)).sum(dim=1).mean()</p>
<p>again if I use<br>
def custom_categorical_cross_entropy(y_pred, y_true):<br>
return -(y_true * torch.log(y_pred)).sum(dim=1).mean()</p>
<p>loss is coming  NaN.</p>
<p>One more tried LogSoftmax and NLLLoss(). there alse after some epoch loss is NaN.</p>
<p>What could be possible issue.</p>

ptrblck: <p>There might be different reasons for the divergence, such as an exploding loss.<br>
I would also not recommend to reimplement already provided loss functions, if the custom ones don’t add any new functionality (e.g. soft labels etc.), as e.g. <code>nn.CrossEntropyLoss</code> uses internally the log-sum-exp trick for numerical stability, which is often missing in custom implementations.</p>

Kapil_Rana: <aside class="quote no-group" data-username="Kapil_Rana" data-post="1" data-topic="114571">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kapil_rana/48/20951_2.png" class="avatar"> Kapil_Rana:</div>
<blockquote>
<p>LogSoftmax and NLLLoss()</p>
</blockquote>
</aside>
<p>I have also tried  LogSoftmax and NLLLoss(). In that case alse loss is coming as NaN.<br>
My question is , is it problem with data or the model. or anything else?</p>

Kapil_Rana: <p>I am using logsoftmax and NLLLoss.<br>
My training accuracy decreased after 88 epoch.<br>
In 88 epoch : Accuracy is 51 %<br>
Suddenly in 89 epoch accuracy is 1.53 %</p>
Build PyTorch from source

jr.r: <p>Hi, while I am building PyToch from source, I got the following error:</p>
<p>$python setup.py install<br>
nvcc fatal   : The version (‘80000’) of the host compiler (‘clang’) is not supported</p>
<p>$clang --version<br>
clang version 8.0.0-3~ubuntu18.04.2 (tags/RELEASE_800/final)<br>
Target: x86_64-pc-linux-gnu<br>
Thread model: posix<br>
InstalledDir: /usr/bin</p>
<p>Thank you</p>
How to create series of training examples from a custom dataset?

torc: <p>For datasets that come integrated with PyTorch, this is very easy.</p>
<p>In the case of MNIST, doing this is enough-</p>
<pre><code class="lang-auto">from torchvision import datasets

train_data = datatsets.MNIST('.', train=True, download=True)

x_train, y_train = train_data.data, train_data.targets
</code></pre>
<p>But how do I create such series from custom datasets that I create using classes subclassing from <code>torch.utils.data.Dataset</code>?</p>
<p>I have a dataset having a series of tuples of tensors and labels. The tensors are shaped 180*180 and the labels are integers.</p>
<pre><code class="lang-auto">&gt;&gt;&gt; dataset[0]
(tensor([[1.5628, 1.5679, 1.5588,  ..., 1.6395, 1.6355, 1.6354],
         [1.5106, 1.5402, 1.5627,  ..., 1.5813, 1.6235, 1.6520],
         [1.5924, 1.6069, 1.5967,  ..., 1.5813, 1.5924, 1.5964],
         ...,
         [1.5945, 1.6138, 1.6241,  ..., 1.6181, 1.6243, 1.6018],
         [1.6006, 1.6283, 1.6591,  ..., 1.6047, 1.6047, 1.6161],
         [1.6181, 1.6181, 1.6129,  ..., 1.5833, 1.5679, 1.6110]]),
 5)
</code></pre>
<p>How do I go, from here, to create say, x of <code>torch.Size([5000, 180, 180])</code>, and y of <code>torch.size([5000])</code>?</p>
<p>I then want to create a <code>TensorDataset</code> from here, from where to finally form a DataLoader.</p>
<p>This is a newbie question, and I cannot find the answer online.</p>

ptrblck: <p>I’m not completely sure I understand the question correctly, so feel free to update the description a bit and correct me, if I’m misunderstanding it.<br>
In case you want to access the complete dataset and are preloading it in the <code>__init__</code> or are passing it directly to it, you should be able to access the internal attributes via e.g. <code>dataset.data</code>.<br>
On the other hand, if you want to create batches of 5000 samples, you can pass this dataset to a <code>DataLoader</code> with a <code>batch_size=5000</code>, which should then return the desired shapes.</p>

torc: <p>I actually thought that I should create a TensorDataset to create DataLoader from it.</p>
<p>But, turns out, I can create a DataLoader directly from the dataset.</p>
Is there a standard way in pytorch to compare convergence speed / efficiency between models?

mtian2018: <p>Say, if I have two models, both have 4 inputs and 2 outputs and one hidden layer, the first model  has 128 nodes, the second 64 nodes. How do I know which is better, in terms of speed and variance, etc. in a quantative way? The models could have two or more layers as well.</p>

Firas_moh: <p>please, any answer to this question?</p>
Install pytorch on jetson nano

ky_Pa: <p>Hi,<br>
I encountered this problem after installing pytorch1.1.0, how can I solve it?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/6/162c46e1f800f946f5eecec821535fa62939953d.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/162c46e1f800f946f5eecec821535fa62939953d" title="微信图片_20210321201734"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/6/162c46e1f800f946f5eecec821535fa62939953d_2_666x500.jpeg" alt="微信图片_20210321201734" data-base62-sha1="3a9mLhDOiLn7p1dn60JWcRZ7EM5" width="666" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/6/162c46e1f800f946f5eecec821535fa62939953d_2_666x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/1/6/162c46e1f800f946f5eecec821535fa62939953d_2_999x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/1/6/162c46e1f800f946f5eecec821535fa62939953d_2_1332x1000.jpeg 2x" data-dominant-color="84617B"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">微信图片_20210321201734</span><span class="informations">1440×1080 187 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
jetson nano, unbuntu, cuda10.2, python3.6</p>

ptrblck: <p>I would recommend to update to the latest PyTorch release, as you might hit an already fixed issue in the older <code>1.1.0</code> release.</p>

ky_Pa: <p>I installed the 1.8 version of pytorch, and the same problem still occurs “illegal instruction”.</p>

eniolasonowo: <p>Nvidia has several containers for pytorch.<br>
Check them out at <a href="http://ngc.nvidia.com" rel="noopener nofollow ugc">ngc.nvidia.com</a></p>
Only one element tensors can be converted

Phuc_Tan: <p>I cacth this error, please help me<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/0/b05998f6ec27b872a610f25ed0d8a5da185bee3c.png" data-download-href="https://discuss.pytorch.org/uploads/default/b05998f6ec27b872a610f25ed0d8a5da185bee3c" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/0/b05998f6ec27b872a610f25ed0d8a5da185bee3c_2_690x289.png" alt="image" data-base62-sha1="pa3ZMUBvaBPv7i9po3kw7o874xC" width="690" height="289" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/0/b05998f6ec27b872a610f25ed0d8a5da185bee3c_2_690x289.png, https://discuss.pytorch.org/uploads/default/optimized/3X/b/0/b05998f6ec27b872a610f25ed0d8a5da185bee3c_2_1035x433.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/b/0/b05998f6ec27b872a610f25ed0d8a5da185bee3c_2_1380x578.png 2x" data-dominant-color="242424"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1865×783 77.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/0/a0e5033118898268d91e6c0897d2a586e1328c43.png" data-download-href="https://discuss.pytorch.org/uploads/default/a0e5033118898268d91e6c0897d2a586e1328c43" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/0/a0e5033118898268d91e6c0897d2a586e1328c43_2_690x225.png" alt="image" data-base62-sha1="mXl3eJPj6PX6xOI59O6KEOvHUt5" width="690" height="225" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/0/a0e5033118898268d91e6c0897d2a586e1328c43_2_690x225.png, https://discuss.pytorch.org/uploads/default/optimized/3X/a/0/a0e5033118898268d91e6c0897d2a586e1328c43_2_1035x337.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/a/0/a0e5033118898268d91e6c0897d2a586e1328c43_2_1380x450.png 2x" data-dominant-color="39393A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1826×597 64 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>The <code>item()</code> operation can only convert a single element to Python scalars as given in the error message, while the result of <code>get_token_high_prob</code> returns multiple elements.</p>
<pre><code class="lang-python">x = torch.randn(2)
x.item() # error
&gt; ValueError: only one element tensors can be converted to Python scalars

x = torch.randn(1)
x.item() # works
</code></pre>
<p>You would thus either have to make sure the mentioned function returns a single value or you would have to remove the <code>item()</code> operation and handle the result differently.</p>
Train for 'n' output channels and test it on 'n-1' output channels in Neural network

bibinwilson: <p>I trained a Convolutional Neural Network with U-Net architecture for a regression problem. I have 20 output channels. After training, I would like to use the trained weights to test it on a dataset with 19 output channels. Is it possible?</p>
<p>Basically, I’m trying to check the output r^2 value of my model if I avoided different output channels. For that, I would like to train the network with all the 20 output channels and then use that inference for the same data set with 19 output channels or 18 output channels up to 1 output channel so that I can note down the effect of different channels r^2.</p>

ptrblck: <p>Would you like to change the output channels of the last conv layer only?<br>
If so, then you could manipulate the filters (and bias) directly via:</p>
<pre><code class="lang-python">with torch.no_grad():
    model.last_conv.weight = nn.Parameter(model.last_conv.weight[:-1])
    ...
</code></pre>
<p>or any other operation besides the slicing.</p>
<p>Manipulating intermediate conv layers would be a bit more tricky, since you would also have to change the <code>in_channels</code> of the successive conv layer.</p>
Segfault when running on 3080

k2lin: <p>Hi</p>
<p>I am running a 3080 GPU on Ubuntu 20.04 LTS with NVIDIA driver 460.39.<br>
For some reason, my training program would randomly receive segmentation fault.<br>
I did multiple tries but could not pinpoint the cause of this issue.<br>
At first I was using nightly version but later switched to 1.8.0 (using CUDA11.1 installed with conda) and the issue persists.<br>
I tested the code on 1080Ti, 2080Ti and RTX Titan. All of them run perfectly fine except the 3080.</p>
<p>Here is a screenshot of the gdb result:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/3/43247d74604491f2e7998971d66dfa8fdb7e22f9.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/43247d74604491f2e7998971d66dfa8fdb7e22f9" title="圖片"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/3/43247d74604491f2e7998971d66dfa8fdb7e22f9_2_690x241.jpeg" alt="圖片" data-base62-sha1="9zYaxTbeA1Y4hjl65R00uqpUFlL" width="690" height="241" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/3/43247d74604491f2e7998971d66dfa8fdb7e22f9_2_690x241.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/4/3/43247d74604491f2e7998971d66dfa8fdb7e22f9_2_1035x361.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/3/43247d74604491f2e7998971d66dfa8fdb7e22f9_2_1380x482.jpeg 2x" data-dominant-color="2A0F1C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">圖片</span><span class="informations">1851×649 378 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>It also sometimes gives “illegal instruction” errors.</p>
<p>Could anyone help me with this?<br>
Thanks.</p>

ptrblck: <p>Do you get a stack trace for the illegal instruction?<br>
Could you also post an executable code snippet to reproduce this issue using random inputs, so that we could debug it?</p>

k2lin: <p>Unfortunately I did not have the stack trace for the illegal instruction error.<br>
This error is a bit random.<br>
I will keep this post updated when I find a way to reproduce it consistently.<br>
Thanks.</p>
Batching with multiple inputs

annamae: <p>I have data that consists of a sequence of word-part of speech pairs. I currently have it set up where each pair is represented as a list of two tensors, one tensor with the one-hot for the word and the other with the one-hot for the part of speech. Each sentence is a list of lists. I use dataloader on this data. I am trying to implement batching in training but when try to train, I get this error:<br>
“TypeError: embedding(): argument ‘indices’ (position 2) must be Tensor, not list”</p>
<p>I am not sure how to represent my data in a way where I can implement batching. I am also happy to provide code.</p>
<p>Thanks so much!</p>

Dwight_Foster: <p>Can you provide the dataloader code and also what are the shapes of the two tensors in the list?</p>
How to feed data through from an LSTM to a Linear layer

Vlad_Dumitru: <p>I am trying to build a reinforcement-learning system via OpenAI that is going to perform actions such BUY, SELL, HOLD a stock from the market.</p>
<p>From now I am trying to implement an actual NN that will be used by my agent but I have some problems regarding how to shape the date correctly.</p>
<pre><code class="lang-auto">
class NeuronalNetwork(nn.Module):
    def __init__(self, input_size, stock_env: StockEnv):
        super(NeuronalNetwork, self).__init__()
        self.stock_env = stock_env
        hidden_dim = 256
        input_size = len(self.stock_env.normalized_dataframe.columns)
        self.hidden_size = 128
        self.num_layers = 1
        output_size = self.stock_env.action_space.n
        self.lstm = nn.LSTM(input_size=input_size, hidden_size=self.hidden_size, num_layers=self.num_layers,
                            batch_first=True)
        self.output_layer = nn.Linear(hidden_dim, output_size)
        self.tanh = nn.Tanh()

    def forward(self, x, hidden=None):
        # N x T x D
        # N - the number of windows sizes
        # T - the window size
        # D - the number of indicators and OHLCV in total
        if hidden is None:
            hidden = (
                torch.zeros(self.num_layers, 1, self.hidden_size).to(device),
                torch.zeros(self.num_layers, 1, self.hidden_size).to(device),
            )
        D = len(self.stock_env.normalized_dataframe.columns)
        T = self.stock_env.window_size
        N = 1
        x = x.view(N, T, D).type(torch.FloatTensor).to(device)
        out, (ht, ct) = self.lstm(x, hidden)
        out = self.tanh(out)
        out = self.output_layer(out)
        return out, hidden
</code></pre>
<p>I am having a pandas dataframe with my Open, High, Low, Close, Volume price data which is my <strong>input_size</strong> and my output_size will be <strong>3</strong> (BUY, HOLD, SELL) because I want to predict the next falling action.</p>
<p>I’m not sure what I am doing wrong but I am getting the following error:</p>
<p><code>mat1 dim 1 must match mat2 dim 0</code> when I am trying to run <code>self.output_layer(out)</code></p>
<p>I’m a newbie here, could anyone of you explain please what I am doing wrong?</p>

Dwight_Foster: <p>The reason you are getting this error is because the number of input neurons in your linear layer is less than the number of inputs to the layer. You need to change the size to match the output size of your lstm. Can you print the shape of the lstm output doing this</p>
<pre><code class="lang-auto">        x = x.view(N, T, D).type(torch.FloatTensor).to(device)
        out, (ht, ct) = self.lstm(x, hidden)
        out = self.tanh(out)
        print(out.shape)
        out = self.output_layer(out)
</code></pre>
<p>whatever that outputs you should change the number of input neurons in you linear layer to.</p>
Large numerical error using trapz

smonsays: <p>I need to perform some basic numerical integrations in pytorch and am currently investigating feasible methods. As a validity check for using the new <code>torch.trapz</code> function, I implemented the <a href="https://uk.mathworks.com/help/matlab/ref/trapz.html#buakefe-1_1" rel="nofollow noopener">Matlab example on multiple numerical integration</a>:</p>
<pre><code class="lang-python">import torch

dx = 0.1
x = torch.arange(-3, 3, dx)
y = torch.arange(-5, 5, dx)
X, Y = torch.meshgrid(x, y)

F = X**2 + Y**2

result = torch.trapz(torch.trapz(F, dx=dx, dim=0), dx=dx, dim=0)
# Returns tensor(646.9882)
</code></pre>
<p>The result is quite far away from what Matlab produces (680.200) and the exact value of 680. Is this expected? If yes, why so?</p>
<p>What would be a simple alternative to using <code>torch.trapz</code>?</p>

LintonVision: <p>I also found that torch.trapz gives a slightly wrong estimate in:</p>
<pre><code class="lang-auto">import torch  # Import PyTorch

samples = 100000
x = torch.linspace(-5, 5, samples)
y = x**2

integral = 0  # Initialise integral = 0
for i in range(0, samples-1):
    integral = integral + (0.5*(y[i]+y[i+1])*(x[i+1]-x[i]))

integral2 = torch.trapz(y,x)

print(integral-integral2)
print(torch.isclose(integral, integral2))
</code></pre>
Compute forward on every variation of my inputs, to get my backward

barthelemymp: <p>Hello,</p>
<p>I am implementing a new energy-based model. My input is sequential and categorical (with a small number of categories). Right now my simple forward map every sequence to a real number. Unfortunately, I want to compute something complicated, a pseudolikelihood.<br>
To do so I need for every sequence to compute the forward of every sequence which are different by one element from my input sequence, then use these values to renormalize with a softmax for every position.</p>
<p>Mainly my question is the following: How to create a function which for every element in my batch will create a temporary batch of all the variation of my sequence so that I can apply my energy computation + renormalization. It is a kind of “sub_batch” for every element in the batch.</p>
<p>Here is a simple numpy implementation of how I would do it in basic numpy with for loops:</p>
<pre><code class="lang-auto">energy_variation = np.zeros(batch_size, sequence_length, number_categories)
for seq_index in range(batch_size):
    seq = batch[seq_index]
    for i in range(sequence_length):
        for cat in range(number_categories):
            temposeq = copy(seq)
            temposeq[i] = cat
            energy_variation[seq_index, i , cat] = self.forward(temposeq)
        energy_variation[seq_index, i , :] = softmax(energy_variation[seq_index, i , :])
</code></pre>
<p>Btw my loss is computed on the results of some value of this energy_variation matrix, meaning it needs to backpropagate through all the forward pass.</p>
<p>Mainly how can I do something efficient in Pytorch for this problem.</p>
<p>Best,</p>
<p>Barthélémy</p>
Hidden layers in MLP

90210: <p>Hi! I am struggling to understand the basic idea of “hidden layers” in an MLP.</p>
<p>Given the following network:</p>
<pre><code class="lang-auto">class Net(torch.nn.Module):
    def __init__(self, num_inputs, num_hidden, num_outputs):
        super(Net, self).__init__()
        self.num_inputs = num_inputs
        self.num_hidden = num_hidden
        self.num_outputs = num_outputs
        
        self.lin = nn.Linear(num_inputs, num_hidden)
        self.lin2 = nn.Linear(num_hidden, num_hidden)
        self.lin3 = nn.Linear(num_hidden, num_hidden)
        self.lin4 = nn.Linear(num_hidden, num_outputs)
        self.ReLU = nn.ReLU()

    def forward(self, x):
        out = self.lin(x)
        out = self.ReLU(out)
        out = self.lin2(out)
        out = self.ReLU(out)
        out = self.lin3(out)
        out = self.ReLU(out)

        return self.lin4(out)
</code></pre>
<p>Do we have 2 hidden layers here, or 3?</p>
<p>How would a network with 2 hidden layers look/work?</p>
Accuracy of linear model

90210: <p>Hi! I am trying to figure out how I can calculate the accuracy of a linear regression model, but can’t seem to think of any way to do so.</p>
<p>Given that I have the <code>y</code> values and the <code>y_hat</code> predictions from my Linear model, how can I calculate the accuracy? Note that the dimensions of each <code>y</code> in a batch is [53, 1] and each <code>y_hat</code> is also the same.</p>
<p>Thank you!</p>

AbdulsalamBande: <p>I do not know of anything like accuracy of a regression model. Instead, you can try using Loss functions like Mean Squared Error (MSE). Example</p>
<pre><code class="lang-auto">&gt;&gt;&gt; loss = nn.MSELoss()
&gt;&gt;&gt; input = torch.randn(3, 5, requires_grad=True)
&gt;&gt;&gt; target = torch.randn(3, 5)
&gt;&gt;&gt; output = loss(input, target)
</code></pre>
<p>In your case input  is <code>[53,1]</code> and target is also <code>[53,1]</code> as you expalined.<br>
Source  <a href="https://pytorch.org/docs/stable/generated/torch.nn.MSELoss.html" rel="noopener nofollow ugc">MSE in Pytorch</a></p>
<p>Hope it helps</p>
SentencePiece Tokenizer

J_Johnson: <p>I’m trying to understand how to properly use the <code>generate_sp_model</code> output as a tokenizer.</p>
<p>A simplified coding example is as follows:</p>
<pre><code class="lang-auto">import torch
import io
import csv
from torchtext.data.functional import generate_sp_model, load_sp_model, sentencepiece_tokenizer, sentencepiece_numericalizer
from collections import Counter
from torchtext.vocab import Vocab

list_a = ["sentencepiece encode as pieces", "examples to try!"]

with open('sample.csv', 'w', newline='', encoding='utf-8') as f:
    writer=csv.writer(f)
    writer.writerows(list_a)

generate_sp_model('sample.csv',vocab_size=21,model_prefix='sample')
vocab_tokenizer = load_sp_model('sample.model')  #spmodel is a tokenizer
sp_tokens_generator=sentencepiece_tokenizer(sp_model=vocab_tokenizer)
sp_id_generator= sentencepiece_numericalizer(vocab_tokenizer)

print(list(sp_id_generator(list_a)))
print(list(sp_tokens_generator(list_a)))

def build_vocab(filepath, tokenizer):
    counter = Counter()
    with io.open(filepath, encoding="utf8") as f:
        for string_ in f:
            counter.update(tokenizer(string_))
    return Vocab(counter, specials=['&lt;unk&gt;', '&lt;pad&gt;', '&lt;bos&gt;', '&lt;eos&gt;'])

en_vocab=build_vocab('sample.csv', vocab_tokenizer)
</code></pre>
<p>The print outs are fine. It just doesn’t seem to like being called as a tokenizer. The error given:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "scratches/scratch_42.py", line 33, in &lt;module&gt;
    en_vocab=build_vocab('sample.csv', vocab_tokenizer)
  File "scratches/scratch_42.py", line 30, in build_vocab
    counter.update(tokenizer(string_))
TypeError: 'torch._C.ScriptObject' object is not callable
</code></pre>
<p>Does that error basically mean I should not be using this <code>generate_sp_model</code> function directly?</p>
<p><a href="https://pytorch.org/text/stable/data_functional.html#torchtext.data.functional.generate_sp_model" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/text/stable/data_functional.html#torchtext.data.functional.generate_sp_model</a></p>

J_Johnson: <p>Okay, figured it out. I changed</p>
<p><code>counter.update(tokenizer(string_))</code></p>
<p>to:</p>
<pre><code class="lang-auto">for elem in list(tokenizer([string_])):
    counter.update(elem)
</code></pre>
<p>And also:<br>
<code>en_vocab=build_vocab('sample.csv', vocab_tokenizer)</code></p>
<p>to:</p>
<p><code>en_vocab=build_vocab('sample.csv', sp_tokens_generator)</code></p>
<p>That allowed me to then save the vocab model with:</p>
<p><code>torch.save(en_vocab, 'sample_vocab')</code></p>
torch.optim.lr_scheduler.CyclicLR

Lincoln10153: <p>Hello everyone!</p>
<p>I have a question related to the function torch.optim.lr_scheduler.CyclicLR. There is a parameter called step_size_up. During the training process, I only set the parameter step_size_up and did not set step_size_down. However, I found that during training, both learning rate increasing and learning rate decreasing exist.</p>
<p>What I understand is that if I only set the parameter step_size_up, there should be not learning rate decreasing.</p>
<p>Could you explain why learning rate decreasing exists?</p>
<p>Thanks!</p>
Error: "trainloader" and "writer" are not defined, when it has been defined, in tutorial on viz model via tensorboard

oat: <p>I’m following the <a href="https://pytorch.org/tutorials/intermediate/tensorboard_tutorial.html" rel="noopener nofollow ugc">PyTorch official tutorial on how to viz model via TensorBoard</a> in my notebook <a href="https://github.com/hopezh/pytorch_official_tutorials/blob/main/viz_model_via_tensorboard.ipynb" rel="noopener nofollow ugc">here</a>.</p>
<p>(Note that I copied the exact code from the tutorial webpage)</p>
<ol>
<li>However, I continued to get the following error on the section to write image to tensorboard, although obviously “trainloader” has been defined in the first cell.</li>
</ol>
<pre><code class="lang-auto">---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
&lt;ipython-input-1-6d95240dcdaa&gt; in &lt;module&gt;
      1 # get some random training images
----&gt; 2 dataiter = iter(trainloader)
      3 images, labels = dataiter.next()
      4 
      5 # create grid of images

NameError: name 'trainloader' is not defined
</code></pre>
<ol start="2">
<li>After I re-ran the first cell to create the trainloader twice, the error points to the writer object, which obviously has been created in previous cell.</li>
</ol>
<pre><code class="lang-auto">NameError: name 'writer' is not defined
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
&lt;ipython-input-5-6d95240dcdaa&gt; in &lt;module&gt;
     10 
     11 # write to tensorboard
---&gt; 12 writer.add_image('four_fashion_mnist_images', img_grid)
NameError: name 'writer' is not defined
</code></pre>
<p>Appreciate if you can advise what could have caused this problem.</p>

ptrblck: <p>Based on the errors I guess your Jupyter notebook might have trouble executing the cells somehow?<br>
Are you seeing the same issues when you run the script from a terminal (outside of a notebook)?</p>
Computational cost, clamp vs pooling

L-Reichardt: <p>Hello,<br>
i am currently looking for some input regarding computational costs.</p>
<p>What would be computationally more efficient?:</p>
<ul>
<li>using 3x3 max pooling on a single channel tensor to create a new variable</li>
<li>or using clamping on a different single channel tensor to create a new variable</li>
</ul>
<p>The dimensions of both tensors are equal. If possible, please provide reasoning <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

ptrblck: <p>Note that the theoretical computational cost might differ from the used implementation, so a proper way would be to profile both workloads via e.g. <code>torch.utils.benchmark</code>.</p>

L-Reichardt: <p>Thank you <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
That function will also be useful to me in the future for other projects.</p>
Add different size of the tensor

111186: <p>x1.size()-&gt; torch.Size([128, 512])<br>
x2.size()-&gt; torch.Size([128, 476])</p>
<p>x = torch.add(x1, x2)</p>
<p>RuntimeError: The size of tensor a (512) must match the size of tensor b (476) at non-singleton dimension 1</p>

ptrblck: <p>You cannot directly add two differently shapes tensors, which contain a different number of elements, as the addition would be performed elementwise.<br>
Could you explain how these two tensors should be added theoretically?<br>
To illustrate your idea, you could lower the shapes to e.g. <code>[2, 3]</code> and <code>[2, 4]</code> to explain it easier.</p>
TypeError: load_state_dict() got an unexpected keyword argument 'strict'

ginobilinie: <p>When I try to finetune a model, I set the ‘strict’ argument to load the state_dict, however, an error appears:</p>
<p>“TypeError: load_state_dict() got an unexpected keyword argument ‘strict’”</p>
<p>My pytorch is 1.2.0, can anyone help me with it?</p>

ginobilinie: <p>Anyone can help, thanks?</p>

ptrblck: <p>The <code>strict</code> argument was already added to <a href="https://pytorch.org/docs/1.2.0/nn.html#torch.nn.Module.load_state_dict"><code>1.2.0</code></a> so could you post a code snippet to reproduce this error?</p>

ginobilinie: <p>Thank you.</p>
<p>Here is the code:<br>
train_model.load_state_dict(<br>
<span class="hashtag">#torch</span>.load(os.path.join(final_output_path, config.model_prefix + str(curr_iter) + ‘.pth’)), resume=True) # this works fine<br>
torch.load(os.path.join(final_output_path, config.model_prefix + str(curr_iter) + ‘.pth’)),strict=False) <span class="hashtag">#this</span> doesnot work and give the error I list above.</p>

ptrblck: <p>Your code is unfortunately not executable and I cannot reproduce the error in PyTorch <code>1.2.0</code>:</p>
<pre><code class="lang-python">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.2.0'
&gt;&gt;&gt; import torch.nn as nn
&gt;&gt;&gt; model = nn.Linear(1, 1)
&gt;&gt;&gt; model.load_state_dict(nn.Linear(1, 1).state_dict(), strict=False)
&lt;All keys matched successfully&gt;
&gt;&gt;&gt; model.load_state_dict(nn.Linear(1, 1).state_dict(), strict=True)
&lt;All keys matched successfully&gt;
</code></pre>
How gradients are accumulated in real

linzhu1206: <p>‘Gradient will not be updated but be accumulated, and updated every N rounds.’ I have a question that how the gradients are accumulated in the below code snippet: in every round of the below loop I can see a new gradient is computed by loss.backward() and should be stored internally, but would this internally stored gradient be refreshed in the next round? How the gradient is summed up, and later be applied every N rounds?</p>
<pre><code class="lang-auto">for i, (inputs, labels) in enumerate(training_set):
    predictions = model(inputs)                     # Forward pass
    loss = loss_function(predictions, labels)       # Compute loss function
    loss = loss / accumulation_steps                # Normalize our loss (if averaged)
    loss.backward()                                 # Backward pass
    if (i+1) % accumulation_steps == 0:             # Wait for several backward steps
        optimizer.step()                            # Now we can do an optimizer step
        model.zero_grad()  
</code></pre>

crcrpar: <p>Hi,</p>
<p>gradients are accumulated until <code>model.zero_grad()</code> is called.</p>
<p>A toy is notebook is here: <a href="https://gist.github.com/crcrpar/a35cd05cb01a8304b6a87da05956c69e" class="inline-onebox" rel="noopener nofollow ugc">how-gradients-are-accumulated-in-real.ipynb · GitHub</a></p>

linzhu1206: <p>Thanks a lot for the clarification and the code! Before coming to this gradient accumulation topic, it`s easy for beginner like me to feel that loss.backward ‘sets’ the grad but not ‘accumulate’.</p>
Custom Dataset to take Images from Various Folders (not based on class)

Adam_Dhalla: <p>Hi All,</p>
<p>I’m wondering how I should augment the <code>Dataset</code> class so that I can make a dataset with images from different folders? <strong>These folders are not based on classes</strong>(which is why I’m not using ImageFolder) - I just needed to store the images in different folders since the amount of files was too large to store in one folder and was causing input/output errors.</p>
<p>I want to specify a few folder locations, and then make a dataset object that contains all images from those folders in one. I am getting the class labels separately by indexing into a csv file.</p>
<p>Any way on doing this?</p>
<p>Thank you!<br>
A</p>

crcrpar: <p>Hi,</p>
<p>I think the snippet below will do. Note that I’ve not tested this.</p>
<pre><code class="lang-python">
class Dataset(ImageFolder):

    def __init__(self, labels, *args_for_ImageFolder, **kwargs_for_ImageFolder):
        super().__init__(*args_for_ImageFolder, **kwargs_for_ImageFolder)
        self.labels = labels

    def __getitem__(self, index):
        return (super().__getitem__(index)[0], self.labels[index])
</code></pre>