I cannot access my dataset instance after call torch.utils.data.random_split

username: Jim_Xu
cooked: <p>Hi ,as the topic said, this is my code:</p>
<p>import imutils<br>
import cv2<br>
from imutils import paths<br>
import torch<br>
import torch.nn as nn<br>
import torch.optim as optim<br>
from torch.optim import lr_scheduler<br>
import numpy as np<br>
import torchvision<br>
from torchvision import datasets, models, transforms<br>
import matplotlib.pyplot as plt<br>
import time<br>
from PIL import Image<br>
import os<br>
import copy<br>
if <strong>name</strong> == ‘<strong>main</strong>’:<br>
# Data augmentation and normalization for training<br>
# Just normalization for validation<br>
data_transforms = {<br>
transforms.Compose([<br>
transforms.Resize((224,224)),<br>
transforms.ToTensor()<br>
]),</p>
<pre><code>}

data_dir = './pill'

image_datasets = datasets.ImageFolder(data_dir,
                                          data_transforms)

dataloaders = torch.utils.data.DataLoader(image_datasets, batch_size=8,
                                             shuffle=True, num_workers=8)

train_size = int(0.8 * len(image_datasets))
test_size = len(image_datasets) - train_size
train_dataset, test_dataset = torch.utils.data.random_split(image_datasets, [train_size, test_size])
</code></pre>
<p>I can get the length by len(train_dataset.dataset), while I can not get train_dataset.dataset[0], this will cause an error:</p>
<p>TypeError                                 Traceback (most recent call last)<br>
 in <br>
----&gt; 1 train_dataset.dataset[0]</p>
<p>~\anaconda3\Lib\site-packages\torchvision\datasets\folder.py in <strong>getitem</strong>(self, index)<br>
138         sample = self.loader(path)<br>
139         if self.transform is not None:<br>
–&gt; 140             sample = self.transform(sample)<br>
141         if self.target_transform is not None:<br>
142             target = self.target_transform(target)</p>
<p>TypeError: ‘set’ object is not callable</p>
<p>Could you help me solve this problem? Thanks a lot!!</p>

username: ptrblck
cooked: <p>The error doesn’t seem to be raised by the usage of <code>random_split</code> and should also be raised, if you try to index <code>image_datasets</code> directly, as the passed transformation seems to be defined in a wrong way.<br>
While you are using:</p>
<pre><code class="lang-python">data_transforms = {
transforms.Compose([
transforms.Resize((224,224)),
transforms.ToTensor()
]),
}
</code></pre>
<p>which will create a <code>set</code> object.<br>
Remove the curly brackets and rerun the code.</p>
<p>PS: you can post code snippets by wrapping them into three backticks, which makes debugging easier. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

username: Jim_Xu
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="91936">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>backticks</p>
</blockquote>
</aside>
<p>Thanks a LOT !!! I will make a try now</p>

username: Jim_Xu
cooked: <p>Thank you so much for your last response!! The data could be accessed, while in the training phase, I meet another problem, could you tell me how may I fix it?</p>
<h2>Epoch 0/24</h2>
<hr>
<p>RuntimeError                              Traceback (most recent call last)<br>
 in <br>
1 model_ft = train_model(model_ft, criterion, optimizer_ft, exp_lr_scheduler,<br>
----&gt; 2                        num_epochs=25)</p>
<p> in train_model(model, criterion, optimizer, scheduler, num_epochs)<br>
36                     # backward + optimize only if in training phase<br>
37                     if phase == ‘train’:<br>
—&gt; 38                         loss.backward()<br>
39                         optimizer.step()<br>
40</p>
<p>~\anaconda3\Lib\site-packages\torch\tensor.py in backward(self, gradient, retain_graph, create_graph)<br>
183                 products. Defaults to <code>False</code>.<br>
184         “”"<br>
–&gt; 185         torch.autograd.backward(self, gradient, retain_graph, create_graph)<br>
186<br>
187     def register_hook(self, hook):</p>
<p>~\anaconda3\Lib\site-packages\torch\autograd_<em>init</em>_.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables)<br>
125     Variable._execution_engine.run_backward(<br>
126         tensors, grad_tensors, retain_graph, create_graph,<br>
–&gt; 127         allow_unreachable=True)  # allow_unreachable flag<br>
128<br>
129</p>
<p>RuntimeError: CUDA error: CUBLAS_STATUS_ALLOC_FAILED when calling <code>cublasCreate(handle)</code><br>
Exception raised from createCublasHandle at …\aten\src\ATen\cuda\CublasHandlePool.cpp:8 (most recent call first):<br>
00007FFB63A875A200007FFB63A87540 c10.dll!c10::Error::Error [ @ ]<br>
00007FFAFF31AEA800007FFAFF319E70 torch_cuda.dll!at::cuda::getCurrentCUDASparseHandle [ @ ]<br>
00007FFAFF31A7D800007FFAFF319E70 torch_cuda.dll!at::cuda::getCurrentCUDASparseHandle [ @ ]<br>
00007FFAFF31B66700007FFAFF31B1A0 torch_cuda.dll!at::cuda::getCurrentCUDABlasHandle [ @ ]<br>
00007FFAFF31B24700007FFAFF31B1A0 torch_cuda.dll!at::cuda::getCurrentCUDABlasHandle [ @ ]<br>
00007FFAFF31320700007FFAFF3124B0 torch_cuda.dll!at::native::sparse_mask_cuda [ @ ]<br>
00007FFAFE81CA9700007FFAFE81B990 torch_cuda.dll!at::native::lerp_cuda_tensor_out [ @ ]<br>
00007FFAFE81E4D200007FFAFE81DF60 torch_cuda.dll!at::native::addmm_out_cuda [ @ ]<br>
00007FFAFE81F64300007FFAFE81F560 torch_cuda.dll!at::native::mm_cuda [ @ ]<br>
00007FFAFF381B0F00007FFAFF31E0A0 torch_cuda.dll!at::native::set_storage_cuda_ [ @ ]<br>
00007FFAFF371B2200007FFAFF31E0A0 torch_cuda.dll!at::native::set_storage_cuda_ [ @ ]<br>
00007FFB2969D94900007FFB29698FA0 torch_cpu.dll!at::bucketize_out [ @ ]<br>
00007FFB296D057700007FFB296D0520 torch_cpu.dll!at::mm [ @ ]<br>
00007FFB2AA2EC7900007FFB2A93E010 torch_cpu.dll!torch::autograd::GraphRoot::apply [ @ ]<br>
00007FFB291E715700007FFB291E6290 torch_cpu.dll!at::indexing::TensorIndex::boolean [ @ ]<br>
00007FFB2969D94900007FFB29698FA0 torch_cpu.dll!at::bucketize_out [ @ ]<br>
00007FFB297B210700007FFB297B20B0 torch_cpu.dll!at::Tensor::mm [ @ ]<br>
00007FFB2A8CB71100007FFB2A8CA760 torch_cpu.dll!torch::autograd::profiler::Event::kind [ @ ]<br>
00007FFB2A8816D800007FFB2A881580 torch_cpu.dll!torch::autograd::generated::AddmmBackward::apply [ @ ]<br>
00007FFB2A877E9100007FFB2A877B50 torch_cpu.dll!torch::autograd::Node::operator() [ @ ]<br>
00007FFB2ADDF9BA00007FFB2ADDF300 torch_cpu.dll!torch::autograd::Engine::add_thread_pool_task [ @ ]<br>
00007FFB2ADE03AD00007FFB2ADDFFD0 torch_cpu.dll!torch::autograd::Engine::evaluate_function [ @ ]<br>
00007FFB2ADE4FE200007FFB2ADE4CA0 torch_cpu.dll!torch::autograd::Engine::thread_main [ @ ]<br>
00007FFB2ADE4C4100007FFB2ADE4BC0 torch_cpu.dll!torch::autograd::Engine::thread_init [ @ ]<br>
00007FFB4A0E0A7700007FFB4A0BA150 torch_python.dll!THPShortStorage_New [ @ ]<br>
00007FFB2ADDBF1400007FFB2ADDB780 torch_cpu.dll!torch::autograd::Engine::get_base_engine [ @ ]<br>
00007FFB9E6B0E8200007FFB9E6B0D40 ucrtbase.dll!beginthreadex [ @ ]<br>
00007FFBA0967BD400007FFBA0967BC0 KERNEL32.DLL!BaseThreadInitThunk [ @ ]<br>
00007FFBA15CCE5100007FFBA15CCE30 ntdll.dll!RtlUserThreadStart [ @ ]</p>

username: ptrblck
cooked: <p>Could you rerun the code via:</p>
<pre><code class="lang-python">CUDA_LAUNCH_BLOCKING=1 python script.py args
</code></pre>
<p>and post the stack trace here?</p>
<p>Also, check the memory usage on your device, as this error message is sometimes raised instead of a proper message pointing towards an out of memory issue.</p>

username: Jim_Xu
cooked: <p>hello,<br>
Thanks for your reply!<br>
Could you show me an classcifacation tast example which use “torch.utils.data.random_split”, I think the problem is not caused by GPU memory, because when I manually splt the dataset to training and testing dataset, the code works, but I really can access the dataset split by “torch.utils.data.random_split” now, it just show the error when the code start training</p>

username: ptrblck
cooked: <p>Here is a simple example how to use <code>random_split</code> and how to access attributes of the underlying dataset:</p>
<pre><code class="lang-python">dataset = datasets.MNIST(
    root=PATH, download=False,
    transform=transforms.ToTensor())

train_dataset, val_dataset = torch.utils.data.random_split(
    dataset, [len(dataset) - 10000, 10000])

train_loader = DataLoader(train_dataset)
val_loader = DataLoader(val_dataset)

# access underlying target directly in `datasets.MNIST`
print(dataset.targets)
# access it via .dataset in the Subset (created by random_split)
print(train_dataset.dataset.targets)
# access it via .dataset.dataset in the DataLoader
print(train_loader.dataset.dataset.targets)
</code></pre>

username: Jim_Xu
cooked: <p>Oh, you’re so kind !! I will make a try</p>