Get file names and file path using PyTorch dataloader

grid_world: <p>I am using PyTorch 1.8 and Python 3.8 to read images from a folder using the following code:</p>
<pre><code>print(f"PyTorch version: {torch.__version__}")
# PyTorch version: 1.8.1

# Device configuration-
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
print(f"currently available device: {device}")
# currently available device: cpu


# Define transformations for training and test sets-
transform_train = transforms.Compose(
    [
      # transforms.RandomCrop(32, padding = 4),
      # transforms.RandomHorizontalFlip(),
      transforms.ToTensor(),
      # transforms.Normalize((0.4914, 0.4822, 0.4465), (0.2023, 0.1994, 0.2010)),
     ]
     )

transform_test = transforms.Compose(
    [
      transforms.ToTensor(),
      # transforms.Normalize((0.4914, 0.4822, 0.4465), (0.2023, 0.1994, 0.2010)),
     ]
     )

# Define directory containing images-
data_dir = 'My_Datasets/Cat_Dog_data/'

# Define datasets-
train_data = datasets.ImageFolder(data_dir + '/train', 
                                  transform = train_transforms)
test_data = datasets.ImageFolder(data_dir + '/test', 
                                 transform = test_transforms)

print(f"number of train images = {len(train_data)} &amp; number of validation images = {len(test_data)}")
# number of train images = 22500 &amp; number of validation images = 2500

print(f"number of training classes = {len(train_data.classes)} &amp; number of validation classes = {len(test_data.classes)}")
# number of training classes = 2 &amp; number of validation classes = 2

# Define data loaders-
trainloader = torch.utils.data.DataLoader(train_data, batch_size = 32)
testloader = torch.utils.data.DataLoader(test_data, batch_size = 32)

len(trainloader), len(testloader)
# (704, 79)

# Sanity check-
len(train_data) / 32, len(test_data) / 32
</code></pre>
<p>You can iterate through the train data using ‘train_loader’ as follows:</p>
<pre><code>for img, lab in train_loader:
   print(img.shape, lab.shape)
   pass
</code></pre>
<p>However, I am interested in getting the file name along with the file path from which the file was read. How can I achieve this?</p>
<p>Thanks!</p>
<p>Guys, I am sorry for using CIFAR-10 dataset when in fact I want to read images from local system folders. The modified code is above.</p>

ptrblck: <p>The <code>CIFAR10</code> dataset doesn’t download all images separately, but the binary data as seen <a href="https://github.com/pytorch/vision/blob/d1ab583d0d2df73208e2fc9c4d3a84e969c69b70/torchvision/datasets/cifar.py#L73-L76">here</a>, so you won’t be able to return paths to each image.<br>
However, in other datasets, which lazily load each image file, you can just return the path with the data and target tensors.</p>

rahulvigneswaran: <p>I think we can achieve it by doing the following, <a class="mention" href="/u/grid_world">@grid_world</a> <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<pre><code class="lang-auto">class your_new_custom_dataset_class(torchvision.datasets.CIFAR10):  #&lt;----Important
    def __init__(self):
        super(your_new_custom_dataset_class, self).__init__()
        self.get_filenames()    #&lt;----Important
    
    def __getitem__(self, index):
        img, label = self.data[index], self.labels[index]
        filename = self.filenames[index]       #&lt;----Important

        img = Image.fromarray(img)

        if self.transform is not None:
            img = self.transform(img)

        if self.target_transform is not None:
            label = self.target_transform(label)

        return img, label, filename      #&lt;----Important
    
    def get_filenames(self):    #&lt;----Important
        self.filenames = []
        if self.train:
            downloaded_list = self.train_list
        else:
            downloaded_list = self.test_list
            
        for file_name, checksum in downloaded_list:
            file_path = os.path.join(self.root, self.base_folder, file_name)
            with open(file_path, 'rb') as f:
                self.entry = pickle.load(f, encoding='latin1')
            self.filenames.extend(self.entry["filenames"]) 
</code></pre>
<p>PS. The implementation is a bit sloppy, just make sure to note all the lines with the <code>#&lt;----Important</code> comment.</p>

grid_world: <p>Apologies for using CIFAR10 dataset. I am interested in getting image file names and path for reading images from local system folder. The modified code is above.</p>

grid_world: <p>Rather than using CIFAR10 dataset, apologies for that, I am interested in getting image file names and path for reading images from local system folder. The modified code is above.</p>

Ruy_Diaz: <p>In a similar case I found this useful:</p>
<p>dataloader_.sampler.data_source.dataset.imgs</p>
<p>It gave me a list with tuples: path_and_filename, class</p>

venki_lfc: <p>Exactly what I needed. Thank you <a class="mention" href="/u/ruy_diaz">@Ruy_Diaz</a> <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

DJF: <p>Just be careful anyone using <code>dataloader_.sampler.data_source.dataset.imgs</code> on a <em>subsetted</em> dataloader. The data source is unaffected, so this gives the full set.</p>