Print an image file name

alameer: <p>I would like to be able to print the file name of the image that has been through the training model. I have tried the solution provided in here:</p>
<aside class="quote" data-post="4" data-topic="4212">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ajong/48/7416_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/dataloader-filenames-in-each-batch/4212/4">DataLoader Filenames in each Batch</a> <a class="badge-category__wrapper " href="/c/vision/5"><span data-category-id="5" data-drop-close="true" class="badge-category " title="Topics related to either pytorch/vision or vision research related topics"><span class="badge-category__name">vision</span></span></a>
  </div>
  <blockquote>
    For anyone who stumbles upon this later, I made a convenient little gist for this:
  </blockquote>
</aside>

<p>but I am getting an error message:</p>
<p>RuntimeError: Found 0 files in subfolders of: /Users/Documents/images</p>
<p>Supported extensions are: .jpg,.jpeg,.png,.ppm,.bmp,.pgm,.tif,.tiff,.webp</p>
<p>There are definitely images in this directory</p>
<pre><code class="lang-auto">import torch
from torchvision import datasets

class ImageFolderWithPaths(datasets.ImageFolder):
    """Custom dataset that includes image file paths. Extends
    torchvision.datasets.ImageFolder
    """

    # override the __getitem__ method. this is the method that dataloader calls
    def __getitem__(self, index):
        # this is what ImageFolder normally returns 
        original_tuple = super(ImageFolderWithPaths, self).__getitem__(index)
        # the image file path
        path = self.imgs[index][0]
        # make a new tuple that includes original and the path
        tuple_with_path = (original_tuple + (path,))
        return tuple_with_path

# EXAMPLE USAGE:
# instantiate the dataset and dataloader
data_dir = '/Users/Documents/images'
dataset = ImageFolderWithPaths(data_dir) # our custom dataset
dataloader = torch.utils.DataLoader(dataset)

# iterate over data
for inputs, labels, paths in dataloader:
    # use the above variables freely
    print(inputs, labels, paths)
</code></pre>

ptrblck: <p>Could you check the root path again? <code>/Users/Documents/Images</code> might miss the actual root folder or are you sure your system contains a <code>/Users</code> folder in the system root folder?</p>

alameer: <p>Sorry, what do you mean by the root here. I am using a mac and i have done pwd in the folder where the images are saved. Do i need to have the script in the same place as the images. The script is in the folder above?</p>

ptrblck: <p>No, the script doesn’t need to be in the same folder.<br>
Note that the passed folder should contain subfolders with the images. The images should not be in the passed root folder. Did you make sure that’s the case?</p>

alameer: <p>Thanks! I have changed it to reflect this now. But, got a different error message which says " dataloader = torch.utils.DataLoader(dataset)</p>
<p>AttributeError: module ‘torch.utils’ has no attribute ‘DataLoader’</p>
<p>I have included those lines of code, but it didn’t help:</p>
<pre><code class="lang-auto">import torch
from torchvision import datasets
import torch.utils.data</code></pre>

aaqqxx: <p><code>from torch.utils.data import DataLoader</code></p>

alameer: <p>Sorry, that didn’t help!</p>

aaqqxx: <p>you use the import statement, and then dataloader = DataLoader(…)</p>

alameer: <p>Sorry! Not clear what you mean? can you explain more please?</p>

alameer: <p>Like this</p>
<pre><code class="lang-auto">import torch
from torchvision import datasets
import torch.utils.data
from torch.utils.data import DataLoader</code></pre>

aaqqxx: <p>Yes</p>
<pre><code class="lang-auto">from torchvision import datasets
import torch.utils.data
from torch.utils.data import DataLoader

dataloader = DataLoader(dataset)
</code></pre>

alameer: <p>I am getting a traceback issue:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "file_location2.py", line 30, in &lt;module&gt;
    for inputs, labels, paths in dataloader:
  File "/Users/nubstech/opt/anaconda3/envs/Cells_Counting/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
    data = self._next_data()
  File "/Users/nubstech/opt/anaconda3/envs/Cells_Counting/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 385, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/Users/nubstech/opt/anaconda3/envs/Cells_Counting/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 47, in fetch
    return self.collate_fn(data)
  File "/Users/nubstech/opt/anaconda3/envs/Cells_Counting/lib/python3.7/site-packages/torch/utils/data/_utils/collate.py", line 79, in default_collate
    return [default_collate(samples) for samples in transposed]
  File "/Users/nubstech/opt/anaconda3/envs/Cells_Counting/lib/python3.7/site-packages/torch/utils/data/_utils/collate.py", line 79, in &lt;listcomp&gt;
    return [default_collate(samples) for samples in transposed]
  File "/Users/nubstech/opt/anaconda3/envs/Cells_Counting/lib/python3.7/site-packages/torch/utils/data/_utils/collate.py", line 81, in default_collate
    raise TypeError(default_collate_err_msg_format.format(elem_type))
TypeError: default_collate: batch must contain tensors, numpy arrays, numbers, dicts or lists; found &lt;class 'PIL.Image.Image'&gt;
</code></pre>

a_d: <p>In your transform did you add a transform.toTensor() at the end of the composition?<br>
Essentially you must have forgotten to transform from PIL image to Tensor</p>

alameer: <p>I didn’t add transform.toTensor(), where does this need to go in my code please?</p>

a_d: <p>In the traditional ImageFolder class you would do something like -</p>
<pre><code class="lang-auto">data = dataloader(
        Imagefolder( path_To_images, transforms.Compose(['YOUR TRANSFORMATIONS'])))
</code></pre>
<p>Essentially it loads the data using the PIL library performs some processing on the loaded image using the PIL library which you specify and then you add a ToTensor() in the compose where it takes the processed image makes it ready to be taken as input for your network, returns the tensor and the class… Dataloader prepares the batches.</p>
<p>So in your pipeline wherever you are loading your images using PIL, just convert them to tensors(or any other data type mentioned in the error) and it should work.</p>

alameer: <p>Many Thanks for your help. I have integrated the code into my training code now. However, when i try to save the image path into the CSV file I get an error message</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "train_filename.py", line 130, in &lt;module&gt;
    fd.write( ','.join(map(str, path.detach().tolist())) + '\n')
NameError: name 'path' is not defined
</code></pre>
<p>Part code for train_filename.py</p>
<pre><code class="lang-auto">class ImageFolderWithPaths(datasets.ImageFolder):
    """Custom dataset that includes image file paths. Extends
    torchvision.datasets.ImageFolder
    """

# override the __getitem__ method. this is the method that dataloader calls
def __getitem__(self, index):
    # this is what ImageFolder normally returns 
    original_tuple = super(ImageFolderWithPaths, self).__getitem__(index)
    # the image file path
    path = self.imgs[index][0]
    # make a new tuple that includes original and the path
    tuple_with_path = (original_tuple + (path,))
    return tuple_with_path

# main training loop
global_step = 0
best_test_error = 10000
for epoch in range(50):
    print("Epoch %d" % epoch)
    model.train()
    for images, paths in tqdm(loader_train):
        images = images.to(device)
        targets = torch.tensor([metadata['count'][os.path.split(path)[-1]] for path in paths]) # B
        targets = targets.float().to(device)

        output = model(images) # B x 1 x 9 x 9 (analogous to a heatmap)
        preds = output.sum(dim=[1,2,3]) # predicted cell counts (vector of length B)
        print(preds)
        with open('preds_base_model.csv','a') as fd:
            fd.write( ','.join(map(str, preds.detach().tolist())) + '\n')
            fd.write( ','.join(map(str, targets.detach().tolist())) + '\n')
            fd.write( ','.join(map(str, path.detach().tolist())) + '\n')</code></pre>

a_d: <p>path over here is supposed be a string right? Also you must have defined path somewhere in your code or passed to your training function. Not declaring path seems to be the problem here</p>

alameer: <p>Yes. It’s a string. During the training the model go through csv file to randomly choose image to train.</p>

a_d: <p>If you want to randomly chose an image to train, you do not need to store the path in a csv file - you could do the following -</p>
<pre><code class="lang-auto">os.chdir('absolute path to images')
images = os.listdir()
image = random.choice(images) #return image name. Now load this image
</code></pre>
<p>Coming to your code, the variable path Is not initialized. Could you share the code?</p>

alameer: <p>Thanks! Please see code below</p>
<pre><code class="lang-auto">class ImageFolderWithPaths(datasets.ImageFolder):
    """Custom dataset that includes image file paths. Extends
    torchvision.datasets.ImageFolder
    """

# override the __getitem__ method. this is the method that dataloader calls
def __getitem__(self, index):
    # this is what ImageFolder normally returns 
    original_tuple = super(ImageFolderWithPaths, self).__getitem__(index)
    # the image file path
    path = self.imgs[index][0]
    # make a new tuple that includes original and the path
    tuple_with_path = (original_tuple + (path,))
    return tuple_with_path

# main training loop
global_step = 0
best_test_error = 10000
for epoch in range(50):
    print("Epoch %d" % epoch)
    model.train()
    for images, paths in tqdm(loader_train):
        images = images.to(device)
        targets = torch.tensor([metadata['count'][os.path.split(path)[-1]] for path in paths]) # B
        targets = targets.float().to(device)

        output = model(images) # B x 1 x 9 x 9 (analogous to a heatmap)
        preds = output.sum(dim=[1,2,3]) # predicted cell counts (vector of length B)
        print(preds)
        with open('preds_base_model.csv','a') as fd:
            fd.write( ','.join(map(str, preds.detach().tolist())) + '\n')
            fd.write( ','.join(map(str, targets.detach().tolist())) + '\n')
            fd.write( ','.join(map(str, path.detach().tolist())) + '\n')</code></pre>