Creating Image Labeling Pipeline, Labels are not Matching Up to Correct Image, instead they load randomly

username: Jennifer_Davis
cooked: <p>Hi there,</p>
<p>I am working on an image labeling pipeline in pytorch.  I have it mostly working except in the <strong>getitem</strong> function.  The function is supposed to load my images, and captions to match each image.  Instead it loads images and captions randomly.  The captions come from a dataframe / csv file.  In the file there is an image_id and its matching caption.  To match the caption one has to match the image id to the file name image_id.  To load all the images one has to iterate through folders (as shown in the code below).  Every time an image is opened it should have a matching caption.</p>
<p>I have tried iterating through the csv at the same time as the image files, but that doesn’t work.  I know this is probably a simple answer, and would appreciate the input.</p>
<pre><code class="lang-auto">### to read all images in all directories, DEBUG CODE

class CaptioningData(Dataset):
    def __init__(self, root, df, vocab):
        self.df = df.reset_index(drop=True)
        self.root = root
        self.vocab = vocab
        self.transform = transforms.Compose([ 
            transforms.Resize(224),
            transforms.RandomCrop(224),
            transforms.RandomHorizontalFlip(), 
            transforms.ToTensor(), 
            transforms.Normalize((0.485, 0.456, 0.406), 
                                 (0.229, 0.224, 0.225))]
        )
    def __getitem__(self, index):
        """Returns one data pair (image and caption)."""
        #row = self.df.iloc[index].squeeze()
        #id = row.image_id
        #image_path = f'{self.root}/{id}.png' # original code, for debug
        for file in files:
            row = self.df.iloc[index].squeeze()
            id = row.image_id
            #print(os.path.join(root, file))
            image = os.path.join(root, file)
            image = Image.open(image).convert('RGB')    
            caption = row.InChI # here need caption to match image_id
            tokens = str(caption).lower().split()
            target = []
            target.append(vocab.stoi['&lt;start&gt;'])
            target.extend([vocab.stoi[token] for token in tokens])
            target.append(vocab.stoi['&lt;end&gt;'])
            target = torch.Tensor(target).long()
        return image, target, caption
    
    #debug
    def choose(self):
        return self[np.random.randint(len(self))]
    
    #code
    #def choose(self):
    #    return self[(len(self))]
    
    def __len__(self):
        return len(self.df)
    def collate_fn(self, data):
        data.sort(key=lambda x: len(x[1]), reverse=True)
        images, targets, captions = zip(*data)
        images = torch.stack([self.transform(image) for image in images], 0)
        lengths = [len(tar) for tar in targets]
        _targets = torch.zeros(len(captions), max(lengths)).long()
        for i, tar in enumerate(targets):
            end = lengths[i]
            _targets[i, :end] = tar[:end] 
        return images.to(device), _targets.to(device), torch.tensor(lengths).long().to(device)
</code></pre>

username: eqy
cooked: <p>I’m not sure how the matching is done here. It looks like with the current setup the target will be what is specified by index but image will simply be the last image assigned (which in this case is the last filename in files). I think there needs to be some correspondence used between index and the file. If you know the filename from the target <code>image_id</code>, can you just use that to open the correct image rather than iterating through all of them? It might be easier to understand if we see an example of the filename <code>file</code> as well as all of the data contained in a single <code>row</code> of the dataframe.</p>
<p>Finally, I don’t even see where <code>files</code> is defined; is it a global variable somewhere? I would guess that unless it is being mutated somewhere the exact same image would be returned every time with the current code.</p>

username: Jennifer_Davis
cooked: <p>Hi Eqy,</p>
<p>Thanks for the response.  All the files are in subdirectories like this:<br>
root/first/second/third/image.png<br>
So I wrote the loop:</p>
<aside class="quote no-group" data-username="Jennifer_Davis" data-post="1" data-topic="125067">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jennifer_davis/48/38619_2.png" class="avatar"> Jennifer_Davis:</div>
<blockquote>
<pre><code class="lang-auto">for file in files:
            row = self.df.iloc[index].squeeze()
            id = row.image_id
            #print(os.path.join(root, file))
            image = os.path.join(root, file)
            image = Image.open(image).convert('RGB')    
            caption = row.InChI # here need caption to match image_id
            tokens = str(caption).lower().split()
            target = []
            target.append(vocab.stoi['&lt;start&gt;'])
            target.extend([vocab.stoi[token] for token in tokens])
            target.append(vocab.stoi['&lt;end&gt;'])
            target = torch.Tensor(target).long()
</code></pre>
</blockquote>
</aside>
<p>Which loops through all directories and opens the images.  It was also supposed to assign a caption to each based on the dataframe’s ‘image_id’.   The problem is exactly as you described.  There’s an index for each image, but the loop just opens the images randomly until it loops through all files.  It doesn’t match on the index as I would like.  The index has the appropriate caption.</p>
<p>Hope this makes sense.</p>

username: Jennifer_Davis
cooked: <aside class="quote no-group" data-username="Jennifer_Davis" data-post="1" data-topic="125067">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jennifer_davis/48/38619_2.png" class="avatar"> Jennifer_Davis:</div>
<blockquote>
<p><code>image_path = f'{self.root}/{id}.png'</code></p>
</blockquote>
</aside>
<p>The above line works fine if there are no subdirectories, but of course it doesn’t loop through all the directories.</p>

username: eqy
cooked: <p>OK, in that case it might be more efficient if you only loop through all the images once and build a mapping from id to the precise filepath for each id, then just look this up when you want to do <code>__getitem__</code>.</p>
<p>For example, you can add something like</p>
<pre><code class="lang-auto">self.id_to_path = dict()
for dirpath, _, filenames in os.walk(root):
    for filename in filenames:
        id, ext = os.path.basename(filename).splitext()
        if ext == '.png':
            # TODO: might want to convert to integer depending on id type in dataframe
            self.id_to_path[id] = os.path.join(dirpath, filename)
</code></pre>
<p>Then in your <code>__getitem__</code> just lookup the proper file:</p>
<pre><code class="lang-auto">    def __getitem__(self, index):
        """Returns one data pair (image and caption)."""
        row = self.df.iloc[index].squeeze()
        id = row.image_id
        image = self.id_to_path[id]
        image = Image.open(image).convert('RGB')    
        caption = row.InChI # here need caption to match image_id
        tokens = str(caption).lower().split()
        target = []
        target.append(vocab.stoi['&lt;start&gt;'])
        target.extend([vocab.stoi[token] for token in tokens])
        target.append(vocab.stoi['&lt;end&gt;'])
        target = torch.Tensor(target).long()
        return image, target, caption
</code></pre>

username: Jennifer_Davis
cooked: <p>Thank you so much!  I will try this right away.</p>

username: Jennifer_Davis
cooked: <p>It works perfectly.  I made one small change:</p>
<pre><code class="lang-auto">class CaptioningData(Dataset):
    def __init__(self, root, df, vocab):
        self.df = df.reset_index(drop=True)
        self.root = root
        self.vocab = vocab
        self.id_to_path = dict()
        for dirpath, _, filenames in os.walk(self.root):
            for filename in filenames:
                id, ext = os.path.splitext(filename)
                if ext == '.png':
                    self.id_to_path[id] = os.path.join(dirpath, filename)
        self.transform = transforms.Compose([ 
            transforms.Resize(224),
            transforms.RandomCrop(224),
            transforms.RandomHorizontalFlip(), 
            transforms.ToTensor(), 
            transforms.Normalize((0.485, 0.456, 0.406), 
                                 (0.229, 0.224, 0.225))]
        )
</code></pre>
<p>Thank you again!</p>

username: eqy
cooked: <p>Nice, good change with removing basename.</p>