Multi-label data loading bottleneck Pytorch

username: serdarrader
cooked: <p>I am trying to write custom dataset and dataloader for pascal-voc-2007. It is a multi-label classification problem. There is csv file to hold the name of the images and their corresponding labels. I want to resize images and create one-hot encoded vectors for labels. Also I don’t want to load images to RAM, I want to load them when they appear in mini-batch.</p>
<p>So far I did all of them but with gtx 1070 (I dont’t think it is relevant when loading data), and i5 8600k, Samsung M.2 solid state, just loading 1 epoch of  data(without feeding to the network, I just iterate over dataloader and look) took 1 min 7s.</p>
<p>Is it normal or can be considered as slow? Or is there some optimization problem in my code?</p>
<p>My code:</p>
<pre><code class="lang-python">d = {}
for e in df["labels"]:
        l = e.split(" ")
        for k in l:
            if k not in d:
                d[k] = len(d)


def create_code(df, idx):
    l = [0] * len(d)
    
    labels = df.iloc[idx]["labels"].split(" ")
    labels_code = []
    
    for e in labels:
        l[d[e]] = 1
    
    return torch.tensor(l, dtype = torch.long).view(1,-1)


def transform_image(pth):
    img = Image.open(pth)
    return (ToTensor()(img.resize((256,256), resample = PIL.Image.ANTIALIAS))).unsqueeze(0).type(torch.float64)


class Dataset():
    def __init__(self, df, path): 
        self.df = df
        self.path = path
        
    def __len__(self): 
        return len(self.df)
    def __getitem__(self, idxs):
        if isinstance(idxs, int):
            imgs = transform_image(self.path/self.df.iloc[idxs]["fname"])
            labels = create_code(self.df, idxs)
        else:
            sub_df = self.df.iloc[idxs]
            imgs = transform_image(self.path/sub_df.iloc[0]["fname"])
            labels = create_code(sub_df, 0)
            for i in range(1, len(idxs)):
                e = sub_df.iloc[i]["fname"]
                imgs = torch.cat((imgs,transform_image(self.path/e)))
                labels = torch.cat((labels, create_code(sub_df, i)))
            
        return imgs, labels

            
class DataLoader():
    def __init__(self, ds, bs): 
        self.ds, self.bs = ds, bs
    def __iter__(self):
        n = len(self.ds)
        l = torch.randperm(n)

        
        for i in range(0, n, self.bs): 
            idxs_l = l[i:i+self.bs]
            yield self.ds[idxs_l]

</code></pre>
<p>Then:</p>
<pre><code class="lang-python">train_ds = Dataset(df_train, path/"train")
train_dl = DataLoader(train_ds, 64)

%%time
for e in train_dl:
    pass

</code></pre>
<p>The last part took 1 min 7s</p>

username: serdarrader
cooked: <p>I don’t know the details but after doing profiling I found that torch.cat is taking too many times. So I updated my code like this:</p>
<pre><code class="lang-auto">class Dataset():
    def __init__(self, df, path): 
        self.df = df
        self.path = path
        
    def __len__(self): 
        return len(self.df)
    def __getitem__(self, idxs):
        if isinstance(idxs, int):
            imgs = transform_image(self.path/self.df.iloc[idxs]["fname"])
            labels = create_code(self.df, idxs)
        else:
            sub_df = self.df.iloc[idxs]
            imgs = []
            labels = []
            for i in range(len(idxs)):
                e = sub_df.iloc[i]["fname"]
                imgs.append(transform_image(self.path/e))
                labels.append(create_code(sub_df, i))
            
        return torch.stack(imgs), torch.stack(labels)
            
</code></pre>
<p>Instead of torc.cat, I am appending the list and then transforming the list and stacking elements, now iteration over 1 epoch takes 11.2 s</p>

username: PistonY
cooked: <p><code>Dataset</code> should only take one sample and <code>Dataloader</code> would stack them to one batch automatically.<br>
I’m not sure you take the entire VOC dataset in <code>Dataset</code>, if so it’s the wrong way.<br>
I think you can find <code>VOCDataset</code> in <code>torchvision</code> which should be a good example.</p>

username: serdarrader
cooked: <p>Thank you for your reply. In dataframe there is file_name column and labels column. I am storing that dataframe in Dataset, and when Dataloader wants random subset from it (index of dataframe’s rows in this case), Dataset just reads file names from dataframe and just loads those images</p>

username: PistonY
cooked: <p>I’m sorry for misunderstanding.<br>
If you just want to know which part cost the most time you could use <code>time</code> module to calculate them.</p>
<pre><code class="lang-auto">import time

start = time.time()
...(some cost time op)
print("This op cost {}".format(time.time()-start))
</code></pre>

username: serdarrader
cooked: <p>I run %prun and torch.cat is taking too much time, I changed it and it works fine now. But I am trying to make for loop parallel so that it can be faster. Thank you very much for your answers <img src="https://discuss.pytorch.org/images/emoji/apple/+1.png?v=9" title=":+1:" class="emoji" alt=":+1:"></p>