How to read dataset from .tar files?

chinnu: <p>how to read dataset from .tar files?</p>

Nikronic: <p>Hi,<br>
This works if you have image dataset in <code>.tar</code> file.<br>
You have to have a <code>.csv</code> or <code>.txt</code> file, including a name per line of your dataset.</p>
<pre><code class="lang-python">from PIL import Image
from torchvision.transforms import ToTensor, ToPILImage
import numpy as np
import random

import tarfile
import io
import os
import pandas as pd

from torch.utils.data import Dataset
import torch

class YourDataset(Dataset):
    def __init__(self, txt_path='filelist.txt', img_dir='data.tar', transform=None):
        """
        Initialize data set as a list of IDs corresponding to each item of data set

        :param img_dir: path to image files as a uncompressed tar archive
        :param txt_path: a text file containing names of all of images line by line
        :param transform: apply some transforms like cropping, rotating, etc on input image
        """

        df = pd.read_csv(txt_path, sep=' ', index_col=0)
        self.img_names = df.index.values
        self.txt_path = txt_path
        self.img_dir = img_dir
        self.transform = transform
        self.to_tensor = ToTensor()
        self.to_pil = ToPILImage()
        self.tf = tarfile.open(self.img_dir)

    def get_image_from_tar(self, name):
        """
        Gets a image by a name gathered from file list csv file

        :param name: name of targeted image
        :return: a PIL image
        """
        image = self.tf.extractfile(name)
        image = image.read()
        image = Image.open(io.BytesIO(image))
        return image

    def __len__(self):
        """
        Return the length of data set using list of IDs

        :return: number of samples in data set
        """
        return len(self.img_names)

    def __getitem__(self, index):
        """
        Generate one item of data set.

        :param index: index of item in IDs list

        :return: a sample of data as a dict
        """

        if index == (self.__len__() - 1) :  # close tarfile opened in __init__
            self.tf.close()

        
        image = self.get_image_from_tar(self.img_names[index])
       
        if self.transform is not None:
            image = self.transform(image)

        sample = {'X': image}

        return sample
</code></pre>

Kupchanski: <p>Hello, Thank you for your solution. I have an issue when create dataloader with dataset from tar archive.<br>
It gives me this error, if I run dataloader more than one time. Can you help me with that problem?</p>
<pre><code class="lang-auto">OSError: TarFile is closed
</code></pre>

Sandhya_Giri: <p>When i tried this method of extracting particular images inside <strong>getItem</strong> method for TinyImageNet dataset, I faced issues with zlib during data_loader collation. The same error disappeared once I extracted images in <strong>init</strong> and stored them in a dict, and then in <strong>getItem</strong> only did a transformation and returned the image and target.</p>

Nikronic: <p>That is possible, I just used this implementation for a large tar file on a single machine. It may not work in few different situations.<br>
But still the best case even for large datasets is to extract entire dataset and loop over it. Here is the original post that lead to this code:</p><aside class="quote quote-modified" data-post="1" data-topic="39598">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nikronic/48/71873_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/fastest-way-to-read-images-from-uncompressed-tar-file-in-getitem-method-of-custom-dataset/39598">Fastest way to read images from uncompressed TAR file in __getitem__ method of Custom Dataset</a> <a class="badge-category__wrapper " href="/c/vision/5"><span data-category-id="5" style="--category-badge-color: #AB9364; --category-badge-text-color: #FFFFFF;" data-drop-close="true" class="badge-category " title="Topics related to either pytorch/vision or vision research related topics"><span class="badge-category__name">vision</span></span></a>
  </div>
  <blockquote>
    Hello everyone 
I have a huge dataset (2 million) of jpg images in one uncompressed TAR file. I also have a txt file each line is the name of the image in TAR file in order. 
img_0000001.jpg
img_0000002.jpg
img_0000003.jpg
...

and images in tar file are exactly the same. 
I searched alot and find out tarfile module is the best one, but when I tried to read images from tar file using name, it takes too long. And the reason is, everytime I call getmemeber(name) method, it calls getmembers() metho…
  </blockquote>
</aside>


Jotaf: <p>Inspired by Nikronic’s solution, I made a similar class that reads from Tar archives.</p>
<p>I noticed that it didn’t work out-of-the-box with DataLoaders with multiple workers (depends on the OS/multiprocessing method), so this new class essentially opens a different TarFile (with unique file handles) per worker process.</p>
<p>I also added some methods to access files, including text and images, to allow easier subclassing; and the Tar archive’s content is cached on first read (takes under a minute for a 140GB Tar file with &gt;1M files).</p>
<p>In case you want to check it out: <a href="https://github.com/jotaf98/simple-tar-dataset" rel="noopener nofollow ugc">Simple Tar Dataset on GitHub</a>.</p>

Soerendip: <p>I am getting</p>
<pre><code class="lang-auto">Error loading image images/image_04781.jpg: cannot identify image file &lt;_io.BytesIO object at 0x722e057cd210&gt;
Error loading image images/image_04729.jpg: broken data stream when reading image file
</code></pre>
<p>when reading images from a tarball. For some images. When extracted, I have no issues.</p>