DataLoaders - Multiple files, and multiple rows per column with lazy evaluation

mxlei01: <p>Hi All,</p>
<p>I’m trying to create a DataSet class that can load many large file, and each file have rows of data a model would need to train.</p>
<p>I’ve read: <a href="https://discuss.pytorch.org/t/loading-huge-data-functionality/346/3">Loading huge data functionality</a></p>
<pre><code>class MyDataset(torch.utils.Dataset):
    def __init__(self):
        self.data_files = os.listdir('data_dir')
        sort(self.data_files)

    def __getindex__(self, idx):
        return load_file(self.data_files[idx])

    def __len__(self):
        return len(self.data_files)


dset = MyDataset()
loader = torch.utils.DataLoader(dset, num_workers=8)
</code></pre>
<p>Essentially if each file is an image, then you can use this functionality to only load the required images in memory.</p>
<p>So what about a file with multiple lines? Then I came across this github thread: <a href="https://github.com/pytorch/text/issues/130" rel="nofollow noopener">https://github.com/pytorch/text/issues/130</a></p>
<pre><code>class LazyTextDataset(Dataset):
    def __init__(self, filename):
        self._filename = filename
        self._total_data = 0
        with open(filename, "r") as f:
            self._total_data = len(f.readlines()) - 1

    def __getitem__(self, idx):
        line = linecache.getline(self._filename, idx + 1)
        csv_line = csv.reader([line])
        .....

    def __len__(self):
        return self._total_data
</code></pre>
<p>Now the problem I have is how do I combine these two so that I can load multiple files that are all large?</p>
<p>Since <code>__getitem__</code> only uses 1 idx, there isn’t a way to load multiple file and each of their lines?</p>
<p>Can someone point me to the right direction?</p>

chenyuntc: <p><code>torch.utils.data.ConcatDataset</code>  maybe helpful</p>
<p><a href="http://pytorch.org/docs/0.3.0/data.html#torch.utils.data.ConcatDataset" class="onebox" target="_blank" rel="nofollow noopener">http://pytorch.org/docs/0.3.0/data.html#torch.utils.data.ConcatDataset</a></p>

mxlei01: <p>I was looking at ConcatDataset too, but one of my question is does it support shuffle between datasets?</p>
<p>Let’s say I have 2 datasets, A, and B.</p>
<p>Can data be shuffled from A to B, and B to A? From the code (I don’t understand the code 100%), it seems that data within their own dataset are shuffled.</p>

chenyuntc: <aside class="quote no-group" data-username="mxlei01" data-post="3" data-topic="11769">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/aeb1de/48.png" class="avatar"> mxlei01:</div>
<blockquote>
<p>Can data be shuffled from A to B, and B to A</p>
</blockquote>
</aside>
<p>yes</p>
<p>shuffle  was  performed  by  dataloader.</p>

mxlei01: <p>Gotcha, I’ll try to use ConcatDataset.</p>
<p>Thanks for your help. Really appreciated your help!</p>

adityagourav: <p>So, does it mean I need to create a Dataset for every file in the directory? I have like 118 million train rows split in 3000 files.</p>
<p>Assuming there are N rows in each file (which now is  Dataset), will the ConcatDatastet read individual Datasets into memory?</p>

mxlei01: <p>I created one dataset for each file, and if there’s only 3000 files then it isn’t that much to hold it inside an array (object that has a reference to).</p>
<p>If you wrote your DataSet with linecache, then it won’t read each file into memory.</p>
<p>At least this is my observation after reading more files than my computer’s memory can support.</p>

praateekmahajan: <pre><code class="lang-auto">class LazyTextDataset(Dataset):
    def __init__(self, filename):
        self._filename = filename
        self._total_data = int(subprocess.check_output("wc -l " + filename, shell=True).split()[0]) - 1

    def __getitem__(self, idx):
        line = linecache.getline(self._filename, idx + 1)
        csv_line = csv.reader([line])
        return next(csv_line)
      
    def __len__(self):
        return self._total_data

path = /where_csv_files_are_dumped/
files = list(map(lambda x : path + x, (filter(lambda x : x.endswith("csv"), os.listdir(path)))))
datasets = list(map(lambda x : LazyTextDataset(x), files))
dataset = ConcatDataset(datasets)
</code></pre>
<p>This should work for multiple csv files where each row is represented by a training example.</p>

Santosh-Gupta: <p>I am going to be working with many files, that all can’t fit into memory at the same time.</p>
<p>Since the data is text that I would like to be loaded quickly, I was hoping to use ether pytorch.load/pickle files, or pandas/parquet files.</p>
<p>However, it looks like <code>ConcatDatastet</code> can’t work for my case, since there’s no way to load specific lines only, like with csv files and linecache.getline, so I’ll have to load all the files at once.</p>
<p>So now I’m thinking I’ll just load a fraction of the files, as much as will fit in memory, and do a <code>ConcatDatastet</code> for those files. Perhaps using pytorch multiprocessing to load those several files at once. And then use <code>ConcatDatastet</code> of that chunk.</p>
<p>I’ll probably have to code additional shuffling code for shuffling among the chunks.</p>
<p>Does this strategy sound like the best approach? Or are there some additional Pytorch tools that can help me around this?</p>

praateekmahajan: <p>Hey <a class="mention" href="/u/santosh-gupta">@Santosh-Gupta</a>, if you’re looking for lazy loading for parquet files I’d recommend exploring <a href="https://github.com/vahidk/tfrecord" rel="nofollow noopener">https://github.com/vahidk/tfrecord</a> or <a href="https://github.com/uber/petastorm" rel="nofollow noopener">https://github.com/uber/petastorm</a>.</p>
<ul>
<li>TFRecord would require you to go from parquet to tfrecord.</li>
<li>Petastorm should work out of the box.</li>
</ul>

p568912: <p>Sorry !I’m new to pytorch.I want to check my guess is right.<br>
while calling <strong>getitem</strong>  in LazyTextDataset ,it will generate I/O one time.<br>
For example, if my training dataset is 40000 ,it will generate I/O 40000 time.<br>
Is it time-consuming job while we have to read training data?</p>

matthewleigh: <p>I have been trying to use this method to go through my dataset of around one thousand csv files, each containing about 300k lines (one training sample per line).</p>
<p>This method leads to a slow buildup of CPU memory as I iterate through the batches until it crashes.</p>
<p>I have tried putting in linecache.clearcache() at the end of each batch but it isnt clearing up the memory.</p>

matthewleigh: <p>Did you find a solution to this? It seems like exactly what I am trying to implement.</p>