Dataloader for a folder with multiple files. PyTorch solutions that is equivalent to TFRecordDataset in TF2.0

username: Yin_Zheng
cooked: <p>Hi,</p>
<p>Suppose I have a folder which contain multiple files, Is there some way for create a dataloader to read the files? For example, after a spark or a mapreduce job, the outputs in a folder is like</p>
<pre><code class="lang-auto">part-00000
part-00001
...
part-00999
</code></pre>
<p>Usually the files in the folder is very large and cannot fit to memory.<br>
For TensorFlow 2.0, we can convert the file to tfrecord format and feed the folder path TFRecordDataset to read the data smoothly and efficiently, like below</p>
<pre><code class="lang-auto">raw_dataset = tf.data.TFRecordDataset(folder_path)
def _parse_example(example_string): 
    feature_dict = tf.io.parse_single_example(example_string, features)
    return feature_dict
dataset = raw_dataset.map(_parse_example)
</code></pre>
<p>However, I did not find a proper way for pyTorch to deal with this situation. I have searched the answer on the internet for a long time but I did not figure it out.<br>
So I come here to get some help.</p>
<p>Actually, I think this is a normal requirements, I would like to have the answer. And I believe the powerful PyTorch should have an elegant way to this problem, like Tensorflow 2.0.</p>
<p>Thanks very much!</p>

username: rwightman
cooked: <p>NVidia’s DALI supports reading TFRecord and MXNet recordIO. The readers are in C++ with a Python interface so it’s probably the most performant option out there.</p>
<p><a href="https://docs.nvidia.com/deeplearning/sdk/dali-developer-guide/docs/examples/dataloading.html" class="onebox" target="_blank" rel="nofollow noopener">https://docs.nvidia.com/deeplearning/sdk/dali-developer-guide/docs/examples/dataloading.html</a></p>
<p>I’ve seen a few stand alone pure Python attempts but they’ll be pretty slow as it’s not a task that Python is well suited for</p>

username: Yin_Zheng
cooked: <p>Hi Ross,</p>
<p>Thanks for the reply! The Nvidia’s DALI is a good way to reading the TFRecord and MXNet format.</p>
<p>However, I have the following questions:</p>
<ol>
<li>Does PyTorch has its own recordIO so that we do not need the TFRecord format?</li>
<li>If the files in a folder is <em>NOT</em> in TFRecord or MXNet format, but in user defined format, such as <code>label key1: value_list1 key2: value_list2 ...</code>,  then what is the recommend way to read the files by PyTorch. Note that there are multiple files in the folder and the total size of  files is large that cannot be put into memory.</li>
</ol>
<p>Thanks very much!<br>
Yin</p>

username: Yin_Zheng
cooked: <p>Hi Ross,</p>
<p>Thanks for the reply! The Nvidia’s DALI is a good way to reading the TFRecord and MXNet format.</p>
<p>However, I have the following questions:</p>
<ol>
<li>Does PyTorch has its own recordIO so that we do not need the TFRecord format?</li>
<li>If the files in a folder is  <em>NOT</em>  in TFRecord or MXNet format, but in user defined format, such as  <code>label key1: value_list1 key2: value_list2 ...</code> , then what is the recommend way to read the files by PyTorch. Note that there are multiple files in the folder and the total size of files is large that cannot be put into memory.</li>
</ol>
<p>Thanks very much!<br>
Yin</p>

username: macwiatrak
cooked: <p>Hey Yin, spark to torch dataloader does require some custom work but is fairly easy to build. How I do it is I use <code>torch.utils.data.IterableDataset</code>. Firstly I load all the avro/parquet (as you are working with spark) to a DataReader object which is a generator (where I do some of my custom processing on each record). Then I build my own <code>collate_fn</code> function to load the data from a generator to a <code>DataLoader</code> object and return a data loader. Below is a brief sketch of how I do it:</p>
<pre><code class="lang-auto">from typing import Dict, Any, Optional, List, Callable

import torch
from fastavro import reader
from functools import partial
from torch.utils.data import DataLoader

class AvroDataReader(torch.utils.data.IterableDataset):

    def __init__(self, data_location: str, custom_fn: Callable):
        self.data_location = data_location
        self.custom_fn = custom_fn

    def __iter__(self):
        avro_files = get_files(self.data_location, '*.avro')
        for filename in avro_files:
            with file_reader(filename) as local_file:
                with smart_open(local_file, 'rb') as i:
                    for record in reader(i):
                        sample = self.custom_fn(record)
                        if sample is not None:
                            yield sample


def _collate_examples(batch_of_data):
    return SomeType(
        token_ids=torch.stack([b.token_ids for b in batch_of_data]),
        label_idx=torch.stack([b.label_idx for b in batch_of_data]),
    )


def get_dataloader(dataset, batch_size: int,
                   collate_fn: Callable):
    return DataLoader(dataset, batch_size=batch_size, collate_fn=collate_fn)


def get_torch_dataloader(data_location: str, batch_size: int, label_vocab: Dict[str, int]):
    dataset = AvroDataReader(data_location=data_location,
                             custom_fn=partial(custom, label_vocab=label_vocab))
    return get_dataloader(dataset=dataset, batch_size=batch_size,
                          collate_fn=_collate_labelled_examples)
</code></pre>
<p>You need to obviously adapt your code to the example I provided, but should give you a good sketch of how it should work and should not throw any memory issues.</p>

username: Yin_Zheng
cooked: <p>Hi Mac,</p>
<p>Thanks very much for your reply! It is really clear and helpful.</p>
<p>Yin</p>

username: Yin_Zheng
cooked: <p>Hi Mac,</p>
<p>I just have another related question. I hope that I can get some suggestions from you. Thanks in advance!</p>
<p>Actually, my data is current in tfrecord format.  Is there some suggestions about how to use the tfrecords files in pytorch？ Should I read the tfrecords by tensorflow and then convert the tfdata to pytorch tensor? Maybe that is inefficient. The other plan is to convert the tfrecords to other format so that pytorch can deserialize directly. But I have no idea what I should do to serialize and deserialize data for pytorch.</p>
<p>Do you have some suggestions? Thanks very much!</p>
<p>Yin</p>

username: macwiatrak
cooked: <p>Hey Yin,</p>
<p>I think there is no ‘easy’ way to do it (i.e. a function which does it efficiently for you in pytorch). I had a similar problem and how I did it is I simply had another job which converted tfrecords to .avro, saved it, and then I read avro to the dataloader. I did it cause it was the easiest way to do and my data is usually in avro/parquet. Probably the easiest way is to built a custom function (in place of <code>custom_fn</code> in the <code>AvroDataReader</code> object in the snippet above), which converts tfrecord to something like tf.float32 and then tf.float32 (but don’t know whether you will need a tf session for that).</p>
<p><a href="https://discuss.pytorch.org/t/read-dataset-from-tfrecord-format/16409/5">This</a> thread looks very relevant.</p>

username: Yin_Zheng
cooked: <p>Hi Mac,</p>
<p>Having both TF and Pytorch in the same script is a little bit weird, and I am afraid that there might be some efficiency problem if we load data by TF and then convert it to Pytorch Tensor for Dataloader. So I think converting tfrecord to arvo and read avro to the dataloader is really a good idea.<br>
Could you please show me a sketch about how to convert tf data to arvo and load them to pytorch tensor? Thanks very much! I really appreciate it.</p>
<p>I have also find the link you sent to me. However, it seems this problem is not addressed in that post. People still complain about the efficiency.</p>
<p>Regards,<br>
Yin</p>

username: Yin_Zheng
cooked: <p>Hey Mac,</p>
<p>I have check the link you sent to me. However, it seems this problem is not addressed in that post. People still complain about the efficiency.</p>
<p>Having both TF and Pytorch in the same script is a little bit weird, and I am afraid that there might be some efficiency problem if we load data by TF and then convert it to Pytorch Tensor for Dataloader. So I think converting tfrecord to arvo and read avro to the dataloader is really a good idea.<br>
Could you please show me a sketch about how to convert tf data to arvo and load them to pytorch tensor? Thanks very much! I really appreciate it.</p>
<p>Regards,<br>
Yin</p>

username: macwiatrak
cooked: <p>Hey Yin, I agree, having both TR and Pytorch is not a good idea, not only in the same script but also in the same python module.</p>
<p>I don’t have the exact code anywhere near as it was a while ago since I used tfrecords, but databricks seems to have some good documentation. See it below:</p>
<pre><code class="lang-auto">from pyspark.sql.types import *

schema = StructType([StructField('image/class/label', IntegerType(), True),
                     StructField('image/width', IntegerType(), True),
                     StructField('image/height', IntegerType(), True),
                     StructField('image/format', StringType(), True),
                     StructField('image/encoded', BinaryType(), True)])

df = spark.read.format("tfrecords").schema(schema).load(input_local_dir+'/flowers_train*.tfrecord')
df = df.limit(3200)
display(df)
</code></pre>
<p>In general, the idea is to read the tfrecords to a pyspark dataframe (do same preprocessing on it) and then save it as avro with <code>df.write.format("avro").save(/tmp/loc)</code>.</p>
<p>Link to the databricks notebook on tfrecords is <a href="https://docs.databricks.com/_static/notebooks/deep-learning/tensorflow-tfrecord.html" rel="nofollow noopener">here</a>.</p>
<p>Also, happy to see people using pytorch and spark, big fan of the combination!</p>