Progressive Compressed Records: Taking a Byte Out of Deep Learning Data

Deep learning training accesses vast amounts of data at high velocity , posing challenges for datasets retrieved over commodity networks and storage devices . We introduce a way to dynamically reduce the overhead of fetching and transporting training data with a method we term Progressive Compressed Records ( PCRs ) . PCRs deviate from previous formats by using progressive compression to convert a single dataset into multiple datasets of increasing fidelity—all without adding to the total dataset size . Empirically , we implement PCRs and evaluate them on a wide range of datasets : ImageNet , HAM10000 , Stanford Cars , and CelebA-HQ . Our results show that different tasks can tolerate different levels of compression . PCRs use an on-disk layout that enables applications to efficiently and dynamically access appropriate levels of compression at runtime . In turn , we demonstrate that PCRs can seamlessly enable a 2× speedup in training time on average over baseline formats . 1 INTRODUCTION . Distributed deep learning exploits parallelism to reduce training time , and consists of three key components : the data pipeline ( storage ) , the forward/backward computation ( compute ) , and the variable synchronization ( network ) . A plethora of work has investigated scaling deep learning from a compute- or network-bound perspective ( e.g. , Dean et al. , 2012 ; Cui et al. , 2016 ; Abadi et al. , 2015 ; Cui et al. , 2014 ; Jouppi et al. , 2017 ; Lim et al. , 2019 ; Zhu et al. , 2018 ; Alistarh et al. , 2017 ; Lin et al. , 2018 ; Wen et al. , 2017 ; Wangni et al. , 2018 ; Zhang et al. , 2017 ) . However , little attention has been paid toward scaling the storage layer , where training starts and training data is sourced . Unfortunately , hardware trends point to an increasing divide between compute and networking or storage bandwidth ( Li et al. , 2016 ; Lim et al. , 2019 ; Kurth et al. , 2018 ) . For example , the transportation of data for machine learning is a key factor in the design of modern data centers ( Hazelwood et al. , 2018 ) , which are expected to be serviced by slow , yet high capacity , storage media for the foreseeable future ( David Reinsel , 2018 ; Cheng et al. , 2015 ; Rosenthal et al. , 2012 ) . This , combined with the memory wall—a lack of bandwidth between compute and memory—suggests that , while computation may be sufficient moving forward , the mechanisms for moving data to the compute may not ( Wulf & McKee , 1995 ; Kwon & Rhu , 2018 ; Hsieh et al. , 2017 ; Zinkevich et al. , 2010 ) . The storage pipeline is therefore a natural area to seek improvements in overall training times , which manifest from the storage medium , through the network , and into the compute nodes . In this work , we propose a novel on-disk format called Progressive Compressed Records ( PCRs ) as a way to reduce the bandwidth cost associated with training over massive datasets . Our approach leverages a compression technique that decomposes each data item into deltas , each of which increases data fidelity . PCRs utilize deltas to dynamically compress entire datasets at a fidelity suitable for each application ’ s needs , avoiding duplicating the dataset ( potentially many times ) at various fidelity levels . Applications control the trade-off between dataset size ( and , thus , bandwidth ) and fidelity , and a careful layout of deltas ensures that data access is efficient at a storage medium level . As a result , we find that for a variety of popular deep learning models and datasets , bandwidth ( and therefore training time ) can be easily reduced by 2× on average relative to JPEG compression without affecting model accuracy . Overall , we make the following contributions : 1 . In experiments with multiple architectures and several large-scale image datasets , we show that neural network training is robust to data compression in terms of test accuracy and training loss ; however , the amount of compression that can be tolerated varies across learning tasks . 2 . We introduce Progressive Compressed Records ( PCRs ) , a novel on-disk format for training data . PCRs combine progressive compression and careful data placement to enable applications to dynamically choose the fidelity of the dataset they consume , reducing data bandwidth . 3 . We demonstrate that by using PCRs , training speed can be improved by 2× on average over standard formats using JPEG compression . This is achieved by selecting a lower data fidelity , which , in turn , reduces the amount of data read without significantly impairing model performance . 2 BACKGROUND . Two complementary concepts make up the process of storing training data : the layout of the data on the storage medium and the representation of the data . Data layout is important because it can help fully utilize the bandwidth potential of the underlying storage system . Data representation is important because it can reduce the amount of data transferred per data unit ( i.e. , a bandwidth requirement reduction ) . An example of data representation within the scope of this work is compression , which increases the computation per bit—a key property to consider as computation increases faster than bandwidth to storage . Compression may lower image quality by introducing artifacts or blur . Record Layouts . Learning from data requires sampling points from a training set , which can cause small , random accesses that are detrimental to the performance of the storage device . Record layouts , such as TensorFlow ’ s TFRecords ( TFRecords ) or MXNet ’ s ImageRecord ( ImageRecord ) , attempt to alleviate this problem by batching data points together to increase access locality . Batches of training data ( i.e. , dataset subsets ) are then accessed together , amortizing delays in access time across multiple data points . These batches of data are called records . The key to any record layout is the serialization , which is the conversion of data structures into byte streams . Record designs have different performance properties ( e.g. , space or access time ) when written to disk , as shown in Figure 1 . Image Compression . Compressed forms are commonly used to represent training data . JPEG ( Wallace , 1992 ) is one of the most popular formats for image compression and is used ubiquitously in machine learning ( e.g. , Deng et al. , 2009 ; Russakovsky et al. , 2015 ; Lin et al. , 2014 ; Everingham et al. , 2010 ) . Most compression formats ( including JPEG ) only allow for the compression level , i.e. , the trade-off between data size and fidelity , to be set at encoding time , which often results in choosing this level independent of the application . This can result in over-compression , which may negatively impact application convergence quality , or under-compression , which results in excess data size , and thus , slower storage system performance . Worse , deep learning pipelines often involve an application-defined post-processing step ( e.g. , data augmentation ) , which may further distort an image and obscure the relationship between image fidelity and model accuracy ( Bishop , 1995 ; Karras et al. , 2018 ; Dziugaite et al. , 2016 ; Arnab et al. , 2018 ) . While setting encoding-time parameters is unavoidable , the ability to decompress data as it becomes available ( i.e. , dynamic compression ) provides a means to avoid some of the bandwidth expenses of under-compression by simply terminating decompression once sufficient fidelity is reached . In Figure 2 , we provide a high-level illustration of the JPEG algorithm , which can be customized to support dynamic compression . First , an image is split into blocks of size 8 × 8 . Each block is converted into the frequency domain , such that frequency 0 is the average color of the block , and higher frequencies encode rapid changes in the block . The low frequencies , such as the average value of the block , store the bulk of the perceptually-relevant content in the image ( e.g. , knowing the block is mostly blue is more important than knowing a white wave is rippling through it ) . Quantization , which discards information from the block and results in compression , thus prioritizes discarding higher frequencies . The resulting quantized table is then serialized into a flat form . Since data is rendered on a screen from left to right , top to bottom , it makes sense to encode the data in this manner , which results in a sequential format1 . Decoding the resulting data is simply a matter of inverting ( albeit losslessly ) the process that we just described . Progressive Image Compression . Progressive formats allow data to be read at varying degrees of compression without duplication . With the sequential case , data is ordered by blocks , and thus , partially reading the data results in “ holes ” in the image for unread blocks ( Wallace , 1992 ) . Dynamic compression ensures that all blocks get some information ( deltas ) before revising them ( with more deltas ) . As progressive formats are simply a different traversal of the quantization matrix , with all else being equal , they contain the same information as sequential JPEG ( JPEGTran LibJPEG ) . Progressive JPEG , combined with an additional rearrangement of data , forms the basis of the idea behind PCRs . In Figure 2 , non-progressive formats serialize the image matrix in one pass , while progressive formats serialize the matrix in disjoint groups of deltas which are called scans . Scans are ordered by importance ( e.g. , the first few scans improve fidelity more than subsequent scans ) . Thus , any references to images generated from scan n will implicitly assume that the image decoder had access to all prior scans ( i.e. , { scan 1 , scan 2 , . . . , scan ( n− 1 ) } ) . The bottom of Figure 2 shows how image fidelity improves from a single scan to utilizing all scans . 3 PROGRESSIVE COMPRESSED RECORDS . In this section , we introduce a novel record format for machine learning training called Progressive Compressed Records ( PCRs ) . PCRs are a combination of both layout and data representation . Efficient layouts guarantees that hardware is fully utilized ( in terms of bandwidth ) , while efficient data representations can reduce the total amount of work that is required of the system . To this end , we introduce the concept of scan groups in Section 3.1 , which leverage both layout and progressive compression to obtain dynamic compression , allowing both high performance reads while reducing the amount of data read . Using progressive compression , scan groups break images into deltas , which are then rearranged in order to facilitate reduced , yet sequential , data access . In Section 3.2 , we discuss how PCRs are implemented , covering both creating PCRs ( encoding ) and reading them ( decoding ) . The benefits of the PCR implementation boiling down to a bit shuffle are that : 1 ) PCRs are easy to implement , 2 ) they are fundamentally lossless , and 3 ) processing them is fast . As we demonstrate in Section 4 , while PCRs can be implemented easily , they manifest in large speedups for a variety of scenarios . Further , PCRs can be generalized beyond images and JPEG .

Summary: This paper introduces a new storage format for image datasets for machine learning training. The core idea is to use progressive JPEG to create sequential scans of the input image, from lower resolution to higher resolution. The authors found that on some datasets, using half of the scans is already enough to reach similar accuracy but speeded up the convergence by a factor of 2.