Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: The paper demonstrates an interesting application of progressive compression to reduce the disk I/O overhead of training deep neural networks. The format encodes the trade-off between data fidelity and I/O bandwidth demand naturally, which could be useful when I/O is the bottleneck.

My major concern is that the paper should be clearer about the setting.
* Does your work target the case where data cannot fit in RAM and should be fetched from local disk or through network? However, the datasets used in the evaluation look small and could fit in RAM.
* How are mini-batches created? You mentioned in the related work that previous work (Kurth et al., 2018) lets each worker sample from a local subset instead of performing a true sampling of the whole dataset. Does your work perform true sampling? How much benefit does it give?
* Is disk I/O really a bottleneck in training? There are many evidence [1][2][3] of almost linear scalability in training ResNet on *full* imagenet across hundreds or even thousands of GPUs. These work focus heavily on network communication rather than disk I/O. Does your setting differ from theirs? How does your approach compare with their techniques for optimizing disk I/O?

That being said, I think this approach should be appealing when the I/O bandwidth is limited and dynamic. Examples include training on edge devices, or federated training where data needs be fetched via ad-hoc network.

Other detailed comments:

* Figure 1 is not very informative and quite puzzling. There is no definition of quality at that point.
* Sec 2 paragraph 3. What is the issue of data augmentation with the standard JPEG compression? Does your compression ease data augmentation?
* Sec 3.1 paragraph 1. "This is turn enables ..." -> "This in turn enables ..."
* How to decide the number of scans? Does it have impact on the I/O efficiency?
* Evaluation
  - I'm not familiar with Ceph. Why choose this particular environment? Does it bring in extra overhead (e.g., communicating with metadata server). What does the network topology look like? Is the data loading stall (figure 7) due to network congestion?
 - It worth evaluating more tasks such as detection and segmentation to measure the impact of compression.


[1] Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour, Goyal et al.
[2] Massively Distributed SGD: ImageNet/ResNet-50 Training in a Flash, Mikami et al.
[3] Image Classification at Supercomputer Scale, Ying et al.
review_title: Official Blind Review #4

final_score: rating: 6: Weak Accept
review_content: This paper introduces Progressive Compressed Records (PCR) which is an on-disk format for fetching and transporting training data in an attempt to reduce the overhead storage bandwidth for training large scale deep neural networks. This is a well written paper that includes all the required background and related works, as well as an easy-to-understand example that runs through the manuscript, explaining what the reader needs to know in order to appreciate the work. The empirical results of several experiments show that the PCR requires up to two times less storage bandwidth while retaining model accuracy.

My only concern is that although the related work section provides a thorough survey of the current methods in the literature, the authors did not demonstrate the performance of state-of-the-art and compare their performance with them. I believe this is necessary to truly validate the superiority of their method over state-of-the-art.

review_title: Official Blind Review #5

final_score: rating: 3: Weak Reject
review_content: The paper proposes using progressive encoding of images and re-arrange of data blocks in images to improve reading speed and therefore training speed.

To fully analyze the maximum possible speed of training, it would be great to the measure upper bound of images/sec, when avoiding reading from disk and just using images from memory. 

Decoding a typical progressive JPEG image usually takes about 2-3 times as much time as decoding a non-progressive JPEG, for full resolution, analyzing the time to read vs time to decode the images would be great. It is not clear how changing the number of total groups would affect the image size and the reading speed.

Based on the current experiments it is not clear what is the impact of the batch size when creating PCRs and when reading the image blocks, or the impact of the batch size on the training speed.

Figure 3 is really hard to read and compare times to convergence, authors should provide a table with times to X% accuracy.  Although time to convergence is the key metric, it would be great to know the difference in images/sec of different settings.

Using ImageNet 100 classes (not clear how the 100 classes were chosen) instead of the usual 1000 classes, can distort the results, since it is not clear if higher resolution would be needed to distinguish more classes or not.

Have the authors considered other image compression formats like WebP? How tie is the proposed record encoding with the image compression?  
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: Summary: This paper introduces a new storage format for image datasets for machine learning training. The core idea is to use progressive JPEG to create sequential scans of the input image, from lower resolution to higher resolution. The authors found that on some datasets, using half of the scans is already enough to reach similar accuracy but speeded up the convergence by a factor of 2.

Detailed feedbacks:
-	The paper presents a simple idea that directly uses the nature of JPEG compression. The paper shows that it can work well and can be potentially integrated into real machine learning dataset storage applications.
-	Related work section is thorough.
-	The experiments are limited to image classifications, and some of the datasets are subsampled (e.g. ImageNet and CelebA). This may not well represent real machine learning tasks, and practitioners may be unsure about the reliability of the compression. The “Cars” dataset contains fine-grained classification, in which the proposed method is
-	Figure 1 is not very clear what is the key advantage of the proposed method, and what are the different mechanisms.
-	Alternatively, one can subsample the pixels and store incremental subsets of those pixels. It would be good if the paper can discuss about this baseline.
-	The data storage format is only loosely related to the main goal of the paper, which is to show that network can still train very well and even faster when receiving partial input data. Once they figured out the number of scans needed for this application, they don’t necessarily need to keep a full lossless version and can just go for a lossy version. In other words, the experiment section can be replaced by any other lossy compression by varying the compression ratio.
-	In my opinion, there could be two reasons for faster convergence. 1) lowered image quality makes the data easier to learn and 2) the smaller data size allows faster reading of data from disk. The paper only shows wall-clock speed-up, but it is unclear which factor is bigger. 2) can be potentially addressed by faster disk reading such as SSD or in-memory datasets. One of the motivations is to help parallel training of dataset and it is also mentioned how non-random sampling of data can hurt training performance. It would be good to showcase how the proposed method can help in those parallel training settings. 

Conclusion: This paper presents a simple and effective idea and can be potentially beneficial. However, my main concern is whether the experiments can be representative enough for large scale experiments (e.g. using non-subsampled ImageNet dataset with parallel training using SSD storage). Therefore, my overall rating is weak accept.
review_title: Official Blind Review #3

Meta-review: Main content: Introduces Progressive Compressed Records (PCR), a new storage format for image datasets for machine learning training.
Discussion:
reviewer 4: Interesting application of progressive compression to reduce the disk I/O overhead. Main concern is paper could be clearer about setting. 
reviewer 5: (not knowledgable about area): well-written paper. concern is that related work could be better, including state of the art on the topic.
reviewer 2: likes the topic but discusses many areas for improvement (stronger exeriments, better metrics reported, etc.). this is probably the most experienced reviewer marking reject.
reviewer 3: paper is well written. Main issue is that exeriments are limited to image classification tasks, and it snot clear how the method works on larger scale.  
Recommendation: interesting idea but experiments could be stronger. I lean to Reject.
Decision: Reject