Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper proposes a method to compensate the high latency brought by data IO/processing in neural network training. Specifically the authors propose to repeat training on the same subset of data during waiting time for the new data.  In this way, the data efficiency is improved, as verified by thorough experiments on various of real-world tasks.

Although the experiments look promising, I have to say the innovation of this paper is limited. The way of reusing the current data during waiting looks more like a straightforward trick, rather than a novel idea that deserved to be published at ICLR.  Furthermore, I’m wondering that how general the scenarios of “t_{downstream}>t_{upstream}” will be. Even in industrial level applications (e.g., billon level recommendation or click prediction task), AFAIK the training (including feedforward/backprop/communication in the distributed setting) consumes most of the time, while the data reading/preprocessing is comparatively cheap. Last but not least, what will the final performance will be given the potentially harmful consecutive reuse of data? Will it be worse than baseline?

review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: The authors propose a simple method for avoiding bottlenecks during NN training, whereby training examples are utilized multiple times per read. The work focuses on cases where the cost of preparing a minibatch exceeds that of a training step (both a forward pass and, subsequently, a parameter update). Working within said regime, the authors investigate different strategies for 'echoing' examples.


Feedback:
  The proposed method itself is very simple: that's fine. While some cursory analysis of data echoing's theoretical implications would be appreciated, I am fine with practically motivated solutions that address real issues. Simple 'tricks' that are easy to implement and widely applicable are often useful tools. Especially when little theoretical analysis is provided, introducing such a trick requires strong empirical evidence to validate its efficacy. As it stands, failure to provide crucial information forces the reader to suspend disbelief when evaluating the proposed method's impact.

  The authors seem to tiptoe around the issue of the relative cost of prefetching a batch versus that of a combined forward pass and parameter update. The work is predicated upon the assumption that the ratio of said costs $R > 1$, but the authors state that an unspecified subset of their experiments violate this assumption. What's more, real-world values of $R$ are not reported (or even measured!). As per Amdahl's law, $R$ upper bounds the potential benefits for data echoing; hence, failing to report $R$ is more than a little bit concerning. By the same token, the appropriate statistic for various result figures would seemingly be time rather than, e.g., the number of fresh examples read. As a reviewer, I would rather see evidence that data echoing provides modest benefits in realistic scenarios than x3.25 speedup in self-described "contrived" examples. Alternatively, consider providing real-world examples where $R > 1$ to help ground your arguments.


Questions:
  - Why was extensive hyperparameter tuning necessary?
  - Why are most results reported in terms of time/steps to achieve a target value? If nothing else, consider providing the corresponding learning curves (as in Figure 8) as an appendix (incl. means and standard errors).


Nitpicks, Spelling, & Grammar:
  - Metaparameters -> hyperparameters
  - Streamline list at end of introduction:
    """
    In this paper, we demonstrate that data echoing:
        1. reduces the...
    """
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: This paper discusses the use of data echoing (re-passing data fetched from drive or cloud) to maximize GPU usage and reduce reliance on data transportation time. The schemes basically are: reusing data at the example level, after data augmentation, or after batching. The experiments measure how much fresh data is needed in order to reach the same level of validation accuracy, with significant speedup when echoing is used. 

I thought the paper is very nicely motivated, although this is out of my area so I cannot comment on how thoroughly the problem of data fetching is investigated in other works. The evaluations are also nice, and appropriately uses a large model (Resnet 50) and dataset (Imagenet). 

The simplicity of the method is a plus, but I question a fundamental part, especially if batch echoing is used--isn't this just the same as running SGD twice, and therefore halving the stepsize and doubling the number of steps? From the optimization viewpoint, it seems that if less data was used and a good validation error level is reached, then how do we not know that less data wouldn't work well in the first place? I understand that all step sizes and decay rates were chosen independently per experiment; can those numbers be shared in a way to see if this is happening or not? Figure 8 also suggests that though it may take a long time for the baseline to reach the same level as that with batch echoing, everyone reaches a pretty low error rate at about the same point, and the difference may be in the "slow converging" phase of the optimization; thus measuring how long it takes to reach a specific low error rate may be an exaggerated measure. 

Basically, what I am saying is that the idea is nice, but the results look a bit magical. I'm happy to increase my score if the authors can upload more intermediary results, like plots of form figure 8, decay rates and schedules, batch sizes, exact repetition schedules, etc. 

minor: page 3 end of paragraph 2: "repeated data would NOT be more valuable than fresh data?"

After rebuttal: The authors have addressed my concerns and I more-or-less believe the results. I see why the contribution can be viewed as minor, but it is well-motivated and looks like a nice set of experiments. I encourage the authors to make their code available so that it can be easily incorporated in applications. 
review_title: Official Blind Review #1

Meta-review: This paper presents a simple trick of taking multiple SGD steps on the same data to improve distributed processing of data and reclaim idle capacity. The underlying ideas seems interesting enough, but the reviewers had several concerns.

1. The method is a simple trick (R2). I don't think this is a good reason to reject the paper, as R3 also noted, so I think this is fine.
2. There are not clear application cases (R3). The authors have given a reasonable response to this, in indicating that this method is likely more useful for prototyping than for well-developed applications. This makes sense to me, but both R3 and I felt that this was insufficiently discussed in the paper, despite seeming quite important to arguing the main point.
3. The results look magical, or too good to be true without additional analysis (R1 and R3). This concerns me the most, and I'm not sure that this point has been addressed by the rebuttal. In addition, it seems that extensive hyperparameter tuning has been performed, which also somewhat goes against the idea that "this is good for prototyping". If it's good for prototyping, then ideally it should be a method where hyperparameter tuning is not very necessary.
4. The connections with theoretical understanding of SGD are not well elucidated (R1). I also agree this is a problem, but perhaps not a fatal one -- very often simple heuristics prove effective, and then are analyzed later in follow-up papers.

Honestly, this paper is somewhat borderline, but given the large number of good papers that have been submitted to ICLR this year, I'm recommending that this not be accepted at this time, but certainly hope that the authors continue to improve the paper towards a final publication at a different venue.

Decision: Reject