Faster Neural Network Training with Data Echoing
Authors: Dami Choi
Alexandre Passos
Christopher J. Shallue
George E. Dahl
Venue:  (2020)
Keywords: systems
faster training
large scale

Abstract: In the twilight of Moore's law, GPUs and other specialized hardware accelerators have dramatically sped up neural network training. However, earlier stages of the training pipeline, such as disk I/O and data preprocessing, do not run on accelerators. As accelerators continue to improve, these earlier stages will increasingly become the bottleneck. In this paper, we introduce “data echoing,” which reduces the total computation used by earlier pipeline stages and speeds up training whenever computation upstream from accelerators dominates the training time. Data echoing reuses (or “echoes”) intermediate outputs from earlier pipeline stages in order to reclaim idle capacity. We investigate the behavior of different data echoing algorithms on various workloads, for various amounts of echoing, and for various batch sizes. We find that in all settings, at least one data echoing algorithm can match the baseline's predictive performance using less upstream computation. We measured a factor of 3.25 decrease in wall-clock time for ResNet-50 on ImageNet when reading training data over a network.

Under review as a conference paper at ICLR 2020
FASTER NEURAL NETWORK TRAINING WITH DATA ECHOING
Anonymous authors Paper under double-bling review
ABSTRACT
In the twilight of Moore's law, GPUs and other specialized hardware accelerators have dramatically sped up neural network training. However, earlier stages of the training pipeline, such as disk I/O and data preprocessing, do not run on accelerators. As accelerators continue to improve, these earlier stages will increasingly become the bottleneck. In this paper, we introduce "data echoing," which reduces the total computation used by earlier pipeline stages and speeds up training whenever computation upstream from accelerators dominates the training time. Data echoing reuses (or "echoes") intermediate outputs from earlier pipeline stages in order to reclaim idle capacity. We investigate the behavior of different data echoing algorithms various workloads, for various amounts of echoing, and for various batch sizes. We find that in all settings, at least one data echoing algorithm can match the baseline's predictive performance using less upstream computation. We measured a factor of 3.25 decrease in wall-clock time for ResNet-50 on ImageNet when reading training data over network.
INTRODUCTION
Over the past decade, dramatic increases in neural network training speed have facilitated dramatic improvements in predictive performance by allowing researchers to train bigger models using larger datasets and to explore ideas more rapidly. As Moore's law ends, general purpose processors no longer rapidly becoming faster, but specialized hardware continues to drive significant speedups by optimizing for narrower of operations. For example, GPUs and TPU TPU-Dopptmizz optimize for highly parallelizable matrix operations, which are core components of neural network training algorithms.
However, neural network training requires more than just the operations that run well on accelerators a training program may need to read and decompress training data, shuffle it, batch it, and even transform or augment it. These steps exercise multiple system components, including CPUs, disks, network bandwidth, and memory bandwidth. It is impractical to design specialized hardware for all these general operations that involve so many different components. Moreover, these operations are not simply executed once at the start of the training program. Since many of today's datasets are too large to fit into an accelerator's memory or even the host machine's 's main memory, most large-scale neural network training systems stream over the training data, incrementally reading it from disk, pre-processing it in main memory, and copying successive batches training examples to the accelerator, which runs the training algorithm. Therefore, each training step involves mixture of operations that do and do not run on accelerators.
There are workloads where the code running on accelerators consumes only small portion of the overall wall time, and this scenario will only become more common accelerator improvements continue to outpace improvements in CPUs. In order to speed up training in these cases, must either (1) make the non-accelerator work faster, or (2) reduce the amount of hon-accelerator work required achieve the desired performance. Option (1) is appealing but requires substantial engineering labor or problem-speciffc techniques (e.g. Ying et al., 2018: Kumar et al.) 2018). Adding more workers might be too expensive. Instead, we focus on option (2) and explore techniques for reducing the total amount of work spent reading and preparing inputs in the training pipeline.
httpp: //www.bbwwb//////// bl bl...looogooooo...oo..o goog e /pproducts /goog la-cl oudd www-ooooo-oo-------------- ne- wew-eeeeelrrrree ng/
For example, after decoding and standard pre- pre-processing, the ImageNet dataset (Russakovsky et al. 2015 700 GB and the Common Crawl dataset is 6.7 TB.

---

Under review as a conference paper at ICLR 2020
Read and decode
Apply augmentation
Apply SGD update
Shuffle
Batch
Figure 1: The training pipeline for ResNet-50 on ImageNet, which is representative of many large- scale computer vision programs.
Upstream
Upstream
Upstream
Upstream
Downstream
Idle
Downstream
Idle
Downstream
Downstream
Downstream
Downstream
time
time
(a) Without data echoing, downstream computational capacity is idle 50% of the time.
(b) Data echoing with echoing factor 2 reclaims downstream computational capacity.
Figure 2: The overlapping computation time for pipeline stages upstream and downstream of the data echoing insertion point, if stages are executed parallel and lupstream dowwmrrrem
Figure Showw the data processing and training pipeline for ResNet-50 (He et al. 2016) on ImageNet, which is representative of many lcage computer vision programs. First, the training program reads each image from disk, decompresses into a dimensional array of values, and pushes into shuffle buffer. The next stage of the pipeline samples images at random from the shuffle buffer to approximate shuffling the entire dataset, but with a fixed memory budget. The next stage performs pre-processing and data augmentation to a 224 x 224 x 3 array, then randomly horizontally flipped, and finally has colors randomly jittered. These random distortions help improve the generalization of vision models, and while these particular operations are specific to images, almost every deep learning pipeline performs some kind of pre-processing its input data. Finally, images and labels are gathered into batches and sent to the accelerator to perform a step of minibatch stochastic gradient descent (SGD). For brevity, we will refer to the operation that updates the model's parameters for given batch of training examples as the "SGD step" throughout the paper, even though variants of the basic SGD algorithm are also popular. Our technique applies equally well for any training algorithm that works on successive batches of training examples.
each image is randomly cropped and resized
To maximize throughput, the training program is often executed as a pipeline process, so that each stage in Figure[ooprrann in parallel from the other stages. Each stage might further employ multiple parallel worker threads or machines. If any of the stages upstream from the SGD step cannot process images at the same rate as the SGD step, the accelerator will be partly idle (see Figure 2a). This can happen for many reasons, including slow transfer from disk or cloud storage, time-consuming pre- reepprocessing operations, or inadequate tuning of the number of CPU threads dedicated each stage of the pipeline. While it can be possible to improve training time by dedicating engineering effort optimizing the input pipeline, such efforts often time consuming and can distract from the practitioner's main goal of improving their model's predictive performance. Instead, we propose data echoing as a simple, cheap, and effective method for reclaiming idle accelerator capacity. Rather than waiting for more data to become available, we propose simply reusing data that is already available. We do this by adding stage the pipeline that repeats (or "echoes"") data from the previous stage. Once practitioner identifies the largest bottleneck in the training pipeline, they can insert an echoing stage after reclaim idle accelerator capacity (see Figure 2b).
In this paper, we demonstrate that:
1. data echoing reduces the amount of upstream computation needed to reach competitive out-of-sample error rate on various datasets and model architectures;
2. data echoing can provide a walltime speedup in practice;
3. data echoing support wide range of echoing factors;
4. the effectiveness data echoing depends on the insertion point the training pipeline;
5. data echoing can benefit from additional shuffling after echoing, but does not require it; and
6. countering expectations, data echoing reaches the same final error rate as well-tuned base- lines.
2

---

Under review as a conference paper at ICLR 2020
1.1
RELATED WORK
Data echoing shares similarities with experience replay (Mnih et al. 2015), which samples batches from buffer containing reinforcement learning agent's past experiences prevent the most recent interactions from dominating the updates. Although both data echoing and experience replay reuse previous data, our implementation of data echoing specifies the number of times to repeat each example, whereas most implementations of experience replay do not control this explicitly. In addition, local SGD algorithms (Zinkevich et al., 2010; Zhang et al. 2016), which perform multiple local model updates before communicating globally, can also be viewed reusing data to save computation. However, local SGD targets communication overhead between workers and thus is orthogonal to data echoing.
We are aware of two previous papers that describe variants of data echoing. Fischetti et al. (2018) describe special case of data echoing they call "minibatch persistency" that reuses minibatches for multiple consecutive SGD updates. They run experiments on CIFAR-10, but do not tune hyperparam- eters for the baseline for their method. Neither their method nor their baseline reach competitive test set numbers in their experiments, leaving open the question of whether minibatch persistency has an advantage over well-tuned baseline. Similarly, [oofer al. (2019) describe special case of data echoing they call "batch augmentation" that repeats examples multiple times within a given batch, but with different augmentations. None of their experiments tune optimization hyperparameters. although their baselines use settings taken from the original papers that introduced each model. Both Fischetti al. 2018) and Hoffer et al. (2019) primarily motivate their work as methods to improve generalization, only tangentially mentioning the possibility of reclaiming idle computational capacity. We would not expect data echoing to improve generalization for fixed batch size and number of SGD updates, since then repeated data would be more valuable than fresh data. Our experiments in Section 3 only show that data echoing can achieve better out-of-sample error for the same amount of fresh data.
Another related line of work accelerates neural network training by subsampling examples non- uniformly [Katharopoulos & Fleuret 2018). These methods superficially similar to data echoing, in that their goal is also faster training and that this is accomplished by modifying the examples provided the accelerator, but fundamentally these approaches work best in different scenarios. With example subsampling, each training example is processed, on average, less than once per epoch, and faster convergence is achieved by reducing the overall time the accelerator spends processing examples. With data echoing, the opposite is true: each training example is processed, on average, more than once per epoch, and faster convergence is achieved by using more accelerator computation per example instead of less. Data echoing does not affect the training data distribution, so it does not change the optimal decision boundary. However, when using non-uniform subsampling, care must be taken to ensure that one converges to the correct minimizer of the non-sampled loss. This is extremely relevant in probability matching tasks like language modeling, but should not be ignored even in other models. In principle, nothing prevents particular training pipeline from adopting both methods, though systems considerations might make it difficult to observe an actual speedup.
2
DATA ECHOING
We implement data echoing by inserting a stage in the training pipeline that repeats (echoes) the outputs of the previous stage. In TensorFlow's (Abadi et al. 2016) tf data library, echoing stage is as simple as
dataset. fl flpt_map (1 alabbda t: ff data. .Dttase.. fron_tensors (t). rrepeat (e)
where e is the data echoing factor, the number of times each data item is repeated. In some cases, we also shuffle the outputs of the echoing stage, but this can require additional memory. If the overhead of repeating data is negligible and the stages on either side of echoing are executed in parallel (e.g. Chien et al. 2018), then the average time for data echoing to complete one upstream step and e downstream steps is
(1) where Eupstream is the time taken by all stages upstream of echoing, &dwwnstream is the time taken by all stages downstream of echoing, and the echoing factor. Non-integral echoing factors can be
max (lupaream: X (doonsstremm
3

---

Under review as a conference paper at ICLR 2020
achieved in expectation by probabilistically repeating data items. We assume that fupstream downsttream throughout the paper, since this is the primary motivation for using data echoing. If we denote the ratio of srrpeamm-to-dosssteee
processing time by R Empstream/ gagssteamm/dwwwteeaammm dowestreamm then the time to complete one upstream step and e downstream steps is constant for all echoing factors less than or equal to to R. In other words, if e I R. the additional downstream steps per upstream step "free" because they utilize idle downstream capacity.
Data echoing aims to decrease training time by reducing the number of upstream steps required to achieve a target predictive performance. When using data echoing, each upstream step is used for (instead of 1) downstream SGD updates. If the required number of SGD updates with data echoing is the same as without, then training time will decrease by a factor of e. However, since repeated data might be less valuable than completely fresh data, data echoing might require more downstream SGI) updates to reach the desired predictive performance, and so the speedup factor might be less than e. We investigate the effect of data echoing on training time in Section 3
Given that every operation in the training pipeline takes some time to execute, the amount of idle downstream time that data echoing can exploit is greatest the echoing stage is applied just before the SGD update. For the ResNet-50 training pipeline in Figure TT this would result in the same minibatch of training examples being used multiple times per epoch. However, we might prefer to insert data echoing earlier in the pipeline if it provides a more favorable trade-off between the number of upstream steps and downstream steps. In particular, the following factors influence the behavior of data echoing at different insertion points:
Echoing before or after batching: Echoing before batching means data is repeated and shuffled at the example level instead of the batch level. This increases the likelihood that nearby batches will be different, at the expense of potentially duplicating examples within a batch. Whether diversification across batches or within batches is more important an empirical question that we address in Section 3) We call the class of algorithms that echo before batching example echoing and the class of algorithms that echo after batching batch echoing.
Echoing before or after augmentation: Echoing before data augmentation allows repeated data to be transformed differently, potentially making repeated data more akin to fresh data. Methods like dropout that add noise during the SGD update can similarly make repeated data appear different (Hoffer et al. 2019), even in the absence of augmentation or when echoing after augmentation.
The behavior of data echoing is also influenced by the amount of shuffling (if any) performed after the echoing stage. Adding shuffle buffer increases the likelihood that nearby SGD) updates use different data, which is likely beneficial. The larger the buffer size, the more repeated data are shuffled, and the closer the training procedure approximates a program that loads the entire training set in memory before sampling data at random. However, since we focused on large-scale workloads, assume that we can only afford a buffer size that is a relatively small fraction of the (augmented) dataset size.
3 EXPERIMENTS
We evaluated data echoing on two language modeling tasks, two image classification tasks, and one object detection task. For language modeling, we trained the Transformer model (Vaswani et al., 2017) on the LMIB (Chelba et al. 2014) and Common Craw datasets. For image classification, we trained ResNet-32 (He et al.) 2016) on the CI dataset (Krizhevsky & Hinton| 2009), and ResNet-50 on the ImageNet dataset RRussakovsky et al. (015).. For object detection, we trained the Single Shot Detector (SSD, Liu et al. (016)) on the COCO dataset (Lin et al. 2014).
The primary question we investigated was whether data echoing could provide a training speedup. We measured training cost the number of "fresh" training examples" required to reach target out-of-sample metric value. The number of fresh examples is proportional to the number of upstream steps in the training pipeline, and therefore proportional to wall time if the echoing factor is less than or equal to the ratio of upstream-tor -o-rammt-toooonttraam processing time, R (see Section 2. We did not assume or measure the value of R in most of our experiments, since R depends on the implementation and sometimes on irregular factors like network congestion. Not all of our tasks satisfied I 2 1 in
http:/ / commoncrawl www.w.orr///.....oooo wwwor///////0000 22201/00//0 /unno//jnne-// 20 1 7- crwww-ww---- wwwww-archii a
"aah time training example is read from disk, it counts as fresh example.
4

---

Under review as a conference paper at ICLR 2020
Table Tasks summary.
Model
Dataset(s)
Task
Evaluation metric
Target
LMIB,
Transformer
Language modeling
Cross entropy
3.9
Common Crawl
ResNet-32
CIFAR-10
Image classification
Accuracy
91%
ResNet-50
ImageNet
Image classification
Accuracy
75%
SSD
COCO
Object detection
mAP
0.24
all experiments. Instead, we designed most of our experiments to investigate whether data echoing could reduce the number of fresh examples needed across various tasks, since this measurement is implementation independent. We confirm that the number of fresh examples is a proxy for walltime in Section 3.2
For each workload, we ran an initial set of experiments without data echoing and tuned the hyperpa- rameters achieve the best out-of-sample performance within a practical computational budget We selected the target metric value to be slightly worse than the best observed in the initial experiments ensure it could be reached reliably. We verified that small changes to our targets did affect our conclusions. Table summarizes the workloads and target metric values we used in our experiments.
We trained the SSD model using SGD with momentum (Polyak] 1964 Rumelhart et al.) 1986) and the Transformer and ResNet models using Nesterov momentum (Nesterov) 1983] Sutskever et al. 2013). We used a constant learning rate for Transformer, and we used learning rate schedules for ResNet (linear decay) and SSD (linear warmup for 3.5 epochs followed by piecewise exponential decay). We preprocessed the text datasets identically to Shalluu et al.] (018). We augmented the image datasets at training time by resizing each image, taking random crop, and randomly horizontally reflecting the cropped images. We randomly distorted the image colors for ImageNet and COCO. Unless otherwise specified, used batch size of 1024 for Transformer and ResNet-50, 128 for ResNet-32, and 256 for SSD. We used batch normalization (loffe & Szegedy (015)) for ResNet-50 and SSD with virtual batch sizes (Hoffer et al. 2017) of 32 and 128, respectively.
In each experiment, we independently tuned the learning rate, momentum, and, where applicable, the parameters governing the learning rate schedule We manually chose the search spaces based on our initial experiments, and we verified after each experiment that the optimal hyperparameter values were away from the search space boundaries. We used quasi-random search (Bousquet et al.) 2017) to tune the hyperparameters with fixed budgets of non-divergen(" trials (100 for Transformer and ResNet-32, and 50 for the more expensive ResNet-50 and SSD models). We then chose the trial that reached the target metric value using the fewest number of fresh examples. We repeated this hyperparameter search 5 times for each search space. All figures this section show the mean number of fresh examples required over these 5 searches, with the minimum and maximum shown as error bars. Appendix A shows the average training curve over these searches for every experiment in this section.
The baseline training program for all of our workloads is shown in Figure Our experiments evalu- ated the effects of adding data echoing various points in the training pipeline. We considered three variants of data echoing: example echoing before augmentation, example echoing after augmentation, and batch echoing. For the example echoing variants, omitted the baseline's "shuffle examples" buffer and inserted a shuffle buffer after the echoing stage with the same size as the baseline's buffer. For batch echoing, we kept the baseline's shuffle buffer and repeated batches without shuffling after the "batch examples" stage. Therefore, our training pipeline always had one shuffle buffer with the same size in all cases, so all data echoing variants used the same amount of memory as the baseline.
$20k steps for LMIB, 60k for Common Crawl, 110k for ImageNet. 150k for CIFAR-10, 10, and 30k for COCO. 'For ResNet, we tuned the number steps over which the learning rate would linearly decay by a factor 100 (for CIFAR-10) or 1,000 (for ImageNet). For SSD, we tuned the two epoch numbers at which the learning rate decayed by a factor of 10.
We discarded trials that diverged, which typically occurred when the learning rate too high.
5

---

Under review as a conference paper at ICLR 2020
x10
x10
21.07
Fresh Examples Read
1.2
1.5
4.0
1.0
0.6
2.0
0.2
0.0
0.0
(a) Transformer on LMIB
(b) Transformer on Common Crawl
(c) ResNet-32 on CIFAR-10
-160
210
Fresh Examples Read
Baseline
0.6-
5.0
Batch echoing
4.0
Example echoing augmentation (if any)
3.0
Example echoing before augmentation
2.0
0.2
0.0
(d) ResNet-50 ImageNet
(e) SSD COCO
Figure 3: Data echoing with echoing factor 2 either reduces does not change the number of fresh examples needed reach the target out-of- sa--pp performance. Dashed lines indicate the expected values if repeated examples were as useful as fresh examples.
We used buffer sizes of 10t for LMIB and Common Crawl, 10 for CIFAR-10, 10 for ImageNet, and 104 for COCO. We explored the effects of increasing the buffer sizes in Section 3.5
3.1 DATA ECHOING CAN REDUCE THE NUMBER OF FRESH EXAMPLES REQUIRED FOR TRAINING
Figure shows the effect of data echoing with echoing factor 2 for all workloads in Table I In all but one case, data echoing requires strictly fewer fresh examples than baseline to reach the target out-of-sample performance. The sole exception (batch echoing ResNet-50) requires about the same number fresh examples as the baseline data echoing provides no benefit, but does not harm training either. The earlier echoing inserted in the pipeline, the fewer fresh examples needed: example echoing requires fewer fresh examples than batch echoing, and echoing before data augmentation requires fewer fresh examples than echoing after. We did not observe any negative interaction between data echoing and batch normalization for ResNet-50 or SSD.
3.2
DATA ECHOING CAN REDUCE TRAINING TIME
If the training time is dominated by upstream operations like reading and pre-processing input data, data echoing should provide a walltime speedup proportional the reduction in the number of fresh examples needed for training. To confirm this, we constructed training pipeline dominated by input latency by streaming our training data from cloud storage service. While this setup was somewhat contrived, streaming training data over network is realistic for many large-scale production workloads. For ResNet-50 on ImageNet, the ratio of postream-to-downsttee training pipeline was approximately R 6, although the exact number fluctuated with the network transfer rate.
time in this
We ran experiments with ResNet-50 on ImageNet with echoing factors between I and 5. Since the bottleneck was transferring data from cloud storage, we inserted data echoing early in the training pipeline before data augmentation. We tuned the hyperparameters separately for each echoing factor, and selected the hyperparameters that reached 75.3% validation accuracy the fastest. We then ran the optimal hyperparameters times for each echoing factor and recorded the number of fresh

---

Under review as a conference paper at ICLR 2020
x10
x10
Fresh Examples Read
Baseline
Baseline
Echoing factor 2
Echoing factor
0.8
Echoing factor -
Walltime (sec)
1.5
Echoing factor --
Echoing
Echoing factor
0.6
Echoing factor
Echoing factor
1.0
0.5
0.2
0.0
0.0
Figure 4: Example echoing before augmentation can reduce training time for ResNet-50 on ImageNet. Dashed lines indicate the expected values if repeated examples were useful fresh examples and there was no overhead from echoing.
x10
x10
Basoline
Fresh Examples Read
Exaiple echoing
Fresh Examples Read
Example echoing
4.0
2.0
0.5
0.0
0.0
1.0
2.0
4.0
8.0
16.0
1.0
2.0
4.0
8.0
16.0
Data Echoing Factor
Data Echoing Factor
(a) Batch size 1024
(b) Batch size 4096
Figure 5: Example echoing reduces the number of fresh examples needed for Transformer on LM1B for echoing factors up (at least) 16. Dashed lines indicate the expected values if repeated examples were as useful as fresh examples.
examples and walltime required reach 75% accuracy. Figure Issows that data echoing does indeed provide walltime speedup proportional to the number of fresh examples read over the network. The reduction walltime is slightly lower than the fractional reduction fresh examples because implementation of data echoing incurs slight overhead. Nonetheless, data echoing provides significant speedup for all echoing factors, up to a speedup factor 3.25 for echoing factor 5.
3.3 DATA ECHOING CAN BE USEFUL UP TO A REASONABLE UPPER BOUND ON THE ECHOING FACTOR
Figure shows the effect of example echoing with echoing factors up to 16 for Transformer on LMIB. For batch size 1024, the maximum useful echoing factor is somewhere between and 8; beyond this value, the number of fresh examples required is larger than for smaller echoing factors. As the echoing factor increases, the number of fresh examples required must eventually exceed the baseline, but even echoing factor large 16 still requires significantly fewer fresh examples than the baseline. For batch size 4096, the maximum useful echoing factor is even larger than 16, suggesting that larger batch sizes can support larger echoing factors than smaller batch sizes.
3.4 DATA ECHOING AS BATCH SIZE INCREASES
With larger batch sizes, batch echoing performs better, but example echoing sometimes requires more shuffling. Figure (hoowss the effect of data echoing with echoing factor for different batch sizes. As the batch size increases, the performance of batch echoing relative the baseline either stays the
7

---

Under review as a conference paper at ICLR 2020
x10'
x10
Baseline
Baseline
Fresh Examples Read
3.5
Batch echoing
Fresh Examples Read
Batch echoing
Example echoing
Example echeing after augmentation
Example echoing augmentation
2.5
2.0
1.5
0.6
1.0
0.4
0.2
0.0
0.0
2
2
2
2
2
2
2
Batch Size
Batch Size
(a) Transformer on LMIB
(b) ResNet-50 on ImageNet
Figure As the batch size increases, performance batch echoing relative to the baseline either stays the same or improves, while for example echoing it either stays the same gets worse. Dashed lines indicate the expected values repeated examples were as useful as fresh examples.
same or improves. This effect makes sense given that repeated batches should approximate fresh batches the batch size approaches the training set size, and in the limit, batch echoing must reduce the required number of fresh examples by the echoing factor. On the other hand, Figure 6 shows that the performance of example echoing relative to the baseline either stays the same or gets worse as the batch size increases. Since the expected fraction of duplicate examples within each batch increases with the batch size, example echoing with larger batches may behave more like smaller batch size in practice. A smaller batch size may increase the required number of SGD updates (Shallue al., 2018), which could explain the example echoing results in Figure 6 Increasing the amount of shuffling for repeated examples (at the cost of additional memory) could improve the performance of example echoing larger batch sizes by reducing the probability of duplicate examples in each batch.
3.5 DATA ECHOING PERFORMS BETTER WITH MORE SHUFFLING
Figure 7 shows the effect of increasing the shuffle buffer size (at the cost of additional memory) for data echoing with echoing factor 2. While all batch echoing experiments in the previous sections repeated batches without shuffling, the performance of batch echoing improves if repeated batches are shuffled, with more shuffling giving increasingly better performance. Similarly, the performance of example echoing improves with increasing shuffle buffer size, even though it does not help the baseline. This because more shuffling reduces the probability of duplicate examples within each batch, as discussed in Section 3.4
3.6 DATA ECHOING DOES NOT HARM PREDICTIVE PERFORMANCE
Although one might be concerned that reusing data could harm final predictive performance, we did observe any case where data echoing with a reasonable echoing factor failed to reach our target metric value. To further demonstrate that data echoing does not degrade solution quality, we ran experiments with Transformer on LMIB and ResNet-50 on ImageNet to find the best achievable performance within fixed budget of fresh examples, both with and without data echoing. We picked the fresh-examples budgets so that the baseline models would achieve at least our target metric values from Table] I We used an echoing factor of 4 for all data echoing experiments. We tuned the hyperparameters for the baseline and for all data echoing variants using 500 trials for Transformer and 100 trials for ResNet-50. Figure 8 shows the trials that reached the best out-of- sample performance any point during training for each experiment. All data echoing variants achieved at least the same performance the baseline for both tasks.
8

---

Under review as a conference paper at ICLR 2020
x10'
x10
Fresh Examples Read
Buffer size 1
Fresh Examples Read
Example echoing
10-
Burfer size 10
1.0
0.5
0.5
0.0
10a
10
10'
Shuffle Buffer Size
(a) Transformer on LMIB with batch echoing
(b) Transformer LMIB with example echoing
Figure 7: Data echoing performs better with more shuffling. Dashed lines indicate the expected
values if repeated examples were as useful as fresh examples.
Validation Cross Entropy
0.8
Baseline
Validation Accuracy
5.5
Batch echoing
0.7
Example echoing
5.0
0.5
Baseline
Batch echoing
4.0
Example echoing after augmentation
Example echoing augmentation
3.5
0.3
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
Fresh Examples Read
x10
Fresh Examples Read
x10
(a) Transformer on LMIB
(b) ResNet-50 ImageNet
Figure 8: Individual trials that achieved the best out-of-sample performance during training.
CONCLUSION
Data echoing a simple strategy for increasing hardware utilization when the training pipeline has bottleneck in one of the upstream stages. Although a priori might worry that SGD updates with repeated data would be useless or even harmful, for every workload we considered, at least one variant of data echoing reduced the total number of examples was true even for Transformer on Common Crawl, a dataset so large that we do not even train for full epoch. In this case, data echoing reached the target predictive performance while seeing only a subset of the examples seen by the baseline. Echoing after augmentation was still effective at reducing the total number of examples read from disk, making it appealing for image datasets that employ expensive data augmentation that runs on the CPU. If reading input data is the bottleneck, then echoing before augmentation will typically provide the greatest speedup. We measured factor of 3.25 decrease in walltime for ResNet-50 on ImageNet when reading training data over network.
needed to read from disk. This
Data echoing an effective alternative to optimizing the training pipeline or adding additional workers to perform upstream data processing, which may not always be possible or desirable. Although the exact speedup depends on the model architecture, dataset, batch size, and how well repeated data are shuffled, setting the echoing factor to the ratio of upsitramm-o-dowwnsteemm processing time maximizes the potential speedup and worked well in our experiments, even for large ratios. As improvements in specialized accelerators like GPUs and TPUs continue to outpace general purpose computation, we expect data echoing and similar strategies to become increasingly important parts of the neural network training toolkit.

---

Under review as a conference paper at ICLR 2020
REFERENCES
Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. Tensorflow: A system for large-scale machine learning. In /2th {USENIX) Symposium on Operating Systems Design and Implementation ({OSDI) 16), pp. 265-283, 2016.
Olivier Bousquet, Sylvain Gelly, Karol Kurach, Olivier Teytaud, and Damien Vincent. Critical hyper-parameters: No random, no cry. arXiv preprint urXiv:1706.0833 2017.
Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. One billion word benchmark for measuring progress in statistical language modeling. In Conference of the International Speech Communication Association, 2014.
Steven WD Chien, Stefano Markidis, Chaitanya Prasad Sishtla, Luis Santos, Pawel Herman, Sai Narasimhamurthy, and Erwin Laure. Characterizing deep-learning I/O workloads in TensorFlow. arXiv preprint arXiv: 1rXxv:8810003053 2018.
Matteo Fischetti, lacopo Mandatelli, and Domenico Salvagnin. Faster SGD training by minibatch persistency. arXiv preprint arXiv:1806.0 07353, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Elad Hoffer, Itay Hubara, and Daniel Soudry. Train longer, generalize better: closing the generaliza- tion gap in large batch training of neural networks. Advances in Neural Information Processing Systems, pp. 1731-1741, 2017.
Elad Hoffer, Tal Ben-Nun, Itay Hubara, Niv Giladi, Torsten Hoefler, and Daniel Soudry. Augment your batch: better training with larger batches. arXiv preprint urXiv:1901.000 2019.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. International Conference on Machine Learning, pp. 448-456, 2015.
Angelos Katharopoulos and François Fleuret. Not all samples are created equal: Deep learning with importance sampling. arXiv preprint arXiv: (003v:/80030044 2018.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Sameer Kumar, Dheeraj Sreedhar, Vaibhav Saxena, Yogish Sabharwal, and Ashish Verma. Efficient training of convolutional neural nets on large distributed systems. 2018 IEEE International Conference Cluster Computing (CLUSTER), pp. 392-401, 2018.
Tsung-Yr Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yar Fu, and Alexander C Berg. SSD: Single shot multibox detector. European conference on computer vision, pp. 21-37. Springer, 2016.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529, 2015.
Yurii Nesterov. A method for unconstrained convex minimization problem with the rate of conver- gence ((1/22). In Doklady AN USSR, volume 269, pp. 543-547, 1983.
Boris T Polyak. Some methods of speeding up the convergence iteration methods. USSR Computational Mathematics and Mathematical Physics, 4(5):1-17, 1964.
10

---

Under review as a conference paper at ICLR 2020
David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning representations by back-propagating errors. Nature, 323(6088):533, 1986.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. ImageNet large scale visual recognition challenge. International Journal of Computer Vision, 11533)211-225, 2015.
Christopher Shallue, Jaehoon Lee, Joe Antognini, Jascha Sohl-Dickstein, Roy Frostig, and George E Dahl. Measuring the effects of data parallelism on neural network training. arXiv preprint arXiv:1811.03600, 2018.
Iyy Sutskever, James Martens, George E. Dahl, and Geoffrey E. Hinton. On the importance of initialization and momentum in deep learning. In International Conference on Machine Learning, pp. 1139-11477 2013.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, pp.
2017.
Chris Ying, Sameer Kumar, Dehao Chen, Tao Wang, and Youlong Cheng. Image classification at supercomputer scale. arXiv preprint arXiv:1811.0663 2018.
Jian Zhang, Christopher De Sa, loannis Mitliagkas, and Christopher Ré. Parallel SGD: When does averaging help? In International Conference on Machine Learning Workshop on Optimization in Machine Learning, 2016.
Martin Zinkevich, Markus Weimer, Lihong Li, and Alex Smola. Parallelized stochastic gradient descent. In Advances in neural information processing systems, pp. 2595-2603 2010.

---

Under review as a conference paper at ICLR 2020
A
BEST TRAINING CURVES FOR ALL EXPERIMENTS
Validation Cross Entropy
5.5
5.5
5.0
5.0
4.5
4.0
4.0
30.0
3.5
0.5
1.0
1.5
2.0
Fresh Examples Read
x10'
Fresh Examples Read
x10
(a) Transformer on LMIB
(b) Transformer on Common Crawl
Validation Accuracy
0.92
0.7
0.90
0.86
0.5
0.84
0.82
0.80
0.3
0.5
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Fresh Examples Read
x10?
Fresh Examples Read
x10*
(c) ResNett33 CIFAR-10
ResNet-50 ImageNet
0.25
0.20
Baseline
Batch echoing
AP
0.15
Example echoing after augmentation (if any)
0.10
Example echoing
before augmentation
0.05
0.00
Fresh Examples Read
x10
(e) SSD COCO
Figure The best training curves for experiments in Section 3.1 Solid lines represent the mean over the best trial from independent hyperparameter searches, and shaded regions represent one standard deviation. The dashed line represents the target metric value.
12

---

Under review as a conference paper at ICLR 2020
0.8
Validation Accuracy
0.7
0.6
Baseline
0.5
Echoing factor 2
Echoing factor 3
0.4
Echoing factor e
Echoing factor
0.3
1.0
Fresh Examples Read
x10*
Figure 10: The best training curves for experiments in Section 3.2 Solid lines represent the mean over the best trial from 5 independent hyperparameter searches, and shaded regions represent one standard deviation. The dashed line represents the target metric value.
Validation Cross Entropy
Validation Cross Entropy
Baseline
Baseline
Echoing factore=2
Echoing factor =
Echaing factor c-4
5.0
Echoing factor e-4
Echoing factor
Echoing factor =
Echoing factor 16
Echoing factor = 16
4.5
4.0
0.0
0.5
1.0
1.5
2.0 x10
Fresh Examples Read
Fresh Examples Read
x10
(a) Batch size 1024
(b) Batch size 4096
Figure 11: The best training curves for experiments in Section 3.3 Solid lines represent the mean
over the best trial from 5 independent hyperparameter searches, and shaded regions represent one standard deviation. The dashed line represents the target metric value.
13

---

Under review as a conference paper at ICLR 2020
Validation Cross Entropy
5.5
Validation Cross Entropy
5.5
3.5
0.0
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.8
1.0
Fresh Examples Read
x10²
Fresh Examples Read
x10'
(a) Batch size 64
(b) Batch size 256
Validation Cross Entropy
5.0
Baseline
4.5
Batch echoing
Example echoing
4.0
3.5
Fresh Examples Read
x10
(c) Batch size 4096
Figure 12: The best training curves for experiments with Transformer on LMIB in Section 3.4 Solid lines represent the mean over the best trial from 5 independent hyperparameter searches, and shaded regions represent one standard deviation. The dashed line represents the target metric value.
14

---

Under review as a conference paper at ICLR 2020
0.8
Validation Accuracy
Validation Accuracy
0.7
0.7
0.6
0.5
0.4
0.4
0.3
0.3
Fresh Examples Read
x10
Fresh Examples Read
x10*
(a) Batch size 256
(b) Batch size 4096
0.8
Validation Accuracy
Baseline
Batch echoing
0.6
Example echoing after augmentation (if any)
Example echoing
before augmentation
0.4
0.0
0.5
1.0
1.5
Fresh Examples Read
x10
(c) Batch size 16384
The best training curves for experiments with ResNet-50 ImageNet in Section 3.4 Solid lines represent the mean over the best trial from independent hyperparameter searches, and shaded regions represent standard deviation. The dashed line represents the target metric value.
Figure 13:
Validation Cross Entropy
B Baseline
B ufffrr Size Buffer Size 10
5.0
B Buffer Size 10?
4.0
0.0
0.5
1.0
2.0 x10'
Fresh Examples Read
Solid lines represent the mean over the best trial from 5 independent hyperparameter searches, and shaded regions represent one standard deviation. The dashed line represents the target metric value.
Figure 14: The best training curves for experiments with batch echoing in Section 3.5
15

---

Under review as a conference paper at ICLR 2020
Validation Cross Entropy
5.5
Validation Cross Entropy
Baseline Example echoing
Baseline
Example echoing
5.0
5.0
4.0
4.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
Fresh Examples Read
x10?
Fresh Examples Read
x10'
(a) Shuffle buffer size 10
(b) Shuffle buffer size 107
Figure 15: The best training curves for experiments with example echoing in Section 3.5 Solid lines represent the mean over the best trial from 5 independent hyperparameter searches, and shaded regions represent one standard deviation. The dashed line represents the target metric value.
16


Reviews:
[{"review_id": "SkltZkCOtr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper discusses the use of data echoing (re-passing data fetched from drive or cloud) to maximize GPU usage and reduce reliance on data transportation time. The schemes basically are: reusing data at the example level, after data augmentation, or after batching. The experiments measure how much fresh data is needed in order to reach the same level of validation accuracy, with significant speedup when echoing is used. \n\nI thought the paper is very nicely motivated, although this is out of my area so I cannot comment on how thoroughly the problem of data fetching is investigated in other works. The evaluations are also nice, and appropriately uses a large model (Resnet 50) and dataset (Imagenet). \n\nThe simplicity of the method is a plus, but I question a fundamental part, especially if batch echoing is used--isn't this just the same as running SGD twice, and therefore halving the stepsize and doubling the number of steps? From the optimization viewpoint, it seems that if less data was used and a good validation error level is reached, then how do we not know that less data wouldn't work well in the first place? I understand that all step sizes and decay rates were chosen independently per experiment; can those numbers be shared in a way to see if this is happening or not? Figure 8 also suggests that though it may take a long time for the baseline to reach the same level as that with batch echoing, everyone reaches a pretty low error rate at about the same point, and the difference may be in the \"slow converging\" phase of the optimization; thus measuring how long it takes to reach a specific low error rate may be an exaggerated measure. \n\nBasically, what I am saying is that the idea is nice, but the results look a bit magical. I'm happy to increase my score if the authors can upload more intermediary results, like plots of form figure 8, decay rates and schedules, batch sizes, exact repetition schedules, etc. \n\nminor: page 3 end of paragraph 2: \"repeated data would NOT be more valuable than fresh data?\"\n\nAfter rebuttal: The authors have addressed my concerns and I more-or-less believe the results. I see why the contribution can be viewed as minor, but it is well-motivated and looks like a nice set of experiments. I encourage the authors to make their code available so that it can be easily incorporated in applications. ", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "S1eXhCVAKH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a method to compensate the high latency brought by data IO/processing in neural network training. Specifically the authors propose to repeat training on the same subset of data during waiting time for the new data.  In this way, the data efficiency is improved, as verified by thorough experiments on various of real-world tasks.\n\nAlthough the experiments look promising, I have to say the innovation of this paper is limited. The way of reusing the current data during waiting looks more like a straightforward trick, rather than a novel idea that deserved to be published at ICLR.  Furthermore, I’m wondering that how general the scenarios of “t_{downstream}>t_{upstream}” will be. Even in industrial level applications (e.g., billon level recommendation or click prediction task), AFAIK the training (including feedforward/backprop/communication in the distributed setting) consumes most of the time, while the data reading/preprocessing is comparatively cheap. Last but not least, what will the final performance will be given the potentially harmful consecutive reuse of data? Will it be worse than baseline?\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HklnaKqnYH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a simple method for avoiding bottlenecks during NN training, whereby training examples are utilized multiple times per read. The work focuses on cases where the cost of preparing a minibatch exceeds that of a training step (both a forward pass and, subsequently, a parameter update). Working within said regime, the authors investigate different strategies for 'echoing' examples.\n\n\nFeedback:\n  The proposed method itself is very simple: that's fine. While some cursory analysis of data echoing's theoretical implications would be appreciated, I am fine with practically motivated solutions that address real issues. Simple 'tricks' that are easy to implement and widely applicable are often useful tools. Especially when little theoretical analysis is provided, introducing such a trick requires strong empirical evidence to validate its efficacy. As it stands, failure to provide crucial information forces the reader to suspend disbelief when evaluating the proposed method's impact.\n\n  The authors seem to tiptoe around the issue of the relative cost of prefetching a batch versus that of a combined forward pass and parameter update. The work is predicated upon the assumption that the ratio of said costs $R > 1$, but the authors state that an unspecified subset of their experiments violate this assumption. What's more, real-world values of $R$ are not reported (or even measured!). As per Amdahl's law, $R$ upper bounds the potential benefits for data echoing; hence, failing to report $R$ is more than a little bit concerning. By the same token, the appropriate statistic for various result figures would seemingly be time rather than, e.g., the number of fresh examples read. As a reviewer, I would rather see evidence that data echoing provides modest benefits in realistic scenarios than x3.25 speedup in self-described \"contrived\" examples. Alternatively, consider providing real-world examples where $R > 1$ to help ground your arguments.\n\n\nQuestions:\n  - Why was extensive hyperparameter tuning necessary?\n  - Why are most results reported in terms of time/steps to achieve a target value? If nothing else, consider providing the corresponding learning curves (as in Figure 8) as an appendix (incl. means and standard errors).\n\n\nNitpicks, Spelling, & Grammar:\n  - Metaparameters -> hyperparameters\n  - Streamline list at end of introduction:\n    \"\"\"\n    In this paper, we demonstrate that data echoing:\n        1. reduces the...\n    \"\"\"", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a simple trick of taking multiple SGD steps on the same data to improve distributed processing of data and reclaim idle capacity. The underlying ideas seems interesting enough, but the reviewers had several concerns.

1. The method is a simple trick (R2). I don't think this is a good reason to reject the paper, as R3 also noted, so I think this is fine.
2. There are not clear application cases (R3). The authors have given a reasonable response to this, in indicating that this method is likely more useful for prototyping than for well-developed applications. This makes sense to me, but both R3 and I felt that this was insufficiently discussed in the paper, despite seeming quite important to arguing the main point.
3. The results look magical, or too good to be true without additional analysis (R1 and R3). This concerns me the most, and I'm not sure that this point has been addressed by the rebuttal. In addition, it seems that extensive hyperparameter tuning has been performed, which also somewhat goes against the idea that "this is good for prototyping". If it's good for prototyping, then ideally it should be a method where hyperparameter tuning is not very necessary.
4. The connections with theoretical understanding of SGD are not well elucidated (R1). I also agree this is a problem, but perhaps not a fatal one -- very often simple heuristics prove effective, and then are analyzed later in follow-up papers.

Honestly, this paper is somewhat borderline, but given the large number of good papers that have been submitted to ICLR this year, I'm recommending that this not be accepted at this time, but certainly hope that the authors continue to improve the paper towards a final publication at a different venue.

Decision: Reject