Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: To efficiently perform multi-task training jobs on overlapped datasets, the authors propose a new data loading method, focusing on data preparation. They design a dependent sampling algorithm(DSA) and domain-specific cache policy to maximize the locality. Moreover, a novel tree data structure is constructed to efficiently implement DSA. The experiments show a greater versatility and superiority of training speed improvement without affecting accuracy.

strengths_and_weaknesses: Strength:
* Propose a new data loading method for training multiple parallel jobs on overlapped datasets 
* Propose a dependent sampling algorithm (DSA) to maximize the sampling locality while ensuring correlated randomness 
* Design a tree-based data structure for efficient cache policy implementation 
* The experiment results are convincing. Joader reduces end2end multitask training time while keeping CPU utilization low.

Weakness:
* To maximize the sampling locality, the authors need extra operations, including dividing the datasets into subsets and calculating the probabilities and conditional probabilities. With the number of training jobs increasing, the overhead could be a problem and they didn't mention it. 
* The motivation is weak. It's a rare case that researchers will train multiple tasks on overlapped datasets on the same machine.
questions: What technologies do they use to make CoorDL achieve better performance than Joader in synchronous cases?
limitations: See above. 

Overall, the being solved problem is a significant problem. The approach used in this work looks new. However, the experimentation should be improved. RefCnt Cache policy looks trivial.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors revealed an interesting and practical problem in training parallel DNNs. The redundant consumption of I/O and data preparation, which seems not as crucial as computation in GPU, affects the DNN training speed, especially when multiple DNNs are training parallelly in a GPU server. To solve this problem, the authors proposed a new data loading method for efficiently training parallel DNNs. The method includes a data sampling algorithm to increase the sampling locality with guaranteed randomness. For practical usage, a data structure-dependent sampling tree and a specific cache. Altogether, a prototype Joader (integrated with PyTorch) is implemented to show the extremely fast performance in training DNNs parallelly.  
strengths_and_weaknesses: Strengths:
1.	The problem is practical and essential to society. The novelty of the method is clear enough. The method keeps the characteristic of random sampling for each DNN training job, and reduces the redundant consumption of I/O and data preparation.
2.	The theoretical proof guarantees the correctness of the algorithm, and the algorithm is proved to reach global optimal in the two-job case.
3.	The authors implemented a prototype Joader, which is integrated with PyTorch. The details of the system design and implementation are provided in Appendix.
4.	The evaluation results clearly show the superiority of training speed improvement from implemented Joader.


Weaknesses
1.	The description of the N-job case is a little bit complicated. After carefully reading the operations of the dependent sampling tree and the pseudo-code of DSA, I could finally understand the N-job case.
2.	The authors proved the optimality in the two-job case, which is very important for this problem theoretically. The algorithm in the N-job case seems to be a greedy algorithm. It is better that the authors could give proof of the optimality.

questions: 1.	 If I understand clearly, the proposed method totally controls to sample the same data when the datasets are the same. Compared to the classic sampling in random, is there a tradeoff between strict correlation and randomness? 
2.	Is it possible to give the proof to the optimality in N-job case?

limitations: N.A.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper presents Joader, a data loading system optimized for the scenario where multiple training jobs share overlapped sources of data and data preprocessing. The proposed system overcomes the constraints where training jobs could vary in training speed which causes cache thrashing.

strengths_and_weaknesses: Strength
1) The proposed idea, dependent sampling algorithm (DSA), is novel and effective with good theoretical guarantee, while providing flexibility in varying speed across training jobs, free starting/stopping of jobs, partitial overlapping;
2) The engineering contribution that integrates Joader into PyTorch and enables distributed training is highly convenient for downstream users of this research;
3) Preliminary experiments demonstrate that this approach is effective.

Weaknesses:
1) The evaluation is only conducted on a single workload or a single series of workloads (ResNet).
2) Distributed training does not exist in experiment setting, while it is indeed an extremely important usecase considering multiple overlapped training jobs in a cluster. Appendix D.4 does briefly mention that there is no much change to make Joader work for distributed training, but no further experiments are conducted.

questions: Is there any specific reason that Joader only uses ResNet series in experiments? While image classification is a quite standard task to test against, it would be more convincing to justify with data that it works with diverse set of models.

limitations: While the proposed system is general and less restrictive comparing with previous work, the experiments are set to specifically justify the claims in previous sections, the reviewer believes that it would potentially attract more users if there is at least one experiment for Joader to focus on distributed training setting.
review_title: null

Meta-review: metareview: The paper proposes a new data loader called Joader for parallel DNN training on overlapped datasets that allows tasks to share memory and computational resources for data preprocessing. Joader implements a new sampling mechanism and cache policy to reduce cache misses due to data access from multiple tasks and a new data structure to facilitate the implementation. Joader has been integrated with PyTorch and shown to be very effective in practice.

All reviewers agree that the paper makes a valuable contribution to the NeurIPS community and I agree with Reviewer 6QhD that the contribution stands even if the system is not covering distributed workloads. I thus recommend acceptance. 

However, for the camera ready version I think it is important that the authors incorporate additional discussion about potential limitations of their approach so that the tool can be used most effectively by researchers. For example, potential overheads in the implementation of the new data structure should be discussed even if they are small in the provided experiments and also the fact that sampling across jobs is now correlated and no longer independent is important to emphasize together with potential implications for the resulting workloads (e.g. can the results of the parallel runs still be used to achieve variance reduction by ensembles?). Beyond that I think the additional experiments to break down the performance gains into individual components is valuable and clarification made in the conversation with the reviewers should be incorporated in the paper.
confidence: Certain
Decision: Accept