Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposed a minimal design API (or mostly API?) of machine learning framework called Flashlight. The key argument of the paper is that Flashlight is modular and agile. The Flashlight captured the key aspects of machine learning frameworks: Tensor and Operation, Memory Management, and Distributed. There are some evaluations to argue the benefit of Flashlight.
main_review: I have to admit that this paper gives me mixed feelings. Machine learning compiler/framework is my major of study, and I understand how we often have the desire to redesign the framework. It is not only (in most of the cases) harder than anticipated, but also (in most cases) just create "another" framework that is not necessary better. It is also very hard to argue that a framework is better, since there isn't a good way to measure the "quality" of a framework.

Strengths:
1. Flashlight captured the "core" of the Machine Learning Frameworks: Tensor and Operation, Memory Management, and Distributed Computation. The core APIs are interesting and somewhat innovative. 


Weakness:
1. I have to admit that it is tremendously hard to argue (with evaluation) that one machine learning framework is better than another. In this paper, we see too many vague arguments such as "agile", "modular", and "nimble". It is unfortunately that such vague arguments are bad for a scientific paper. On the same cord, the evaluations in the paper doesn't really mean too much. For instance, Table 1 argues that Flashlight is small. Table 2 argues that Flashlight is quick to compile. Table 3 show model performance, but most models are similar (AlexNet is small and small models are easier to be faster in a simple framework). None of the evaluations support the argument of "agile", "modular", and "nimble".

2. The interesting parts (or, we could say the "pain points" of machine learning frameworks in general) are memory management and distributed computation. The paper highlighted the importance of them, but the contribution of Flashlight in these aspects seems minimal (just described the minimal APIs and connected to downstream implementations). I agree that other researchers might be able to take advantage of these APIs, but the contribution seems small.

How could the paper be improved?

Honest, I am not sure. To be fair, Pytorch paper was published years after it was popular. TensorFlow was published when it was already well-known. Recent, MLIR made a big noise in machine learning framework community, but still, the popularity of the tool is way more important that the publication of it.

Taking MLIR as a reference, I think the reason it is loved so much is that it facilities (greatly) the development of IRs and IR transformations. I think it will be helpful to show case how Flashlight facilities other researchers by the "agile and minimal" design. That would be a more convincing argument of the value of Flashlight.


summary_of_the_review: I empathy the difficult of arguing the "betterness" of a machine learning framework. However, I think the paper in general has not made a convincing argument or provide enough scientific contribution. So I recommend rejecting this paper. However, I am very interested to check out the open source implementation of Flashlight, and see how amazing it is.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a neural network training and inference framework aimed at framework researchers with a focus on modularity, simplicity of design, and extensability by researchers.
main_review: The authors have built what appears to be a well thought out, and cleanly implemented neural network framework with researcher modification and extensibility in mind. The paper is well written, with clear background sections on the current state of framework design, and the need for a more research-oriented extensible framework. Indeed a simplistic, easily modifiable/extensible framework that can be used to easily implement new research methods that might require custom forms of memory management, runtime, or tensor functions is immediately appealing.

While I commend the authors for what is no small undertaking, I have a few concerns:

1. The first and primary one is whether this work sufficiently distinguishes itself in a crowded field. Between the PyTorch C++ library, Tensorflow Lite, Tensorflow Lite Micro, etc., there are many options for implementing custom built solutions, some of which offer a similar level of control over memory management, and operator behavior as described in Flashlight (albeit somewhat less cleanly perhaps). Via the PyTorch C++ library, custom operators can be implemented, memory allocation behavior can be customized as desired, and hooking into the PyTorch ecosystem carries significant advantages. So the question I'm left asking myself is whether Flashlight provides any significant advantages over the PyTorch C++ library for the majority of use cases. Unless a researcher is implementing functionality that really does involve altering very large portions of framework internals, I'm not convinced that optimizations are worth giving up for the sake of readability.

That being said, the framework seems nicely designed, with clean interfaces and its use of ArrayFire for (nice) cross-platform tensor calculation, and a small binary size and compile time to boot which is great for development purposes (although again, extending PyTorch with a custom op does not require compiling the library from scratch). I could see using Flashlight in order to roll a very custom solution.

I am willing to believe that the code cleanliness, modularity and practicality of Flashlight could be beneficial to some researchers, but beyond that, I'm unsure as to what Flashlight brings to the table other than that it's more barebones and a good jumping off point for research than other frameworks. The methods it employs are (unless I'm mistaken) not exactly unique to the work, and I'm unsure if a conference paper is necessarily the right venue for this work. 

2. I assume the 9MB binary size stated in the paper does not include ArrayFire... Does the PyTorch 528mb include Aten? and similar for Tensorflow?

3. I would like to hear more about why Flashlight performs better than PyTorch or Tensorflow. Is this an Aten v.s. ArrayFire speedup?
summary_of_the_review: Overall the paper was interesting, and I commend the authors for their work. It seems the Flashlight framework could be beneficial for researchers looking for a boilerplate template framework on which to build their very highly customized solutions. However, for anything other than an solution requiring rewriting large portions of a neural network framework, the researchers using this library will give up optimizations and other benefits of using a more mature library such as PyTorch C++ (for which custom ops and some other features mentioned in this work are possible without rewriting internals).
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper describes the design philosophy and structure of the Flashlight deep learning framework. Flashlight is modular, small, and narrowly oriented toward systems researchers. Rather than (or in addition to) high level productivity, Flashlight focuses on internal and external simplicity of ML tools. The authors evaluate Flashlight by training several standard reference models and it achieves slightly better performance than PyTorch or TensorFlow

main_review: Flashlight’s narrow use-case, as a testbed for systems research in machine learning, is important and poorly supported by its major competitors. This comes at the cost of other use-cases like model design research (limited feature set, need to compile models in C++) or usage in production environments (avoidance of low-level optimizations, absence of model servers for deployment).

While Flashlight with its native support for efficient memory management, distributed training and end to end benchmarking, shows promise as a research tool. However, it has not demonstrated its capabilities in practice. Whereas a long list of well-established (reference) models in speech, vision, and text applications are provided, the only discussion of novel work enabled by Flashlight is a brief mention of wav2letter in Section 3. The performance study on well-established models gives confidence in the basic implementation of Flashlight, which is of limited use by itself. This would be a strong paper if it contained a case study showing how its design contributed to systems research.

summary_of_the_review: Flashlight is well-designed and shows promise as a research tool for systems research in machine learning. However, it has not demonstrated its usefulness in practice.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper presents Flashlight, a C++-based deep learning framework, based on the ArrayFire
tensor library, and implemented all the necessary functionalities in the core infrastructure,
including memory management, distributed training, data loader, etc. The major contribution includes:
- An end-to-end framework with solid engineering efforts, which comes with highly optimized models
including speech, vision and text;
- The system is modularized and customizable with simple C++ APIs, and each component is equipped
with highly optimized default implementation;
- The system is benchmarked with a variety of the state-of-the-art models, and supports end-to-end
packaging for these applications.

main_review: 
**Strength of the work.**
- The paper provides performance comparison for end-to-end training performance on 6 representative
workloads, and the proposed system outperforms PyTorch and TensorFlow in most of the cases;
- The proposed system allows customized memory manager, which is a missing piece in many of the existing
deep learning frameworks;
- The system is ready to use in production, which makes it outstanding given the robustness requirement
in the production environment.

**Weakness.**
- Ablation study is missing from the paper. For example, training ResNet-50 on a single GPU
has become a well-studied and highly-optimized task. However, the system is still able to outperform
existing large systems like TensorFlow. Therefore, it would be desirable to showcase why the system
is able to do better. Is it because Flashlight does better latency hiding in data loading, or the
low-overhead nature of C++ APIs, or any other reasons?
- More direct and fair comparison might be required for the reviewer to better understand
in Table 1 and 2. According to Table 1, FlashLight has far fewer operators than PyTorch (110 vs 2166),
so it seems nature that it compiles faster than PyTorch (27min vs 754min), and generates smaller binaries (9MB vs 527MB).
- Claims need more justification with data and comparison with other frameworks. For example, in Section 3,
it's claimed that the proposed system is research-first. It would be desirable to quantifiably understand the reason
behind the claim. Compared with TensorFlow where a lot of research have been done on for distributed training,
why the proposed system is more suitable?

**Correctness.** To the best of the reviewer's knowledge, there is no correctness issue.

**Clarity.** The paper is not hard to read, but might need more data to justify many of it claims.


summary_of_the_review: In summary, the paper presents a solid deep learning framework written in C++, and has been put in production for a wide range of applications. However, in terms of paper writing, it would be more desirable for the readers and reviewers to understand the novelty and quantifiably justify the claims in the paper.
review_title: null

Meta-review: This paper describes Flashlight, a tool for ML researchers with specific design considerations for conducting systems research. The needs for such tool are significant, and recent advances in this topic have been relatively slow, so this research is timely and important. 

Reviewers are positive about the importance of the problem and the nice design of Flashlight. It seems the tool has been used by researchers with positive feedback. At the time of the original submission, reviewers expressed some concerns about the novelty and the weak arguments for convincingly showing the advantages over other similar tools. 

Authors provided nice replies including specific case studies, but with the short time period to reassess the proposed changes and additions, some reviewers remain hesitant, and thus this paper cannot be accepted at this time. I strongly encourage the authors to incorporate all of the proposed revisions and resubmit to a future venue.
Decision: Reject