Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposed to parallelize the inference and training of GRU networks (a type of recurrent neural networks) at the `time` dimension. 
The main contribution is the application of multigrid reduction in time (MGRIT) solver, and a new GRU architecture (Implicit GRU) that handles the stiffness in the architecture. As a result, the evaluation shows 6.5 times faster training time.
main_review: First of all, I have to admit that I am not very familiar with the multigrid reduction in time solver technique and its application to ordinary differential equations. My understanding of this paper is thus not very deep.

Strengths:
1. The paper provides a new view to parallelize the training task of machine learning models. Previously, progresses in parallelization techniques have lend great power to the efficiency of parallel training, and helped reduce idle time when many devices are used to train large models with large amount of training data. Each new idea in this domain is a great step forward.
2. The paper is based on solid techniques (multigrid reduction in time solver) that have been used in other domains (solving ordinary differential equations). Pivoting a proven technique to new applications always lends more credibility to the proposed solution.


Weakness:
1. Since the parallelization technique in this paper gives best performance for recurrent neural networks with long sequences, the impact of this paper might not be that wide. 
2. The GRU architecture has to be adapted to achieve the result mentioned in the evaluation. This is a bit limiting because machine learning professional might have their reason to use a specific recurrent architecture, or might want 

Questions:
Since I am not very familiar with the details of the technique, I would rather use this reviewing opportunity to ask a few questions.
1. If my understanding is not way-off, the technique trades off computation for parallelization. In other words, more computation is needed for the same number of iterations of training, so that the GRU can be parallelized at the `time` dimension. If this understanding is correct, I am curious where exactly the speed up (6.5 times) is from. Is it because we can utilize the multiple devices more efficiently (there is less idle time on the devices)? Could I say that for the same number of iterations of training, more FLOPs are actually needed, but less time is taken since the devices are running in parallel? If the same number of iterations of training is ran, should we expect the technique to have the same degree of model accuracy?

2. Let's assume that the model is as simple as a 2 layer GRU. If we unroll the model, it just looks like a long repetitive model with lots of GEMMs and activations. There are still data dependencies between the repetitive units, just like the GRU data dependencies. For the unrolled model, there are published parallelization techniques (such as GPipe and PipeDream). I think GPipe might require the batch size to be large enough for good parallelization factors, but PipeDream seems to be very good even if the batch size is not big. PipeDream also has certain inaccuracy, in the sense that the gradients are always stale. I am just wondering how do the authors compare this technique with PipeDream?

Nits:
1. in page 2, the second usage of ODE has explanation "ordinary differential equation", but it should be explained in the first use (end of line 2)
2. in page 4 (around the middle) there is a "the the hidden state".
3. in page 6 section 3.4, there is a "fewer then would be required" but I think it should be "fewer than would be required"

 
summary_of_the_review: I think the paper provides a very interesting new approach of `time-parallelization`, which can benefit the machine learning community a lot. Thus I think this paper can be accepted.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper aims to address the limitations of existing approaches for training Gated Recurrent Unit (GRU) given long sequence in terms of both training time and model accuracy. To tackle this challenge, author propose a novel parallel training scheme (called parallel-in-time) for GRU based on a multigrid reduction in time (MGRIT) solver.  Specifically, the key to achieving speedup is a hierarchical correction of the hidden state to accelerate end-to-end communication in both the forward and backward propagation. Authors gives experimental results on two public datasets to demonstrate the performance improvement in the long sequence scenario. 
main_review: Strengths:
In this paper, authors propose a novel parallel-in-time (PinT) training method based on a MGRIT solver for GRU networks with long sequences. This method has been well motivated by illustrating the limitations and unsolved problems of the existing methods.
The proposed algorithm enables the accelerated parallel training of GRUs on long sequence, which is a unique capability that permits growth in this dimension.
Experimental results provided on the HMDB51 dataset is impressive, which demonstrate the proposed parallel training scheme of GRU achieves up to 6.5× speedup over a serial approach. 
Overall, the paper is well written and clear, especially the part of methodology introduction.

Weaknesses:
Can you give a discussion about the different performance of proposed method in parallel inference versus serial inference?
There is a confusion that regarding the test accuracy, the experiment setting for UCI-HAR and HMDB51 dataset is a bit different. Can you show the results/plot in the same manner or provide an explanation to it?
Can you provide the test accuracy of baseline in Figure 9 as well? Otherwise, the accuracy performance of MGRIT for full HMDB51 is unsure, which is necessary to demonstrate whether the remarkable speedup on HMDB51 dataset is meaningful.
The experiment results are not clear to demonstrate the statement in Remark 3, “Our results below indicate consistency between serial and parallel inference even when training is done with MGRIT. ” Consequently, confusion is introduced for the statement as follows, “This suggests that error introduced by MGRIT is not a severe limitation.” A more convincing illustration is expected. 

Comments:
Grammar errors(underline): Sec. 4.1 - “When training with MGRIT three levels are with a coarsening rate of 4 are used for the length 128 sequences. ”
In Sec. 4.1 - “Here we see that the the test accuracy using parallel inference (red line) is modestly lower then purely serial case. ”, “then” -- > “than” 
Grammar errors(underline): Sec. 4 – “In this section, we evaluate the performance our new parallel-in-time training approach on the sequence classification tasks such as human activity recognition and video classification. ”
In Conclusion – using “6.5\times” would be better than “6.5x(letter x)”
summary_of_the_review: N/A
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper describes a technique for evaluating GRU networks based on the multigrid reduction in time (MGRIT) technique. These techniques are not new, in general or to neural network training, but the contribution here is their application to GRU layers. After presenting some of the theory behind ordinary differential equation (ODE) representations of the GRU and laying out the mathematical framework for the MGRIT method, the method is evaluated on two datasets. The results show impressive scalability up to 32 processes (CPU-only) at the cost of a slight loss of accuracy compared to the traditional, sequential GRU
main_review: The paper describes an iterative technique for evaluating GRU networks based on the multigrid reduction in time (MGRIT) technique. Key idea is partitioning of a long sequence into shorter sub sequences. These subsequences are trained in parallel, error corrections between grids are applied to correct for inexact computation. 

The paper is well-organized, and the mathematical descriptions of the method are clear. The application of the method to GRU layers is novel and will likely be a useful strategy for overcoming the inherently sequential nature of the GRU. The authors describe the tradeoffs in play here well: more resources will be effectively utilized (to a limit noted in the discussion) in applying a fast and scalable, though inexact, method at the cost of a small amount of accuracy in the trained model relative to exact evaluation of the GRU. Obviously, a GPU implementation of this technique would be interesting as well, and the authors note that this is in development. Even without it, though, the results are compelling enough.

Related work is incomplete, recent work in sequence modeling are particularly missing.

More details on “Training with MGRIT” (section 3.4) are needed. In the absence of a reproducible (anonymized) source code, pseudo code or code snippet would provide more clarity (sufficient as supplementary materials). 

More could be done in experimental evaluations e.g., quantify impacts of coarsening factor on model quality.

There were some minor grammatical issues; careful copy-editing is recommended. Figure 6 is missing legend.
summary_of_the_review: A paper describing a novel parallelization strategy for GRU networks that adds dramatic scalability with some minimal loss in accuracy.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The main focus/goal of the submitted paper is the parallelization of the Gated Recurrent Unit (GRU) (the authors focus on classification problems). The authors describe the incorporation of a multigrid reduction in time (MGRIT) solver to speed-up and better parallelize the 
application of forward and back propagation of information. The proposed technique seems to provide a speedup of about an order of 
magnitude (at most) when implemented on distributed/shared memory hybrid computing environments.
main_review: Major comments:

-) The written English can be improved. 

-) The paper is hard to follow as the authors assume that readers are familiar with MGRIT. I understand that this is a technical paper which is geared towards researchers that are seasoned in the topic, and there is also a page limitation. Nonetheless, the authors should put every possible effort to include details, even in the form of an appendix.

-) Unfortunately, it is not clear at all what factors affect parallelism and scalability. 

-) Restricting all MPI processes within the same node does not really lead to interesting results. Large-scale networks will need way more resources than a 2-socket Intel node. In fact, I am not sure I understand why there is no benefit beyond 36 MPI processes, at least for the larger examples.

-) MPI is undefined -- please do so and add a proper citation. The appendix should provide guidance as to how the experiments are organized and might be replicated. I understand that this might be a considerable effort but any additional information will be particularly 
helpful. 

-) The main drawback of this paper is novelty. There is already some work done in the field (the authors cite it properly) and it is not 
clear to me what is the main differentiation factor. MGRIT is discussed for example in the paper by Kirby et al. (2020) and I am not 
sure that simply focusing on GRUs (together with the discussion in Section 3.3) warrants a publication at ICLR. 

-) The main positive part of this paper is the speedup and engineering of making MGRIT work for GRUs. Again, I do not think this is a 
major advancement in the field, especially for such a small number of MPI processes.  The authors claim that a GPU version is under 
development. This would be a very nice addition -- note though that the related paper cited by the authors uses MPI+GPU with experiments on up to several tens of GPUs.

-) Are hyper-parameters used? I did not see any detailed study. How the algorithm performs for different combinations of T and c_f? 
Maybe the authors include some and I missed them.

-) Figures 5, 8, and 9 are too small. Figures 8 and 9 could be combined together.


Minor comments:
Section 4.1: 

-)"When training with MGRIT three levels are with a coarsening rate of 4 are used for the length 128 sequences." Please rephrase.
-)"Here we see that the the test accuracy." Remove "the".
summary_of_the_review: I find the idea to apply MGRIT to GRUs interesting, however I am not sure that there is sufficient differentiation (and major advancement) between the methodology discussed in this paper and previous work cited by the authors (I am aware of some of this paper by my own reading). This is further pronounced by the fact that previous work also considers larger hardware installations of MGRIT; the amount of 
parallel resources considered by the authors does not allow for a trustworthy scaling analysis as inter-node communication is not currently present.
review_title: null

Meta-review: This paper presents a way of using multigrid techniques to parallelize GRU networks across the time dimension. Reviewers are uniformly in favor of accepting the paper.  The main strength is that the paper provides a new perspective on dealing with long input sequences by parallelizing RNNs across time. The main weaknesses are around the experiments: only CPU experiments are run, and sequences are not very long (max 128 length). All-in-all, though, it provides an interesting perspective that should be valuable to the community.
Decision: Accept (Poster)