Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This work proposed an approach of online learning and control of nonlinear systems relying on the insights from Koopman operator theory, and included empirical examples.

main_review: The importance of online nonlinear control problem is clear and should be studied.
However, for this work, the reviewer has several major concerns.

Major concerns
1) The overall approach is very similar to embed to control [https://arxiv.org/pdf/1506.07365.pdf].
   Embed to control primarily considers locally linear model over latent space, but it also mentions that it can be easily applied to globally linear cases.
   It tackles the problem of controlling agents from raw image inputs and studies the empirical results thoroughly.
   Therefore, except for the fact that the current work explicitly says it is doing online updates, the reviewer hardly sees any difference from the line of work mentioned above.  (see [https://arxiv.org/pdf/1910.08264.pdf] as well)
   
2) For online update aspects; the difficulty of online learning for control problems basically stems from the difficulty of exploration.
   Without exploration, for example, [https://arxiv.org/pdf/1703.04680.pdf] this work studies convergence of EDMD under certain conditions, and it is straightforward to use this idea to study the trade-off between the amount of data and approximation accuracy, which can then be used for robust control for example.
   Also, [https://arxiv.org/pdf/1906.05194.pdf] this work already studies active learning of Koopman operator for optimal control problem under certain conditions, which is basically about the online learning.
   If the current work considers so carefully about the online learning problem either experimentally or theoretically that it gives us more insights than the line of work mentioned above, those should be clearly stated (the reviewer thinks it is not clarified in the current form)
   
3) The author(s) claim that this work is for controlling complex dynamical systems; however, as long as the underlying dynamics is pendulum, it would be hard to justify this claim.
   Also, it is not always possible to write a nonlinear controlled system by  z_t+1 = Az_t + Bu+t even if the space of z is carefully selected.  (B should be dependent on x in general)
   The advantage of this linear model is the ease of computing optimal control using Riccati equation; however, if this transformation cannot be applied to general nonlinear systems, it is hard to say that this formulation can be applied to complex dynamical systems.  Then, what are the advantages of this approach over existing model-based RL (which includes online algorithm as well)?
   
Here are some minor comments
1) line 251;  what is "??" here?  there are some minor typos, please check them again.

-------------
review updated after the initial response
limitations_and_societal_impact: Limitations could be discussed more (for example, the concerns the reviewer mentioned above)
Social impact is discussed well.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper, inspired by (approximations of) Koopman operator theory, proposes an adaptive algorithm that learns a (controlled) dynamics model online. The learned model consists of an encoder and estimated (and continuously updated) A, B matrices. The method is shown to work well (with low RMSE and good control performance) on several low-level control tasks with images as the observations (hence the encoder).
main_review: I think that this paper is interested and could be accepted for NeurIPS, however there are some improvements that could be done:

* Discussion of algorithmic limitations. Experiments could be extended to higher DOF control systems to address this issue. Failures/problems with auto-encoder should be mentioned, how/when do these effect the introduced approach? In 3.4 how would the approach work if one were learning and controlling at the same time? How would some of the hyperparameters influence the results if not specified well? e.g. h, \rho, 

* How would the approach work if B(x) also depended on x, with x the image or if B(z), if z is the latent variable? In control theory, the control affine dynamics x_{t+1} = A(x) x_{t} + B(x) u_{t} is very popular.

* How does minimizing (12) work together with the least squares estimation for A, B? Maybe including algorithm (with the full loop) could clarify this.

* Koopman operator theory seems almost tangential to the introduced approach. How does this theory influence/guide the introduced approximate approach? When could the approximations be problematic, does the theory offer any insights?

* Comparisons: are there other control/model learning papers (also those that don't use autoencoders?) that work on images? Could these be included in the comparisons?

Minor comments:

- The title is too general, please change to restrict to the specific method that is introduced (adaptively learning latent linear models from images)

- There are a few mistakes, some hindering the readability, please go over the manuscript carefully.

- Captions are not adequate, the figures need to be self-explanatory. Check also line 251 for missing figure.

===== UPDATE =====

After reading the rebuttal and re-reading the paper, I still think that the paper can be accepted. The authors have replied well to most points raised also. However I cannot raise my score more because one of my main concerns, that of the Koopman operator theory not guiding the introduced methodology, was not answered sufficiently. For future work (regardless whether the paper is accepted or not), I would advise the authors to look at when or if Koopman operator theory could guide the design of approximate frameworks. When could features be sufficient for a control task? Can we know if by constraining the features and the dynamics model within some parametric form + dimensionality, we are not sacrificing much (i.e. staying 'close' to the operator P in some metric?)
limitations_and_societal_impact: N/A
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper discusses the problem of controlling a dynamic system based on raw sensory inputs. Their method uses an encoder/decoder to map the sensor information (video frames in this paper) to a learned embedded space. Next, it proposes to build a linear function that evolves the dynamic system’s states in the embedded space. This paper evaluates the performance of the proposed pipeline on three classic control problems: cart-pole, single pendulum, and double pendulum.
main_review: While the proposed pipeline seems reasonable, the method is evaluated on three classical control problems only. In my opinion, a NeurIPS paper needs more competitive examples.
- It is a bit too much to call cart-pole/(double) pendulum “complex dynamical systems” in the title/abstract/introduction. Please consider toning down the writing in title/abstract/introduction.
- The discussion on Eqn. (3) and (5) would be clearer if some context about the dimension of n and m could be given. As far as I understand, Eqn. (3) and (5) are two quite different optimization problems that should be used in different scenarios: if AZ_1 = Z_2 is an under-determined system (e.g., n >> m), then Eqn. (5) is preferred. On the other hand, if AZ_1 = Z_2 is a overdetermined system (n << m), one should stick to Eqn. (3) because the equality condition AZ_1 = Z_2 in Eqn. (5) may not be feasible at all. Which case is this paper faced with?
- From what I understand, all three examples (cart-pole, pendulum, and double pendulum) deal with 2D dynamics only because their motions are constrained on 2D planes, although they can be rendered as if they are 3D scenes. I would like to see a 3D example (e.g., quadrotor, ant, cheetah, or humanoid robots) in simulation, and more importantly, I am curious to see if the pipeline from video frames to embedded space still works for a 3D task. I would guess this would create some additional challenges due to the ambiguities in understanding 3D scenes from 2D images.

limitations_and_societal_impact: This paper did a good job of discussing its limitations in its low-dimensional, simulation-only experiments. I agree with these limitations, but instead of leaving them as future work, I would suggest trying to solve them in this submission so that it could be a competitive paper.

I do not see major societal and ethical issues from the proposed method, and I appreciate that the authors explicitly discuss them in the paper.
review_title: null

Meta-review: From the SAC. This is an instance where the rebuttal and the discussion worked. While the original decision for this paper was to not accept, it is being raised to a recommended accept. The primary reason is the quality of the rebuttal, and the useful technical discussion between authors and reviewers that ensued that seems to have been revealing. To the authors: I trust that you will take all reviewer feedback into account and most importantly, that all of the things in your rebuttal and discussion that were promised will be done in the next version of the paper. 
Decision: Accept (Poster)