Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 1: Reject
review_content: This paper considers the problem of changing environments for LQR. The authors model this through the use of a decoder that maps an incoming context (C,D) to the LQR matrices (A,B). They provide an algorithm for this setting based on a UCB strategy, prove sample complexity and regret bounds, and experimental results.

Overall the paper was well written but I had several concerns.

1. The results of this paper were not contrasted with other papers in this area. For example, if C,D are constant, and \Theta_* is a Block diagonal matrix with A,B on the diagonal - then the contextual case reduces to the standard LQR problem. It's unclear how the results given compare to past results in this setting, for example those of Abbasi-Yadkori/Szepesvari 2011.

2. I did not fully understand the UCB nature of the algorithm. In each round \Theta^(k) (the least squares estimator) seems to be used to compute the optimal policy (line 10 of the algorithm) instead \tilde{\Theta}^(k)  -the optimistic estimate. The optimistic estimate is only used in line 16 - a randomized procedure that is unmotivated.

3. Building on (1), it is hard to understand the results as given since there are no lower bounds given nor is there a discussion of the problem dependent parameters that arise. For example, in Theorem 1, is dp^2 suspected to be tight? Since the number of parameters in \Theta^{\ast} is d(p+p'), perhaps this is off by a factor of p?

3. I struggled to understand the setup of the experiments - as described the algorithm given was not used at all, rather \Theta^(k) was approximated and beta^k was set to be a constant. This does not seem like a fair evaluation of the method. 

In summary, I would reject this submission unless the authors couch it better in past work, explain their results better, and improve the experiment setup.

Finally, a typo: I think the indexing variable in the equation on the top of page 4 is h' not h. 

review_title: Official Blind Review #4

final_score: rating: 6: Weak Accept
review_content: In order to generalize the RL agent to unseen environment, in this work the authors studied the theoretical learning problem of building a decoder on top of linear continuous control using linear quadratic regulator (LQR). They presented a simple, UCB-based algorithm that refines the estimates of the encoder while doing LQR and balances  the exploration-exploitation trade-off. In the online setting, the proposed algorithm has a O(\sqrt{T}) regret bound, where T is the number of environments the agent played. This also implies after certain exploration, the agent is able to transfer the learned knowledge to obtain a near-optimal policy to an unseen environment. To justify their theoretical bounds the authors also present experiments that demonstrate the effectiveness of the algorithm.


The work of designing decoder on top of RL/control in order to generalize to new, unseen environments is very interesting, and is pretty novel to my knowledge. The problem formulation of LQR is standard until the part where the authors introduced the output matrices (C,D), which extends the fully-observable case of LQR (that is based on state feedback only) to partially observable. Leveraging the theoretical analysis of LQR, the authors extended the analysis to the setting of output feedback with particular structures of decoder matrices (C,D) sampled from decoder \mu. The algorithm proposed is quite standard in the output-feedback LQR literature (in control or RL). But the work is still interesting because to my knowledge I am not aware of general theoretical analysis of this setting (while most analysis is based on the full state feedback).
I haven't checked the proofs very carefully in the appendix, but from the description in the main paper it seems the analysis of contextual transfer learning performance is sound, and under certain regularity assumptions the authors did provide a high-probability regret bound for this contextual transfer learning problem. It would be great if the experiments are more involved as they are a bit too simple at this point, (where the unseen environment is the change in the physical constants). I also have some difficulties understanding all the dots in figure 2. Perhaps the authors can simplify the number of trajectories plotted there to make the presentation clearer. Another comment is about the current title, currently by looking at it I have no idea that is about contextual transfer learning and LQR. It would be great if that can be more specific. 

review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: 
# Summary
- The paper proposes a UCB-inspired algorithm for a contextual LQR problem. The problem itself is introduced in this paper and is similar in spirit to CMDPs, with the difference that instead of learning a mapping from context to transition matrix, a mapping from context to matrices [A, B] figuring in the system dynamics of LQR is learned.
- The proposed algorithm is an online-learning algorithm shown to have sublinear regret in the number of experienced environments. A toy experiment with a 2D moving mass is presented to illustrate the theory.

# Decision
Although the problem setting is interesting and it is encouraging to have a guarantee, several important unclear points in the paper and a missing comparison to a straightforward baseline stop me from recommending it for publication in its present form. I detail my concerns below.

# Concerns
1) First, a conceptual question. I can see a straightforward algorithm that can learn the linear mapping \theta from context to [A, B] as follows.
        - In episode k, obtain trajectory (x_{1:H}, u_{1:H-1})
        - By least squares, find [A, B] from the obtained trajectory
        - Since context [C, D] is observed, find \theta : [C, D] -> [A, B] again by least squares
One can do this using data from K episodes if needed, one can sequentially update the controller for collecting data, etc.
=>  A comparison to such a basic approach should be definitely included in the paper, in my opinion.

2) The authors might argue that the algorithm suggested above has no guarantee. I would be curious to hear in this regard a comment on the practical implementation suggested in the paper. Namely, after deriving the bounds etc., the authors make further approximations and modifications in the practical algorithm. From my point of view, these modifications defeat the purpose of the bounds, because then only empirical evaluation can confirm that these approximations have not destroyed the analysis. Alternatively, one needs to incorporate the introduced approximation errors in the analysis. In more detail,
        - Eq. (9) is not solved exactly but by random sampling. In the 2D toy task, it may be OK, but in higher-dimensional spaces, a significant error can be introduced which is not accounted for.
        - More importantly, the UCB bound \beta in Eq. (11) is not used at all in the experiments.
            => To my understanding, it is the crucial point of UCB to use the UCB-bound. If it is not used, how should one judge the resulting algorithm?

3) This is a concern regarding clarity. I didn't get (i) if matrices [Q, R] are context-dependent or not and (ii) if the agent observes them or not. This is not clearly communicated in the text.
=> Clarify whether [Q, R] are context-dependent and observed.

# AFTER REBUTTAL
After authors' clarifications and improvements on the paper, I update my score to weak reject. The reason I am still against acceptance is the lack of stronger empirical evaluations. As R4 pointed out, some clarifications on the side of the algorithm are also required.

review_title: Official Blind Review #2

Meta-review: This work considers the popular LQR objective but with [A,B] unknown and dynamically changing. At each time a context [C,D] is observed and it is assumed there exist a linear map Theta from [C,D] to [A,B]. The particular problem statement is novel, but is heavily influenced by other MDP settings and the also follows very closely to previous works. The algorithm seems computationally intractable (a problem shared by previous work this work builds on) and so in experiments a gross approximation is used. 

Reviewers found the work very stylized and did not adequately review related work. For example, little attention is paid to switching linear systems and the recent LQR advances are relegated to a list of references with no discussion. The reviewers also questioned how the theory relates to the traditional setting of LQR regret, say, if [C,D] were identity at all times so that Theta = [A,B]. 

This paper received 3 reviews (a third was added late to the process) and my own opinion influenced the decision. While the problem statement is interesting, the work fails to put the paper in context with the existing work, and there are some questions of algorithm methods.  
Decision: Reject