Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: The authors present a method for recovering cost functions from observed state trajectories through inverse optimal control with signal-dependent noise for linear systems and quadratic costs. 
They formulate a POMDP and distinguish between the agent’s and the experimenter's inference problems since the agent does not have access to the experimenter's hidden states.
They derive the joint dynamics of the state and estimate by stacking the individual transition functions.
The resulting probabilistic belief transition function for the linear-quadratic Gaussian problem with signal-dependent noise is approximated through moment matching since it involves a product of two Gaussians (which does not result in a Gaussian).
In a similar fashion to stacking the agent and experimenter dynamics, the authors suggest additionally stacking the observer dynamics and solving the transition jointly for the case that they don’t have access to the full state.
The approach is validated on synthetic and experimental data and applied to human sequential sensorimotor behavior.

main_review: Strengths:
- The problem is interesting and the solution is appropriate for a limited number of use cases.
- A concise background and description of LQG. Although this is nearly 1:1 from [1] I believe it is necessary to follow the notation and problem setting.
- Visualizations are intuitive and easy to understand
- Focus on methods that increase interpretability of behavior
- Good discussion of limitations in the last section

Weaknesses:
- It would have been good to more clearly state which matrices are given and what quantities are inferred.
Introducing an algorithm before section 3.1 would be helpful to make this more clear.
- Similarly, it would have been good to detail more closely how K_t and L_t are iteratively determined.
- I believe a more complete discussion about the impact of the moment matching assumption is needed. This could be done either theoretically or empirically through Monte-Carlo simulation and discussion of an intuitive example.
- The related work section does a good job referring to previous work, but overall I feel like it is too dated. There have been many more recent works in this domain over the last 5 years.
- “While more recent and more general methods for optimal control in high-dimensional continuous domains exist, they rely on function approximation methods, which is useful in engineering applications but may not provide a computational level explanation of behavior” I feel that this argument needs to be substantiated since similar approaches in the continuous inverse optimal control setting exist which could potentially be adapted to the signal-dependent noise setting [2]. These do not necessarily rely on NN function approximation, allow for interpretable non-linear features, and are computationally tractable.
- Overall the applicability of the approach is very limited. While the authors hint at a more capable integration with iLQG, I had hoped to see this in this paper already.
- The evaluation is nicely visualized but more than 1-dimensional experiments are needed. While the authors state that the approach does not scale to higher dimensions, I would have liked to see a dimensionality of at least 3.
- The evaluation lacks comparison to any baselines. No ablations are done. Additionally to related work, it would be good to ablate how well the costs be inferred through more primitive versions of the algorithm (e.g. no noise assumption, no signal-dependent noise, etc.)

Overall, I believe the work is interesting and timely but needs improvements, focusing on the evaluation. I would also like to encourage the authors to increase the scope and applicability of the work.

Typos:
“an which in general is intractable”
“requires a progressively larger number function evaluations”

[1] Todorov, Emanuel. "Stochastic optimal control and estimation methods adapted to the noise characteristics of the sensorimotor system." Neural computation 17.5 (2005): 1084-1108.

[2] Levine, Sergey, and Vladlen Koltun. "Continuous inverse optimal control with locally optimal examples." arXiv preprint arXiv:1206.4617 (2012).

EDIT: I thank the authors for addressing my concerns with respect to the impact of the moment matching approximation and incorporating two baselines. Nonetheless, I believe that using a single experiment is not sufficient for this conference. Due to the limited experiments and limited applicability of the method (without a simple iLQG integration, which should not be a huge effort), I believe this paper needs additional work and would like to keep the current rating as is.
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper presents a theoretical framework for cost function estimation for linear, quadratic, Gaussian systems with action- and state-dependent noises. 
Algorithms for estimating the probabilities of latent variables and observed signals are derived for both the cases of complete and partial observation of the task variables by the experimenter. Then a gradient-free optimizer BOYAQA is used to obtain the maximum likelihood estimate of the parameters of the objective function.
The performance was examined with 1D reaching simulation and real arm reaching and saccadic eye movement experimental data.

main_review: Originality:
The method is straight forward applications of dynamic Bayesian inference and maximum likelihood estimate, but the formulation of inference by the agent and by the experimenter as graphical models as presented in Figure 1 is eye-opening.

Quality:
The mathematical derivation is solid and clear.
In the 1D reaching simulation, why isn't the end-point position error included in the cost function? Is it supposed to be zero cost as long as the end point is within a target range?
The case of center-out arm reaching clearly violates the linear assumption. The method seems to be working nevertheless, and the reason why would be worth discussing.

Clarity:
Probably not all the readers are familiar with the impact of Harris & Wolpert paper, it may worth more intuitive explanation.
If I understand correctly, the method assumes that the experimenter knows the dynamics model (A, B, V, Ci, H, W, Di). It is worth mentioning how they can be estimated

Significance:
Although the theory was derived under linear, quadratic, Gaussian assumptions, its applicability with real arm and eye movement data is encouraging.

limitations_and_societal_impact: These are well considered and documented.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper presents an algorithm to infer cost parameters of LQG problems given trajectories. The effectiveness of the algorithm is demonstrated on simulated arm and eye movements as well as random control problems.
main_review: Pro:

1. The proposed method is demonstrated on a wide range of tasks, including real world dataset of a rhesus monkey reaching for objects.

2. The proposed method is able to infer the parameters under partially observed trajectories.

3. Latent velocity and acceleration can also be estimated using the proposed approach.

Con:

1. Only linear dynamics and quadratic const are considered. Although the limitation is discussed.

Questions:

1. It is not clear to me if the dynamics is assumed to be known (i.e, A, B matrix). If it is known, then how is the dynamics for the monkey estimated?

2. How important is the assumption that A, B is constant? Can we work with a time varying A, B in the current framework. This will be important to extend this to iLQG settings.

3. An important application of inferring the cost parameters is using it to predict behaviors that is not in the dataset or generate controllers that can mimic the behaviors of the agent. Any thoughts on this?
limitations_and_societal_impact: They are well discussed.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper formulates an inverse optimal control solution for a control system with partial observability and noise properties characteristic of the human sensorimotor system - that is, control signal-dependent motor noise, and state-dependent observation noise. The contribution is that this formulation can enable an experimenter to estimate parameters of the cost function of a control system (e.g. a human’s motor control system) under assumptions that that control system acts optimally, has full knowledge of the system dynamics (an accurate internal model), and has a functional form similar to that of the proposed formulation (a variant of the LQG case). Through a series of experiments comparing their inverse model to simulated reaching data, real macaque reaching data, simulated saccadic eye movements, and simple non-biological control problems, the authors show that their method can accurately estimate unknown cost function parameters to reverse-engineer an approximate model of the control system in question.
main_review: The paper takes a previous formulation by Todorov (2007) of the control problem faced by an agent with noise properties similar to that of the human sensorimotor system (and who acts optimally under full knowledge of its own dynamical system), and instead considers the inverted problem of how to discover the cost function parameters of this agent. The authors use probabilistic methods to derive this formulation and provide an algorithm for computing the approximate likelihood of the control parameters. The main contribution is thus to combine inverse optimal control in the LQG setting with the signal-dependent sensory and motor noise modelled by Todorov.
The authors validate that their method works in a series of well conceived experiments using primate reaching and saccade movement simulators, real primate psychophysical reaching data (previously published) as well as a classic control problem. The method does seem to work well, based on the data presented.
The surrounding literature seems well reviewed, and I agree that this method could in theory be applied to estimating control properties in psychophysical motor experiments, if of course, the objective is still to estimate what trade-offs a human or non-human primate is making when they move a single part of their body in a highly constrained environment (under the assumption of optimality, and that the agent controls using the same state-space basis as the model assumes). Psychology and neuroscience experiments moving towards more naturalistic movements and environments will pose a major problem for this form of control work, as the state-space grows and becomes more difficult to explicitly specify. 
The point of modelling a separation between the agent’s beliefs and those of the experimenter was a little lost on me, as it’s not clear to me what exact scientific application this could have.

*EDIT:*
I thank the authors for clarifying their motivation for modelling a separation between the agent's and experimenter's beliefs, and on reflection I agree this contribution is a valuable one. I have increased my overall score accordingly, and would recommend that the authors attempt to make the points they have listed here in their response clearer in the paper if possible. Thanks also for the clear response to my note about the limitations of these traditional frameworks for assessing motor behaviour. I agree that acknowledging this point in the limitations of the paper would be helpful for readers.
limitations_and_societal_impact: Yes.
review_title: null

Meta-review: The paper introduces a theoretical framework for estimating cost functions in linear, quadratic, Gaussian systems with action- and state-dependent noises. Algorithms for inference are derived for the complete and partial observation cases and distinguish between the agent’s and the experimenter's inference tasks. The approach is validated on synthetic and experimental data and applied to human sequential sensorimotor behavior.

Whilst the proposed approach is a simple applications of dynamic Bayesian inference and maximum likelihood estimate, the proposed inference formulation was considered original. The theory was derived under linear, quadratic, Gaussian assumptions, but the approach is demonstrated to work well in real-world applications, such as real arm and eye movement data. The clarity of the paper could be improved, especially w.r.t. to some literature.




Decision: Accept (Poster)