Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper tackles the problem of generalizing to out-of-distribution human behavior when using sim2real methods to develop assistive robotics capabilities. To that end, the authors formulate what they call the 'Assistive Personalization Problem': an assistive problem setting focused on generalization to unseen human behavior and preferences. To train policies for this problem setting, the authors propose the Prediction-based Assistive Latent eMbedding (PALM) framework, which involves learning a latent space by predicting future human actions from a history of observations and actions. Empirical results show that this approach outperforms state-of-the-art sim2real approaches used in non-assistive tasks and human-robot interaction approaches that train representations by predicting observed states or rewards. Additionally, the authors include a mechanism to adapt the learned latent space at test time and show empirical results that this helps with generalization.
strengths_and_weaknesses: **Strengths**
- In general, I found the paper to be well written, easy to follow, and enjoyable to read
- The problem setting is well motivated and clearly described
- The proposed method of learning a latent space (including test-time optimization) is a natural and elegant approach to the problem
- The experiments and ablations are sensibly designed
- The empirical results presented in the paper are promising

**Scope for Improvement**
- The tasks used to evaluate the approach are relatively simple (reaching and itch scratching). I think the contribution could be stronger if the authors included additional, potentially more complex, assistive tasks in their empirical analysis (e.g. dressing, bathing, feeding, drinking). 
- Sim2Real is used to motivate the work, however the work is done entirely in simulation with synthesized human behavior. It would strengthen the submission if the authors included some evaluation with realistic human preferences and behavior.
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: I am recommending to accept the paper because it presents a novel problem formulation that is interesting and well motivated. Additionally, the proposed learning-based approach to the problem is sensible and elegant -- and the empirical results presented in the paper are strong.
issues: To improve the submission, I would suggest including another couple of assistive tasks to your empirical results or adding some discussion that makes a strong argument for why the experiments presented are expected to generalize to other tasks.

Because sim2real transfer is not really addressed in your experiments, you might also consider reframing the paper to have less emphasis on sim2real in the motivation.  Or you might consider including additional discussion to help readers conceptually bridge the gap between this work and the idea of sim2real transfer that motivates it.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper addresses the problem of sim2real in the context of assistive tasks, in which there is an agent that is acting and requiring assistance.  If the diversity of behaviors is not properly captured, trained agents may encounter out-of-distribution policies at test time.  The authors posit that the robot will have access to interaction history at test time, which motivates learning a “good” latent parameterization that generalizes to humans and which are adaptable with test-time interaction data.  The authors offer empirical evidence to support that the latent representation may be interpreted and leveraged using human policies trained from a policy distribution, and that if out-of-distribution humans are encountered, the parameterization can be adapted accordingly.  The supplement and video also provides further experimental studies, including physical human-robot experiments.
strengths_and_weaknesses: Better computational approaches to solving the problem of human-robot cooperation and assistance is important for the robotics community in general.  In many studies, the human is assumed to be an adversary, where the robot must exploit the learned human strategy.  In this paper, the authors provide a solution where robot must assist the human in assistive HCI settings, such as itch scratching.  The high-level problem statement - a latent representation amenable for interaction, and which can be adapted to fine-tune the latent embedding according to any distribution shifts seen at test time - is a very important and impactful problem.

The paper is well-written, and comprehensive.  The authors provide reasonable context, and demonstrate, through compelling examples, the PALM approach, which illustrates the overall utility and provide a basis for comparison with other methods.  The comparisons between other approaches (RMA, LILI, RILI) strengthen the paper’s arguments.  The authors also provide (in the supplement and video) a real user study to further demonstrate their approach.

One of the weaknesses of this paper is that the link between how the proposed computational approach addresses generalizability is not clear.  After reading the paper several times, I could not figure out what aspects of the approach allow for a latent structure that is amenable to assistance.  Does the human have to behave optimally and rationally for the approach to work?  Relatedly, the term “generalizability” seems to be a misnomer in the context used in the paper.  If $D_{train}$ and $D_{test}$ are sufficiently different (by way of counterexample, suppose that $D_{test}$ leads to adversarial policies), then the learned structure may be completely useless at test time.  

Also, a complete study of the generalizability of the approach was never fully explored, probably since it is not required for the assistance domain, since it is assumed that humans are mostly cooperative.  Could the authors adopt a more specific definition for $D_{train}$ and $D_{test}$, or provide assumptions on the nature of these distributions or the nature of the human behaviors, to make the assistance and generalization claims clearer?

Related to the points above, in the Experiments section, it is never explained why the results of the other methods (LILI, RILI, RMA) do not generalize well, yet PALM does.  Particularly, the results for the reacher domain show that the circular latent parameterization correctly emerges for PALM, even without test-time optimization, though this fails to emerge in the other approaches.  From a computational standpoint, is it one of the loss components (e.g. the PPO or BC loss components), or is it differences in decoder output or less reliance on rewards that causes PALM to predict a human better than LILI or RILI?

In Sec. 3.2, why does the adaptation step only update $\theta$ (and not also $\phi$)?  Clearly, the decoder was trained on $D_{train}$, and will suffer the same domain-shift issues as the encoder when presented with $D_{test}$?

Another significant weakness is that, in Section 3.2, the presented adaptation scheme is not at all convincing for solving the objective in eq. (2).  How does the approach deal with catastrophic forgetting of the original latent space?  How many iterations does there need to be for the test distribution to be accurately captured?  Since the solution to (2) cannot really be achieved (since there are no proofs to this effect), then this is entirely a heuristic / empirical claim.  What are the hyperparameters (e.g. $\delta$), and how are they selected? 

In the Assistive Itch Scratching domain, the co-optimization of $\pi_R$ and $\pi_H$ seems limiting to implement in practice, since there are no ground truth rewards and humans could behave unlike eq. (5).  How would the approach work if the ground truth human policy $\pi_H$ was not co-trained? 

There are various formatting issues.  For instance, Figure 5 (on p. 8) is dislocated from where it is first referenced (p. 6).
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: The proposed approach for learning the right representation for human-robot interaction in assistance settings is an important and sometimes overlooked area.  While this paper does not seem to propose a novel learning paradigm, it does focus on setting up the problem such that a coherent latent representation can be learned, and seems to generalize, to an extent, other similar approaches (LILI, RILI).  I believe that the proposed adaptation scheme does not solve the overall objective in eq. (2), and should be re-written.  However, I also feel that the paper carries strong empirical evidence pointing to promise of using such approaches in realistic settings.  My recommendation leans toward acceptance.
issues: 1. The authors are strongly encouraged to add a description of why other approaches (LILI and RILI) are not able to handle the assistive domain, and what aspects of the proposed approach lend to better identification of a "more correct" latent representation than these other approaches.  
 - Does this relate at all to any unstated definitions or assumptions for $D_{train}$ and $D_{test}$?  For instance, are there limitations to the domain shift from $D_{train}$ to $D_{test}$ or can the shift be arbitrarily large?
 - The authors should also highlight any such differences in the experimental comparisons of Sec. 4.

2. There are missing details in Sec. 3.2:
  - Please explain why $\phi$ is not included in the test-time adaptation update. 
  - Please explain how catastrophic forgetting is dealt with in the adaptation procedure.
  - Also, how does this factor into the selection of the hyperparameters such that the distribution shift to the test distribution is properly handled by the adaptive procedure, yet does not destroy the ability to assist within the original training distribution.

3. The authors should explain whether their approach in general can handle non-rational or suboptimal human participants, or whether any extensions are necessary.

4. Please fix the formatting issues.  There are not many, but one example is Figure 5, which is dislocated from where it is first referenced.
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper focuses on learning latent representations for generalization in assistive tasks such as reaching and itch-scratching. The latent space is learned directly from the history of interactions by predicting partner actions during training. During testing, the learned space is adapted by observing the partner actions to account for OOD partner policies.
strengths_and_weaknesses: Strengths:

- The paper considers learning latent space representations from history of human actions that are observable during assistance, instead of states / rewards / environments
- The paper compares their methods with multiple baselines with different strengths

Suggestions for Improvement:

- The paper can be significantly improved by being more cohesive about the motivation, the claims, and the assumptions of the methods. The paper motivates itself by citing the use-case of robotic caregiving. Robotic caregiving is a multifaceted and highly contextual domain. No robotic caregiving context is complete without specifying the four things: 1) target user, 2) the caregiver (optional), 3) the environment, and 4) the robot. Without knowing the target user of this technology, it is unreasonable to assume that the user is able to move their arms during the activity, which affects the POMDP formulation, choice of action space, as well as the co-optimization framework that induces reward seeking behavior from the humans. If a patient could move their arms, can the paper justify why they would need robotic assistance with itch scratching. Without knowing the environment and the robot, it is unreasonable to assume that partner interactions themselves would be able to capture the differences between training time and test time for generalization. The paper needs to clarify the setting - if it is a robot-arm mounted on a wheelchair, are they considering generalization to a different user who is lying on a hospital-bed for itching? The assumptions need to be very clearly specified in the paper and the claims should be carefully considered and re-worded.

- There is a lack of real-world experimentation for these methods, which can provide more insights on the practicality of these approaches for the robotic caregiving context. Is this a realistic assumption that during training time the privileged information would be available for robotic caregiving tasks? Can the paper discuss how much data would be needed to train an encoder that takes in interaction data to generate latent space, and if that scale of robotic caregiving data collection is feasible in the real world? How would these methods translate beyond simulation?

- The paper can also be stronger by clarifying what personalization means in the context of learned embeddings (Section 3). Personalization in HRI and in the robotic caregiving world can have many meanings and it would make the paper clearer, if it can explain clearly in what context this is used.

- For itch scratching, can the paper justify why the action space only considers joint and end-effector positions? Without contact forces, how would itch scratching be possible to generalize between different users?

- How would this framework generalize when there is a human caregiver present to help the robot with the care recipient (two humans - one robot)? This is a very probable scenario for robotic caregiving (in fact more probable than just a dyadic robot-user scenario presented in this paper) and a discussion on extending the framework from a two-player Dec-POMDP would make the paper stronger.

- The regularization term (eq. 3) is motivated by VAE formulation and helps in grouping similar humans closer, assuming a normal distribution. By similar humans, the paper probably means similar itch locations and history of actions ( a clarification would be helpful). Have the authors considered other formulations that directly attempt to use VAE structures for sequential data such as Variational RNNs ( https://proceedings.neurips.cc/paper/2015/hash/b618c3210e934362ac261db280128c22-Abstract.html )? A discussion on the modeling choice would be helpful.

- Can the paper clarify how behavior cloning is used to stabilize their policy training? How is covariate shift dealt with in this case?

- The paper should be carefully revised for typos, punctuation, other formatting etc. Some examples: 
     - Line 36: comma after when they are not
     - k_H should be defined in Section 4.1
     - Fig. 2 can use a more detailed caption.
     - Is the avatar in the assistive gym a female avatar? Kindly clarify the choice of pronouns (Line 84)

robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: The paper in its current state needs realistic evaluations and justification of the choice of methods. 

##########################
After rebuttal period: 
##########################

I thank the authors for engaging in thorough discussions with me and responding to many of my comments. I really appreciate it. I have read the rebuttal in detail and I am revising my decision from "Strong Reject" to "Weak Reject". While I am still not fully convinced that the paper is showing evidence of what it is claiming which is to learn representations that enable generalization in assistive tasks, the paper has improved through these revisions. I have updated my scores accordingly. A side note: I understand that recruiting people with disabilities is indeed hard, but one thing to consider in the future would be to engage the stakeholders (occupational therapists, caregivers, care recipients etc.) even through some remote zoom discussions, atleast to confirm that the formalization and the experimental design is real-user inspired. Also, while experimenting with different activity levels is great, the mobility limitations of end users do not vary globally but are instead a function of relative local joint constraints which can be characterized by their AROM and PROM. Thank you for your work. 

issues: See the “Suggestions for Improvement” section above.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper proposes that explicit modeling of a latent space results in better performance with OOD human policies on assistive tasks. The PALM framework learns an encoder that maps a history onto a latent code that is the policy is conditioned upon. Experiments show that PALM policies lead to good performance on test human policies on an assistive reacher and assistive itch scratching task. 
strengths_and_weaknesses: Strengths:
- Assistive robotics is an important area and this paper is relevant to the community. Adapting to unseen users is essential and methods that generalize well would be of interest to researchers applying machine learning methods towards assistive robots and more generally, HRI.
- To my knowledge, the method is novel (up to the points below); I have not seen work in assistive robotics that adapt the test time latent space. The unseen reward setting is interesting. 
- The experiments do show that learning a suitable latent space is important for performance and that test time adaption helps. 

Weaknesses:
- The paper could better describe the method, particularly time-time adaptation. If my understanding is correct, the authors propose to optimize the latent space with specific test user data. It was unclear to me why it is necessary to adapt the entire latent *space*, rather than just the latent code $z$? If a good latent space is already learnt, why is there a need to further refine it? Specifically, how does training distribution $p_{train}$ differ from test distribution $p_{test}$ in a manner that refining the latent space becomes necessary? 
- On a related note, it is unclear how PALM would scale to more complex settings. It seems like test time adaption of the whole space may require significant amounts of data and more than a few gradient steps? Also, with insufficient data, there is a risk of overfitting to specific scenarios (even with a specific test user)? 
- It would also be helpful if the authors can clarify how such a system could be used in a deployed real-world system. 
- Moreover, the experimental domains are relatively simple, e.g., compared to the lunar lander and driving domains in [17]. Can the authors provide some discussion around how the method scales? As stated above, it would be helpful to clarify how the test time adaptation was performed in the experiments. 
- The paper would benefit from additional details and minor corrections to improve clarify:
  - line 92: $\mathbf{o}^H$ ($t$ subscript is missing?)
  - What is the episode length (time-steps) for both experiments? For each test user, how many episodes are tested and used for adaptation? 
  - I could not understand Fig 6; can the authors provide a summary of what the dots represent and how they relate to the arm?
  - line 190: “reward-seek” -> “reward-seeking”? 
  - Figs 4 and 5 seem to have $c_{kl} = 0.1$ for the first two algorithms? I believe the first bar should be  $c_{kl} = 0$? 

robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: Getting robots to behave appropriately when helping unseen users is an important topic and I would like to support this work. However, there are technical issues that give me pause and I am unable to recommend an accept at this point. I am willing to adjust my scores based on the clarifications. 

-- Post Response --
I have updated my score to a Weak Accept based on the clarifications. 
issues: Please see the weaknesses above. I would especially appreciate clarity regarding the fine-tuning method proposed.
review_title: null

Meta-review: metareview: This paper considers the task of training policies for human-robot collaborative tasks and observes the need for out of distribution generalisation for human policies as a pre-requisite for training. The authors present an approach for learning a latent representation for human policies that may be adapted online. 

Reviewers consider the motivation compelling and highly relevant for assistive robotics applications. In particular, reviewers consider the idea of learning a latent space for human policies as elegant. Further, thorough comparisons with related efforts strengthen the paper’s arguments. 

The primary weakness of the paper is that the tasks used to evaluate the approach are relatively simple (reaching and itch scratching). This raises the question of whether and to what extent the method can scale to more realistic tasks. Any experiments to suggest that the approach can bridge the sim2real gap have been suggested since the current results are purely in simulation. A better technical exposition of the test time adaptation has also been suggested. 

During the rebuttal phase,  the authors provided additional clarification/results for key reviewer questions such as mechanism for generalisation, overfitting, choice/training of encoders and their inclusion strengthens the overall paper. The major quantitative evaluation is on moderately complex tasks (e.g., itch scratching) supplemented by a user study. The user study is conducted in lab scenario (not with real users) and the setup bears resemblance to human studies in other human-collaborative domains (such as driving, collaborative assembly etc.). 

Overall, the paper contributes an approach for learning latent representations from interaction data that can be adapted online based on distribution shifts observed online. The manuscript can be strengthened by a clearer exposition on the mechanism/nature of generalisation attained, mechanisms for preventing overfitting and and an analysis of potential scalability to complex scenarios. Further, since the learning problem is scoped towards the specific domain of assistive tasks (as opposed to general robot tasks), a clearer setup of the problem context and a statement on the scalability of the results to end-users may also be considered by the authors. 

confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept (Poster)