Learning Representations that Enable Generalization in Assistive Tasks
Authors: Jerry Zhi-Yang He
Zackory Erickson
Daniel S. Brown
Aditi Raghunathan
Anca Dragan
Venue: CoRL 2022 Poster (2022)
Keywords: assistive robots
representation learning
OOD generalization

Abstract: Recent work in sim2real has successfully enabled robots to act in physical environments by training in simulation with a diverse ``population'' of environments (i.e. domain randomization). In this work, we focus on enabling generalization in \emph{assistive tasks}: tasks in which the robot is acting to assist a user (e.g. helping someone with motor impairments with bathing or with scratching an itch). Such tasks are particularly interesting relative to prior sim2real successes because the environment now contains a \emph{human who is also acting}. This complicates the problem because the diversity of human users (instead of merely physical environment parameters) is more difficult to capture in a population, thus increasing the likelihood of encountering out-of-distribution (OOD) human policies at test time. We advocate that generalization to such OOD policies benefits from (1) learning a good latent representation for human policies that test-time humans can accurately be mapped to, and (2) making that representation adaptable with test-time interaction data, instead of relying on it to perfectly capture the space of human policies based on the simulated population only. We study how to best learn such a representation by evaluating on purposefully constructed OOD test policies. 
We find that sim2real methods that encode environment (or population) parameters and work well in tasks that robots do in isolation, do not work well in \emph{assistance}.  In assistance, it seems crucial to train the representation based on the \emph{history of interaction} directly, because that is what the robot will have access to at test time. Further, training these representations to then \emph{predict human actions} not only gives them better structure, but also enables them to be fine-tuned at test-time, when the robot observes the partner act.

Learning Representations that Enable Generalization in Assistive Tasks
Jerry Zhi-Yang He', Aditi Raghunathan", Daniel Brown', Zackory Erickson', and Anca D. Dragan'
UUiiveriity of California Berkeley, PCarnegie Mellon University, University of Utah, (hzyjerry, @berkeber edu, dsbrownecs. utah. edu, {raditi,
edu
Abstract: Recent work in sim2real has successfully enabled robots act in physical environments by training in simulation with a diverse "population" of enviromments (i.e. domain randomization). In this work, we focus on enabling generalization in assistive tasks: tasks in which the robot is acting to assist user (e.g. helping someone with motor impairments with bathing or with scratching an itch). Such tasks are particularly interesting relative to prior sim2real successes because the environment now contains human who is also acting. This complicates the problem because the diversity of human users (instead of merely physical environment parameters) is more difficult to capture in population, thus increasing the likelihood of encountering time. We advocate that generalization to such OOD policies benefits from (1) learning good latent representation for human policies that test-time humans can accurately be mapped to, and (2) making that representation adaptable with test-time interaction data, instead of relying on it to perfectly capture the space of human policies based on the simulated population only. We study how to best learn such representation by evaluating on purposefully constructed OOD test policies. We find that sim2real methods that encode environment (or population) parameters and work well in tasks that robots do in isolation, do not work well in assistance. In assistance, it seems crucial to train the representation based on the history of interaction directly, because that is what the robot will have access Further, training these representations to then predict human actions not only gives them better structure, but also enables them to be fine-tuned test-time, when the robot observes the partner act. https: / //aa/paaaeeeeeeee caregiver weeiveerggtttte github. io.
-istiiibiiiiitoionnn (OOD) human policies at test
test time.
Keywords: assistive robots, representation learning, OOD generalization
Introduction
Our ultimate goal is to enable robots to assist people with day to day tasks. In the context of patients with motor impairments, this might mean assistance with scratching an itch, bathing, or dressing [1, 2, 3]. These are tasks in which doing reinforcement learning from scratch in the real world not feasible, and so sim2real transfer is an appealing avenue of research. Sim2real methods for physical robot tasks in isolation typically work by constructing diverse "population" " of environments and training policies that can work with any member of the population (e.g. range of parameters of physies simulator or a range of lighting and textures) [4, 5, 6, 7, 8, 9, 10, 11].
Similarly, population-based (self-play) training has proven successful in zero-sum games against humans [12. 13, 14]. But unlike tasks the robot does in isolation, assistance requires coordinating with human who is also acting. And unlike competitive settings, assuming the human to be optimal when they are not, can result in dramatically poor performance [15]. Thus, in sim2real for assistance. we have to design population of potential users and strategies to train with, akin to the physical environment parameters in typical sim2real tasks, rather than the standard population- based training approaches used in competitive settings. But designing population that is diverse and useful enough to enable generalization to test-time humans, each with their own preferences, strategies. and capabilities, remains very challenging. making it likely that test-time partners might lie outside of the
6th Conference on Robot Learning (CoRI.202) Auckland, New Zealand.

---

oo-no .(. (ov.
Encoder
Latent
Decoder
Policy
Test-time
Test
(Oc-Nu a(--N (O
af
Adaptation
The framework for jointly learning the Personalized Latent Embedding Space and the robot policy. During training time, we train all components end-to-end to optimize for action prediction (orange) and the robot policy (green). At test time, resulting latent space captures underlying structure of the preferences and strategies of the training humans. distribution the population was drawn from. Therefore, sim2real methods for assistance will need to be ready generalize to out-of- didttribbittiioiionn partner policies.
Figure I:
further optimize for this objective perform test-time adaptation (red). The
In this work, we identify two principles as key to enabling better generalization. First is that we benefit from learning latent space of partners that distills their policies down to structure that is useful for the robot's 's policy and that makes it easy to identify partners at test time. Second that need to be prepared for this space not perfectly capture the space of real human policies, and design so that is adaptable with real test-time interaction data.
We thus propose framework that learns latent space directly from history of interaction by predicting the partner's 's actions. Our framework allows a robot to capture the relevant information about the human partner that the robot actually identify when starting to interact, and also enables test-time adaptation of the latent space itself when observing the partner's actions. When evaluated with partner policies purposefully design to be out-of-distribution we find that our approach leads to better generalization than prior methods which either do not learn a latent space at all [16], do not learn latent space directly based interaction history [7]. train latent space based on other observables, like states or rewards [17, 18]. Our contributions are four-fold:
We introduce an assistive problem setting where the focus is explicitly on generalization to out-of-die of-distrrbutton partner policies.
training policies for this problem setting, Prediction-based Assistive I Latent eMbedding (PALM). This enables us to study different methods for learning latent representations on how well they enable generalization.
We introduce a framework
3. We identify that the design choice of training latent space by predicting partner actions directly from history outperforms (I) state-of-the. -hrte-rr sim2real approaches used in non-assistive tasks that are based embedding environment parameters [7] well as (2) human-robot interaction approaches that train representations by predicting observed states or rewards [17, 18].
4. We propose adapt the learned latent space test time, upon observing the partner's actions, and show it leads to generalization performance gains.
The Assistive Personalization Problem
In this section, introduce the personalization problem in assistive context. In particular, goal is to learn a robot policy лæ that can assist a novel human partner zero-shot fashion, or with a small amount of test-time data.
Two-player Dec-POMDP. An assistive task can be modeled as a two-agent, finite horizon decen- tralized partially Markov decision process (Dec-POMDP) and is defined by tuple

---

(S.a.AR, AH. 2Qg. Qq,, Here S is the state space and AR are the human's and the robot's action spaces, respectively. The human and the robot share real-valued reward function R: P:Sx AR x Ah - R: however, we assume that the reward function is not necessarily observed by the robot, i.e. its parameters (e.g. the location of the human has an itch) are in the hidden part of the state. T S x Ag X Ah xS - [0, 1] is the transition function, which outputs the probability of the next state given the current state and all agents' actions. SRR and 22 are the sets of observations for the robot and human, respectively, and O Sx AR X Ah - S2R x S2z represents the observation probabilities. We denote the horizon of the MDP by T.
Target User. We target users with partial motor functions - common impairment for individuals with partial arm functions. This is an impairment that can occur in some people with cervical SCI, ALS, MS, and some neurodegenerative diseases - leading to the need for robotic assistance. We model the extent of the impairment as the privileged information in the Dec-POMDPP The robot does not know this a-priori and thus needs to adapt to individual users' capabilities.
The Robotic Caregiving Setup. We define the observation space for the robot and the human following [3]: the robot observes its own joint angles, and the human's joint positions in the world coordinate and contact forces; the human observes their joint angles (proprioception) and the end- effector position of the robot. When training with simulated humans, the robot gets reward signal (which depends on privileged information), and has use that signal to learn to implicitly identify enough about the human to be useful; at test time, the robot does not observe reward signal and must use what has learned training time to identify the human's 's privileged information and be helpful.
Distributions of Humans. Let function HH S2 x AH - [O, be the human policy that maps from local histories of observations of = (of ......,' over 22 to actions. We define two distributions of human policies πи E Duain Pws. In the assistive itch scratching, Ruain can be set of humans with different itch positions on their arms, which lead to their different movements, We refer to them as in-distribution humans. Aest contains out-of of-distribution humans whose itch position differ from those in the Puainn At training time. the robot has access to Prrain- Thus, it has ground-truth knowledge about the training human's privileged information, such as each human's 's itch position. At test time, we evaluate the robot policy by sampling humans πн Brest and directly pairing them with the robot policy. We evaluate the zero-shot and few-shot adaptation performance of the robot policy.
Objective. The main problem we study is how to leverage the training distribution to learn a robot policy πR: 22g Ar such that we achieve the best performance on test humans. Concretely, we define the performance of the robot and human as
T Σ
JTT: TH) =
R(s
(1)
Only given access to Fuain. our objective is to find the robot policy Λя arg max ((,TTH).TT Drest-
3 Learning Personalized Embeddings for Assistance with PALM
In this section, we present Prediction- based Assistive Latent eMbedding (PALM). We introduce the general framework of using a latent space to perform personalization an assistive context. We then highlight the advantage of action prediction in contrast to prior works. Finally, we describe how we can optimize PALM test time to work with dusto--distribution humans.
3.1 Learning an Assistive Latent Space
Given a training distribution of humans Ptrain 1, we would like to learn robot policy that can adapt to assist new users. To achieve that, a robot must learn to solve the task while efficiently inferring the hidden component that differs across humans. One natural way to do is to learn a latent space that succinctly captures what differs across humans in a way that affects the robot's policy. When deployed on a test human, the robot infers this latent embedding and uses it to generate personalized assistance.
We denote the latent space as 2 ~ E0(2: 2; TIx), where & encodes the trajectory T so far and outputs latent vector 2, The robot uses this latent space to compute its actions a. TT (of
(R 2.). We train
we describe how generate this distribution in Sec. 4.1
3

---

the base policy and the latent space encoder Eg jointly as they are interdependent [19] better robot policy leads different trajectories across humans, which turn leads to distinguishing z. See Appendix D on training details. Ideally, we would like to capture sufficient information to differentiate the humans, similar to performing a "dimensionality reduction" on human policies TH We hereby introduce different objectives for learning such latent space, and how our method learning by action prediction - makes a good fit for the assistive personalization problem.
3.2 How to Construct the Latent Space
Prior work and limitations LILI [17] and RILI [18] learn a latent embedding of the humans by predicting the next observations and rewards. While they have been shown work in predicting and influencing human behaviours, both methods assume access to the reward function at test time, which we do not have access to in the assistive setting - we don't know a-priori the preference and needs of a new user. RMA [7] enables fast robot adaptation by learning a latent space of environment parameters, such friction, payload, terrain type, etc. While works for single robot, it is unclear in human-robot settings, how we can encode human motions and preferences as environment parameters.
Learning by action prediction. Given history Tr-N: = ((of-NN .......).. (o.%.)) of N robot observation and human action pairs, as outlined in Fig. 1, we embed this trajectory to a low- dimensional manifold and use to predict a." The intuition is that if we able predict the next action by this human, we extract the sufficient information about the human's 's policy πи. The latent vector is representative of the trajectory so far and indicative of the person's 'ss future actions. We do this by training a decoder Ds parameterized by & to predict the next action from the encoder's output (Z)
Lpedd = min minllm [1 - - 11 +CKL KL(EG(ZTT.)IINNC) 0.0
(2)
The encoder & is recurrent neural network parameterized by 0. Here the second term is regularization term motivated by Variational Autoencoder [20, 21J, that enforces the latent space to follow a normal prior distribution. This encourages nearby terms in the latent space to encode similar semantic meanings. In the context of assistive tasks, this helps us better cluster similar humans closer in the latent space, and we show didactic example in Sec. 4.3.
3.3 Latent Space Adaptation at Test Time
work with new user, we would like encoding of the new user match the true latent information, z" of that user. In other words, we would like to minimize [[E(r)- (r)))22 =1' Because we do not know about the new users a-priori, we refer to as test time adaptation.
At test time,
only optimize for this objective via proxy, which
Since the PALM latent space is based on action prediction, we can adapt it to a new user by further optimizing the latent space. Note that because Eq. (2) requires only observation-action data, we do not need any additional label to perform test time adaptation. More formally, we collect small dataset of test-time interaction trajectories, T, and perform a few gradient steps to optimize both the encoder and decoder for Eq. (2): (0.6) (0, - -V (0.0) Epee (E..Ds.2) Dz.r). r).
The idea of test-time optimization has been shown improve perceptual robustness for grasping in sim2real research [22]. We follow similar pipeline, where we can improve the latent encoding by collecting unsupervised action data from test users. Here the main difference is instead of perceptual differences, our goal is to reduce the domain gap on test users.
4 Experiments
In this section, we evaluate our method PALM (Prediction-based Assistive Latent e.Mbedding) in collaborative human-robot environments varying tasks and varying populations of human models. In particular, we focus the distriibiitiiioo generalization by constructing different forms of -iddstrbbuttttn populations. We focus on empirically investigating the benefits of learning a latent space, the effect of different kinds of prediction on learning useful latent space, the properties of learned latent spaces, and the gains from test-time adaptation to humans.
We do not assume access to the person's 's sensorimotor action (e.g. joint torques). We define human action change in the person's 's Cartesian pose, which can be tracked externally.

---

4.1 Environments
Here we introduce two environments where we study assistive personalization. In both envi- ronments, the robot has to infer some hidden information from the human in order to success- fully solve the task. Note that these are examples meant for demonstrating the effectiveness of the algorithm, and we do not claim to solve the full robotic caregiving problem.
Center
Hidden
Assistive Reacher (Fig. 2) is 2D collaborative environment where two-link robot arm assists a point human agent to get to the target posi- tion. This target located (d aH, a.. sin where d is fixed value, and @H [-x,x] x is known the human, but not the robot. The human agent initialized randomly in the 2D plane with random hidden parameters @y E [-, E [0.5, The robot can only identify the target position by physical interactions - once the robot initiates contact, the human applies a force kH (cosarn + I. sin 7). Only by recognizing the human in terms of on, ku can the robot compensate the force, and successfully move the human to the hidden target. Each episode has 40 timesteps.
The assistive reacher environment. Left: the robot's goal is to move the human agent towards the hidden target. Right: the hidden goal's 's position can be inferred 90 degrees from the humans force output.
Figure 2:
The Scope of Generalization. We define Iwain as 36 samples uniformly sampled from ah [-x, k1. E [.5] and Diest as 12 samples uniformly sampled from oh E [5.T], E [0.5, 1.5].
Assistive Itch Scratching (Fig. 1) is adapted from assistive gym [3]. It consists of a human and wheelchair-mounted 7-dof Jaco robot arm. The human has limited mobility - they can only move the 10 joints the right arm and upper chest, and needs the robot's assistance scratch the itch. An itch spot is randomly generated on the human's 's right arm. The robot does not directly observe the itch spot, and relies on interaction with the human infer its location. Each episode has 100 timesteps.
We use co-optimization to create Quain and Prest for Assistive Itch Scratching. A benefit of the co-optimization framework is that it naturally induces reward-secking behaviour from the human and the robot, which simulates assistance scenarios. For instance, to generate more inactive human policies, can introduce weighting term in the reward function for human action penalties Rp fp=cq Cp- [|>H(s))]]] where Cp is a constant controlling the penalty. The overall objective becomes
max E
Σ [R(s).T((s)),TR(G)) c ||T((s)) (
(3)
The Scope of Generalization. We are motivated by real world applica- tions where users tend to have different levels of mobility limitations, or itch locations in different body parts. To generate synthetic population to capture such diversity in itch scratching task, we explore different co-optimization settings (1) we assign different human action penalty to be cp = 3,5,,,4 where larger penalties lead to the human agent exerting less effort. (2) We simulate different itch positions on the human's 's arm and train co-optimized human and robot policies conditioned on them. This leads to qualitatively different strategies for the human and the robot. Note that this serves first step to understanding how different methods generalize, since we never expect to be able to capture the diversity in humans perfectly. For training, we use Proximal Policy Optimization (PPO) to optimize human and robot policies an interleaving fashion. Note that we also keep the co-optimized robot policy and use it to obtain expert actions for assistive policy training (see Sec. 3.1 and supplement for full details).
Figure 3: Definition of Puain and Pest
To construct Zuain and Piest we divide the two arms' areas into four equal portions, as shown in Fig. 3, and generate human policies conditioned on itch positions in these areas. Prain consists of three of the four portions and Ziest consists of the remaining one. We then construct three distribution sets in increasing order of difficulties. In the first distribution Φ1, we confine itch positions from line-shaped region. In 2², we sample from all the arm areas. Note that D and D² are constructed by setting action penalty Cp 3. In Φ3 we combine humans of cp 3,3.5, 4. each trained with two
5

---

different random seeds. This adds the extra complexity of human activity levels. We simulate 12 in-distribution humans from each of the three training portions under each action penalty.
4.2 Baselines
We compare with baselines that do not learn an explicit latent space well as existing methods for adaptation via learning latent embeddings.
MLP and RNN. We follow [16] that trains sequential models to enable adaptation to simulated humans. We explore using a recurrent neural network or a feed-forward network on concatenated state-action histories. The models directly output robot action, and there is no latent space modeling.
ID-based Human Embeddings. In contrast learning latent space from history, another class of method studies encodes designde environment parameters [7, 19, 23]. We focus specifically on RMA [7], two-phased method that first learns to encode task-ID (phase I) and then trains recurrent network to regress to the embeddings from observation history (phase II). For training a quadruped robot, RMA encodes the physical parameters (friction, payload, etc) of the environment. The first stage trains policy with ground-trun information, and the second phase performs environment identification. While RMA is shown to be effective for learning policy for in-distribution environments, it is unclear how well it generalizes to out-of- distribution environments. Furthermore, in assistive tasks, it is unclear how to construct the "ground-truth ID" for phase that quantifies the user characteristics. We study RMA-Param and RMA-Onehot, where we assign each training human one-hot vector. For RMA-Param. we use a three- dineeniimnsional vector that includes the x,y position of the itch position and the arm index. When there are multiple human activity levels as mentioned in Sec. 4.1, we introduce fourth dimension with an integer to indicate the action penalty Cp.
0.01
= 0.1
9n = 10
Ca = 3.0
Figure 4: Latent space of PALM in the assis- tive reacher environment when we sam- ple humans ar E [-x (red).,r n (blue)) con- tinuously.
LILI and RILL We consider two other methods of the PALM framework: LILI [17] and RILI [18]. As mentioned in Sec. 3.1, LILI jointly predicts future observation and reward, and RILI predicts reward. Given that reward is only available training time, we cannot perform test time optimization for LILI and RILI.
Ablations of PALM. Our method has several components: we use recurrent neural network to encode interaction history, and use its output to minimize prediction loss Lpred and policy loss Lpol- We also use the KL term in Eq. (2) to regularize the human embeddings. To test the effectiveness of our method, we separate different parts and create a set of baselines. We hereby describe them in detail: (1) No Lpred: the model shares the same encoder and policy network architecture, yet don't optimize for Lpred- By removing the prediction loss in Fig. 1, the latent space is not explicitly trained contain human information.
(2) No CKL: regularization in the latent space. (3) Frozen embedding: instead of jointly training embeddings and the policy network, we first train the encoder on expert data, freeze it, and then train robot policy. We include an ablation study of our main experiments in the appendix.
4.3 Didactic Experiment in Assistive Reacher Environment
Can PALM learn a meaningful distribution from the interaction? Unlike other ID-based methods like RMA, PALM does not have access to the human parameters at training time. We study whether PALM can learn a meaningful latent space without explicitly knowing this information. We sample training humans from ah [-x,r] continuously. We train PALM with different amount of prior regularization, I from Eq. (2). We train using a recurrent window of length 4 and a batch size of 512 episodes. Additional training details can be found in the supplementary material.
We average test results using 100 episodes and visualize the results in Fig. 4.3. Given that humans are parameterized by ah, KH, the ideal embedding space looks like ring with a small blob in the center.

---

PALM 1g-0.), 0.J, opt
RMA param
RMA
RNN
OOD
80
RMA Latent Space
200
PALM =
PALM 0.1 PALM C22 0.1 test optim
RNN
LILI
RILI
Figure 5: Top left: evaluation of PALM and baselines on in distribution (green) and out-of of-distribution (pink) humans. Right and bottom: visualization of the latent embeddings of different methods. OOD humans highlighted red crosses. Best viewed electronically.
The ring corresponds to the 2D projection of on and the blob denotes the initial part of interaction before contact, which is indistinguishable. We find that while PALM never observes the underlying parameter @H, it can learn latent space that characterizes an- Interestingly, varying the amount of regularization qualitatively affects the shape of the latent space. Setting the VAE regularization CKL = 0.1 recovers latent space that most resembles to the ideal latent space.
4.4 Assistive Reacher Main Experiment
Experiment Setting. We use the finite Puain described in Sec. 4.1 and train all baselines for 200 epochs with 512 batch size. We then evaluate the trained policies on Bees. We normalize the resulting reward with respect to oracle reward.
Results. We average test results using 100 episodes. On in-distribution humans, we find that all methods successfully follow the right policy that assists the human to reach their goal. This shows that they all successfully predict the human latent information explicitly or implicitly. On out-of-distribution humans, the methods are no longer guaranteed to predict the correct embedding. PALM with action prediction significantly outperforms other methods. With test-time adaptation, PALM further improves.
Visualizing the latent space. We qualitatively study generalization visualizing the latent space as well as the mapped embeddings of both in-distribution and out-of-distribution humans (in red crosses) Fig. 5. Interestingly, only PALM with action prediction can infer the "ring" structure. RMA, RNN, LILI and RILI fail to do so. We hypothesize that because hand-crafted human IDs do not convey the information about human policy, RMA warped the IDs in arbitrary what that are harmful for generalization. The same happens with RILI and LILI. We hypothesize this is due to the inherent ambiguity in reward prediction: a low reward does not necessarily recover the human policy structure.
The visualization also offers some insight into why having CKI. regularization is helpful for generaliza- tion. Compare the latent space of PALM CKI. = and PALM скı. 0. the latter induces smoother distribution where test humans are better fitting in the "missing are" of the "ring". Further more, we that with test time optimization, the PALM latent space embeds the OOD human better, by filling in more of the arc.
4.5 Assistive Itch Scratch Main Experiment
Results. We follow a similar procedure as the reach environment to train itch scratching policy, and train for 240 epochs with 192 trajectories for batch size. As shown in Fig. observe PALM with action prediction has better generalization performance than other baselines. We see that in the simplified distribution 91, RILI and MLP have the best generalization performance among baselines. yet as the complexity of the training human distribution increases, they deteriorate. Detailed results of the ablation study are included in the appendix.
7

---

01-
0,001 opt
RMA param
1-56
j-zz
Presi (-36
Phert (432
Plean 1+230
Phosi
Thak Distributions
OOD Evaluations (normalized reward)
Latent Space
Latent Space
Figure 6: In assistive itch scratching. we sample humans by different itch positions and activity levels (varying action penalty cp). We visualize the in- and cut-of-distributin humans the two-link arm figures. We also visualize the embedding space of PALM and RMA, where color-code the embeddings of in-distribution humans. Here leave the embeddings other baselines supplementary material.
Visualizing the Latent Space To further investigate why PALM generalize better to OOD human than RMA baselines, We visualize the latent space of the "straight-line" distribution. As we see in Fig. 7,
Rel
PALM can capture the structure in human training distribution as two clusters, and also correctly embed the OOD humans distribution as a part of the upper the
Figure To better under extrapolations to OOD human, which have new itch locations highlighted in red on the left, we visualize the embeddings of both the IND (colored) and OOD (red) humans on the right.
RMA-based methods, other hand, can discover the structure of training humans. Yet qualitatively, they fail the correctly embed the OOD humans in proximity to the upper arm distribution.
arm distribution.
4.6 Limitations and Failure Cases
Although PALM achieves good average-case performance, it works best with humans sampled near the training distribution. If we pair the robot with an adversarial human, PALM is likely to fail lacks fall-back safety policy.
The major limitation of PALM is the requirement of generating human population. While provide one way to generate human populations based on weighted human-robot co-optimization, we lack ways systematically generate diverse and realistic human motions. One important direction for future work is to incorporate real user data to create training populations. Improving the realism of the training human population is likely crucial step supporting transfer real partners.
One future direction is extending to settings with one patient and one human caregiver. While our framework still applies, this leads to challenges including (1) learning joint separate latent space for human patient and caregiver, (2) modeling a population human caregivers for training in simulation, and (3) modeling communication between the human caregiver and patient.
5 Conclusion
Generalization is an important task for assistive robotics, and in this paper, we formulate a problem setting that focuses on Out-Of-Distriuttion users. To that end, we contribute framework PALM for learning a robot policy that can quickly adapt to new partners at test time. PALM assumes a distribution of training humans and constructs an embedding space for them by learning to predict partner actions. We can further adapt this embedding at test time for new partners. Experiments show that PALM outperforms state-of- ofe--ofttheee the-hrr-t approaches. We are excited by the potential of using PALM to enable robotic assistance in the future.

---

Acknowledgments
We would like to thank Ashish Kumar for discussions on the RMA method and Annie Xie for providing the implementation for LILI We would also like thank Charlie C. Kemp for feedbacks and insights on assistive tasks. This research was supported by the NSF National Robotics Initiative.
References
[1] D. P. Miller. Assistive robotics: an overview. Assistive technology and artificial intelligence, pages 126- 136, 1998.
[2] S. W. Brose, D. J. Weber, B. A. Salatin, G. G. Grindle, H. Wang, J. J. Vazquez, and R. A. Cooper. The role of assistive robotics in the lives of persons with disability. American Journal of Physical Medicine & Rehabilitation, 89(6):509-522 2010.
[3] Z. Erickson, V. Gangaram, A. Kapusta, C. K. Liu, and C. C. Kemp. Assistive gym: A physics simulation framework for assistive robotics. In 2020 IEFE International Conference on Robotics and Automation (ICRA), pages 10169-10176. IEFE, 2020.
[4] J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 JEEE/RSJ international conference on intelligent robols and systems (IROS), pages 23-30. IEEE, 2017.
[5] J. Tan, T. Zhang, E. Coumans, A. Iscen, Y. Bai, D. Hafner, S. Bohez, and V. Vanhoucke. Sim-to-real: Learning agile locomotion for quadruped robots. arXiv preprint arXiv: 10332, 2018.
[6] W. Yu, V. C. Kumar, G. Turk, and C. K. Liu. Sim-to-real transfer for biped locomotion. In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 3503-3510. IEEE, 2019.
[7] A. Kumar, Z. Fu, D. Pathak, and J. Malik. Rma: Rapid motor adaptation for legged robots. arXiv preprint arXiv:2107.04000 2021.
[8] J. Mahler, Liang, S. Niyaz, M. Laskey, R. Doan, X. Liu, J. A. Ojea, and K. Goldberg. Dex-net 2.0: Deep learning plan robust grasps with synthetic point clouds and analytic grasp metrics. arXiv preprint arxiv: weo.:700..00 09000.0931,.220 2017.
[9] Y. Wu, W. Yan, T. Kurutach, L. Pinto, and P. Abbeel. Learning to manipulate deformable objects without demonstrations. arXiv preprint arXiv: 1910. 13439, 2019.
[10] D. Seita, A. Ganapathi, R. Hoque, M. Hwang, F. Cen, A. K. Tanwani, A. Balakrishna, B. Thananjeyan, Ichnowski, N. Jamali, K. Yamane, S. Iba, J. Canny, and K. Goldberg. Deep Imitation Learning of Sequential Fabric Smoothing From an Algorithmic Supervisor. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2020.
[11] Hietala, D. Blanco-Mulero, G. Alcan, and V. Kyrki. Closing the sim2real gap in dynamic cloth manipulation. arXiv preprint wrXiv:2109.047711 2021.
[12] D. Silver, T. Hubert, J. Schrittwieser, I. Antonoglou, M. Lai, A. Guez, M. Lanctot, L. Sifre, D. Kumaran, T. Graepel, et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 362(6419):1140-1)
2018.
[13] D. Balduzzi, M. Garnelo, Y. Bachrach, W. Czarnecki, J. Perolat, M. Jaderberg, and T. Graepel. Open-ended learning in symmetric zero-sum games. In International Conference on Machine Learning, pages 434-443. PMLR, 2019.
[14] M. Jaderberg, W. M. Czarnecki, I. Dunning, I.. Marris, G. Lever, A. G. Castaneda, C. Beattie, N. C. Rabinowitz, A. S. Morcos, A. Ruderman, et al. Human-level performance in 3d multiplayer games with population-basee reinforcement learning, Science, 364(6443):859-888 865,,86,,55 2019.
[15] M. Carroll, R. Shah, M. K. Ho, T. Griffiths, S. Seshia, P. Abbeel, and A. Dragan. On the utility of learning about humans for human-ai coordination. Advances neural information processing systems, 32, 2019.

---

[16] D. Strouse, K. McKee, M. Botvinick, E. Hughes, and R. Everett. Collaborating with humans without human data. Advances in Neural Information Processing Systems, 34, 2021.
[17] A. Xie, D. Losey, R. Tolsma, C. Finn, and D. Sadigh. Learning latent representations to influence multi-agent interaction. arXiv preprint arXiv: 2020.
[18] S. Parekh, S. Habibian, and D. P. Losey. Rili: Robustly influencing latent intent. arXiv preprint arXiv:2203.12705, 2022.
[19] E. 7. Liu, A. Raghunathan, P. Liang, and C. Finn. Decoupling exploration and exploitation for meta-reiinforeement learning without sacrifices. In International Conference on Machine Learning, pages 6925-6935. PMLR, 2021.
[20] D. P. Kingma and M. Welling. Auto-encoding variational bayes, 2013, URL https://arxiv.. org/abs/ httg/abs/331....e /tt.//112...1... 6114.
[21] M. Watter, T. Springenberg. J. Boedecker, and M. A. Riedmiller. Embed to control: A locally linear latent dynamics model for control from raw images. CoRR, abs/1506.07365, 2015, URL http://arxiv. wwtpr//iiiorggaa/////aaaaaaaa/// aa5// 07365.
[22] T. Yoneda, G. Yang. M. R. Walter, and B. C. Stadie. Invariance through latent alignment. CoRR, abs/2112.08526,, 2021. URL https: /ttps:///rxiv... .ww./vorggggoo wwt./og////////ee/e/.oo.. /bs/ 08526.
[23] D. Kalashnikov, J. Varley, Y. Chebotar, B. Swanson, R. Jonschkowski, C. Finn, S. Levine, and K. Hausman. Mt-opt: Continuous multi-task robotic reinforcement learning at scale. CoRR, abs/2104.08212, 2021. URL https: //ap:v////iivv//// http://aaxxvv..raab///s/sss/////
08212.
[24] A. Clegg, Z. Erickson, Grady, Turk, C. C. Kemp, and Liu. Learning to collaborate from simulation for robot-assisted dressing. IEEE Robotics and Automation Letters, 5(2): 2746-2753, 2020.
[25] A. Gupta, A. Pacchiano, Y. Zhai, S. M. Kakade, and S. Levine. Unpacking reward shaping: Understanding the benefits of reward engineering sample complexity. arXiv preprint wxXi:2210.09579 2022.
[26] Y. Zhai, C. Baek, Z. Zhou, J. Jiao, and Y. Ma. Computational benefits of intermediate rewards for goal-reaching policy learning. Journal of Artificial Intelligence Research, 73:847-896, 2022.
[27] S. Bowman, L. Vilnis, O. Vinyals, A. M. Dai, R. Józefowiez, and S. Bengio. Generating sentences from a continuous space. CoRR, 0tbs//511.003444 2015. URL http: http:/axxvvv./// org/ 0bs/1511.06344
[28] Chung, K. Kastner, L. Dinh, K. Goel, A. C. Courville, and Y. Bengio. A recurrent latent variable model for sequential data. CoRR, abs/1506.02216 2015. URL http:://axxiv.orrg abs/1506. 02216.
[29] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: /aop:0777006444 2017.
[30] Ross, G. Gordon, and D. Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 627-635. JMLR Workshop and Conference Proceedings, 2011.
10


Reviews:
[{"review_id": "vy_mSQg9gTK", "reviewer": "Reviewer_Up7D", "summary": "This paper proposes that explicit modeling of a latent space results in better performance with OOD human policies on assistive tasks. The PALM framework learns an encoder that maps a history onto a latent code that is the policy is conditioned upon. Experiments show that PALM policies lead to good performance on test human policies on an assistive reacher and assistive itch scratching task. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Assistive robotics is an important area and this paper is relevant to the community. Adapting to unseen users is essential and methods that generalize well would be of interest to researchers applying machine learning methods towards assistive robots and more generally, HRI.\n- To my knowledge, the method is novel (up to the points below); I have not seen work in assistive robotics that adapt the test time latent space. The unseen reward setting is interesting. \n- The experiments do show that learning a suitable latent space is important for performance and that test time adaption helps. \n\nWeaknesses:\n- The paper could better describe the method, particularly time-time adaptation. If my understanding is correct, the authors propose to optimize the latent space with specific test user data. It was unclear to me why it is necessary to adapt the entire latent *space*, rather than just the latent code $z$? If a good latent space is already learnt, why is there a need to further refine it? Specifically, how does training distribution $p_{train}$ differ from test distribution $p_{test}$ in a manner that refining the latent space becomes necessary? \n- On a related note, it is unclear how PALM would scale to more complex settings. It seems like test time adaption of the whole space may require significant amounts of data and more than a few gradient steps? Also, with insufficient data, there is a risk of overfitting to specific scenarios (even with a specific test user)? \n- It would also be helpful if the authors can clarify how such a system could be used in a deployed real-world system. \n- Moreover, the experimental domains are relatively simple, e.g., compared to the lunar lander and driving domains in [17]. Can the authors provide some discussion around how the method scales? As stated above, it would be helpful to clarify how the test time adaptation was performed in the experiments. \n- The paper would benefit from additional details and minor corrections to improve clarify:\n  - line 92: $\\mathbf{o}^H$ ($t$ subscript is missing?)\n  - What is the episode length (time-steps) for both experiments? For each test user, how many episodes are tested and used for adaptation? \n  - I could not understand Fig 6; can the authors provide a summary of what the dots represent and how they relate to the arm?\n  - line 190: “reward-seek” -> “reward-seeking”? \n  - Figs 4 and 5 seem to have $c_{kl} = 0.1$ for the first two algorithms? I believe the first bar should be  $c_{kl} = 0$? \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Getting robots to behave appropriately when helping unseen users is an important topic and I would like to support this work. However, there are technical issues that give me pause and I am unable to recommend an accept at this point. I am willing to adjust my scores based on the clarifications. \n\n-- Post Response --\nI have updated my score to a Weak Accept based on the clarifications. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "Please see the weaknesses above. I would especially appreciate clarity regarding the fine-tuning method proposed. ", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}, {"review_id": "tFpWL93HUX", "reviewer": "Reviewer_EjUi", "summary": "The paper focuses on learning latent representations for generalization in assistive tasks such as reaching and itch-scratching. The latent space is learned directly from the history of interactions by predicting partner actions during training. During testing, the learned space is adapted by observing the partner actions to account for OOD partner policies.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The paper considers learning latent space representations from history of human actions that are observable during assistance, instead of states / rewards / environments\n- The paper compares their methods with multiple baselines with different strengths\n\nSuggestions for Improvement:\n\n- The paper can be significantly improved by being more cohesive about the motivation, the claims, and the assumptions of the methods. The paper motivates itself by citing the use-case of robotic caregiving. Robotic caregiving is a multifaceted and highly contextual domain. No robotic caregiving context is complete without specifying the four things: 1) target user, 2) the caregiver (optional), 3) the environment, and 4) the robot. Without knowing the target user of this technology, it is unreasonable to assume that the user is able to move their arms during the activity, which affects the POMDP formulation, choice of action space, as well as the co-optimization framework that induces reward seeking behavior from the humans. If a patient could move their arms, can the paper justify why they would need robotic assistance with itch scratching. Without knowing the environment and the robot, it is unreasonable to assume that partner interactions themselves would be able to capture the differences between training time and test time for generalization. The paper needs to clarify the setting - if it is a robot-arm mounted on a wheelchair, are they considering generalization to a different user who is lying on a hospital-bed for itching? The assumptions need to be very clearly specified in the paper and the claims should be carefully considered and re-worded.\n\n- There is a lack of real-world experimentation for these methods, which can provide more insights on the practicality of these approaches for the robotic caregiving context. Is this a realistic assumption that during training time the privileged information would be available for robotic caregiving tasks? Can the paper discuss how much data would be needed to train an encoder that takes in interaction data to generate latent space, and if that scale of robotic caregiving data collection is feasible in the real world? How would these methods translate beyond simulation?\n\n- The paper can also be stronger by clarifying what personalization means in the context of learned embeddings (Section 3). Personalization in HRI and in the robotic caregiving world can have many meanings and it would make the paper clearer, if it can explain clearly in what context this is used.\n\n- For itch scratching, can the paper justify why the action space only considers joint and end-effector positions? Without contact forces, how would itch scratching be possible to generalize between different users?\n\n- How would this framework generalize when there is a human caregiver present to help the robot with the care recipient (two humans - one robot)? This is a very probable scenario for robotic caregiving (in fact more probable than just a dyadic robot-user scenario presented in this paper) and a discussion on extending the framework from a two-player Dec-POMDP would make the paper stronger.\n\n- The regularization term (eq. 3) is motivated by VAE formulation and helps in grouping similar humans closer, assuming a normal distribution. By similar humans, the paper probably means similar itch locations and history of actions ( a clarification would be helpful). Have the authors considered other formulations that directly attempt to use VAE structures for sequential data such as Variational RNNs ( https://proceedings.neurips.cc/paper/2015/hash/b618c3210e934362ac261db280128c22-Abstract.html )? A discussion on the modeling choice would be helpful.\n\n- Can the paper clarify how behavior cloning is used to stabilize their policy training? How is covariate shift dealt with in this case?\n\n- The paper should be carefully revised for typos, punctuation, other formatting etc. Some examples: \n     - Line 36: comma after when they are not\n     - k_H should be defined in Section 4.1\n     - Fig. 2 can use a more detailed caption.\n     - Is the avatar in the assistive gym a female avatar? Kindly clarify the choice of pronouns (Line 84)\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper in its current state needs realistic evaluations and justification of the choice of methods. \n\n##########################\nAfter rebuttal period: \n##########################\n\nI thank the authors for engaging in thorough discussions with me and responding to many of my comments. I really appreciate it. I have read the rebuttal in detail and I am revising my decision from \"Strong Reject\" to \"Weak Reject\". While I am still not fully convinced that the paper is showing evidence of what it is claiming which is to learn representations that enable generalization in assistive tasks, the paper has improved through these revisions. I have updated my scores accordingly. A side note: I understand that recruiting people with disabilities is indeed hard, but one thing to consider in the future would be to engage the stakeholders (occupational therapists, caregivers, care recipients etc.) even through some remote zoom discussions, atleast to confirm that the formalization and the experimental design is real-user inspired. Also, while experimenting with different activity levels is great, the mobility limitations of end users do not vary globally but are instead a function of relative local joint constraints which can be characterized by their AROM and PROM. Thank you for your work. \n", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "See the “Suggestions for Improvement” section above.\n", "reviewer_expertise": "5: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature"}, "extra_text": {}}, {"review_id": "jSCSbTWESrG", "reviewer": "Reviewer_QYB2", "summary": "This paper tackles the problem of generalizing to out-of-distribution human behavior when using sim2real methods to develop assistive robotics capabilities. To that end, the authors formulate what they call the 'Assistive Personalization Problem': an assistive problem setting focused on generalization to unseen human behavior and preferences. To train policies for this problem setting, the authors propose the Prediction-based Assistive Latent eMbedding (PALM) framework, which involves learning a latent space by predicting future human actions from a history of observations and actions. Empirical results show that this approach outperforms state-of-the-art sim2real approaches used in non-assistive tasks and human-robot interaction approaches that train representations by predicting observed states or rewards. Additionally, the authors include a mechanism to adapt the learned latent space at test time and show empirical results that this helps with generalization.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- In general, I found the paper to be well written, easy to follow, and enjoyable to read\n- The problem setting is well motivated and clearly described\n- The proposed method of learning a latent space (including test-time optimization) is a natural and elegant approach to the problem\n- The experiments and ablations are sensibly designed\n- The empirical results presented in the paper are promising\n\n**Scope for Improvement**\n- The tasks used to evaluate the approach are relatively simple (reaching and itch scratching). I think the contribution could be stronger if the authors included additional, potentially more complex, assistive tasks in their empirical analysis (e.g. dressing, bathing, feeding, drinking). \n- Sim2Real is used to motivate the work, however the work is done entirely in simulation with synthesized human behavior. It would strengthen the submission if the authors included some evaluation with realistic human preferences and behavior.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I am recommending to accept the paper because it presents a novel problem formulation that is interesting and well motivated. Additionally, the proposed learning-based approach to the problem is sensible and elegant -- and the empirical results presented in the paper are strong.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "To improve the submission, I would suggest including another couple of assistive tasks to your empirical results or adding some discussion that makes a strong argument for why the experiments presented are expected to generalize to other tasks.\n\nBecause sim2real transfer is not really addressed in your experiments, you might also consider reframing the paper to have less emphasis on sim2real in the motivation.  Or you might consider including additional discussion to help readers conceptually bridge the gap between this work and the idea of sim2real transfer that motivates it."}}, {"review_id": "hm0nDD0pipk", "reviewer": "Reviewer_tdQt", "summary": "This paper addresses the problem of sim2real in the context of assistive tasks, in which there is an agent that is acting and requiring assistance.  If the diversity of behaviors is not properly captured, trained agents may encounter out-of-distribution policies at test time.  The authors posit that the robot will have access to interaction history at test time, which motivates learning a “good” latent parameterization that generalizes to humans and which are adaptable with test-time interaction data.  The authors offer empirical evidence to support that the latent representation may be interpreted and leveraged using human policies trained from a policy distribution, and that if out-of-distribution humans are encountered, the parameterization can be adapted accordingly.  The supplement and video also provides further experimental studies, including physical human-robot experiments.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Better computational approaches to solving the problem of human-robot cooperation and assistance is important for the robotics community in general.  In many studies, the human is assumed to be an adversary, where the robot must exploit the learned human strategy.  In this paper, the authors provide a solution where robot must assist the human in assistive HCI settings, such as itch scratching.  The high-level problem statement - a latent representation amenable for interaction, and which can be adapted to fine-tune the latent embedding according to any distribution shifts seen at test time - is a very important and impactful problem.\n\nThe paper is well-written, and comprehensive.  The authors provide reasonable context, and demonstrate, through compelling examples, the PALM approach, which illustrates the overall utility and provide a basis for comparison with other methods.  The comparisons between other approaches (RMA, LILI, RILI) strengthen the paper’s arguments.  The authors also provide (in the supplement and video) a real user study to further demonstrate their approach.\n\nOne of the weaknesses of this paper is that the link between how the proposed computational approach addresses generalizability is not clear.  After reading the paper several times, I could not figure out what aspects of the approach allow for a latent structure that is amenable to assistance.  Does the human have to behave optimally and rationally for the approach to work?  Relatedly, the term “generalizability” seems to be a misnomer in the context used in the paper.  If $D_{train}$ and $D_{test}$ are sufficiently different (by way of counterexample, suppose that $D_{test}$ leads to adversarial policies), then the learned structure may be completely useless at test time.  \n\nAlso, a complete study of the generalizability of the approach was never fully explored, probably since it is not required for the assistance domain, since it is assumed that humans are mostly cooperative.  Could the authors adopt a more specific definition for $D_{train}$ and $D_{test}$, or provide assumptions on the nature of these distributions or the nature of the human behaviors, to make the assistance and generalization claims clearer?\n\nRelated to the points above, in the Experiments section, it is never explained why the results of the other methods (LILI, RILI, RMA) do not generalize well, yet PALM does.  Particularly, the results for the reacher domain show that the circular latent parameterization correctly emerges for PALM, even without test-time optimization, though this fails to emerge in the other approaches.  From a computational standpoint, is it one of the loss components (e.g. the PPO or BC loss components), or is it differences in decoder output or less reliance on rewards that causes PALM to predict a human better than LILI or RILI?\n\nIn Sec. 3.2, why does the adaptation step only update $\\theta$ (and not also $\\phi$)?  Clearly, the decoder was trained on $D_{train}$, and will suffer the same domain-shift issues as the encoder when presented with $D_{test}$?\n\nAnother significant weakness is that, in Section 3.2, the presented adaptation scheme is not at all convincing for solving the objective in eq. (2).  How does the approach deal with catastrophic forgetting of the original latent space?  How many iterations does there need to be for the test distribution to be accurately captured?  Since the solution to (2) cannot really be achieved (since there are no proofs to this effect), then this is entirely a heuristic / empirical claim.  What are the hyperparameters (e.g. $\\delta$), and how are they selected? \n\nIn the Assistive Itch Scratching domain, the co-optimization of $\\pi_R$ and $\\pi_H$ seems limiting to implement in practice, since there are no ground truth rewards and humans could behave unlike eq. (5).  How would the approach work if the ground truth human policy $\\pi_H$ was not co-trained? \n\nThere are various formatting issues.  For instance, Figure 5 (on p. 8) is dislocated from where it is first referenced (p. 6).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The proposed approach for learning the right representation for human-robot interaction in assistance settings is an important and sometimes overlooked area.  While this paper does not seem to propose a novel learning paradigm, it does focus on setting up the problem such that a coherent latent representation can be learned, and seems to generalize, to an extent, other similar approaches (LILI, RILI).  I believe that the proposed adaptation scheme does not solve the overall objective in eq. (2), and should be re-written.  However, I also feel that the paper carries strong empirical evidence pointing to promise of using such approaches in realistic settings.  My recommendation leans toward acceptance.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "1. The authors are strongly encouraged to add a description of why other approaches (LILI and RILI) are not able to handle the assistive domain, and what aspects of the proposed approach lend to better identification of a \"more correct\" latent representation than these other approaches.  \n - Does this relate at all to any unstated definitions or assumptions for $D_{train}$ and $D_{test}$?  For instance, are there limitations to the domain shift from $D_{train}$ to $D_{test}$ or can the shift be arbitrarily large?\n - The authors should also highlight any such differences in the experimental comparisons of Sec. 4.\n\n2. There are missing details in Sec. 3.2:\n  - Please explain why $\\phi$ is not included in the test-time adaptation update. \n  - Please explain how catastrophic forgetting is dealt with in the adaptation procedure.\n  - Also, how does this factor into the selection of the hyperparameters such that the distribution shift to the test distribution is properly handled by the adaptive procedure, yet does not destroy the ability to assist within the original training distribution.\n\n3. The authors should explain whether their approach in general can handle non-rational or suboptimal human participants, or whether any extensions are necessary.\n\n4. Please fix the formatting issues.  There are not many, but one example is Figure 5, which is dislocated from where it is first referenced."}}]

Author rebuttal:

Decision: Accept (poster)