First Contact: Unsupervised Human-Machine Co-Adaptation via Mutual Information Maximization
Authors: Siddharth Reddy
Sergey Levine
Anca Dragan
Venue: NeurIPS 2022 Accept (2022)
Keywords: reinforcement learning
human-computer interaction

Abstract: How can we train an assistive human-machine interface (e.g., an electromyography-based limb prosthesis) to translate a user's raw command signals into the actions of a robot or computer when there is no prior mapping, we cannot ask the user for supervision in the form of action labels or reward feedback, and we do not have prior knowledge of the tasks the user is trying to accomplish? The key idea in this paper is that, regardless of the task, when an interface is more intuitive, the user's commands are less noisy. We formalize this idea as a completely unsupervised objective for optimizing interfaces: the mutual information between the user's command signals and the induced state transitions in the environment. To evaluate whether this mutual information score can distinguish between effective and ineffective interfaces, we conduct a large-scale observational study on 540K examples of users operating various keyboard and eye gaze interfaces for typing, controlling simulated robots, and playing video games. The results show that our mutual information scores are predictive of the ground-truth task completion metrics in a variety of domains, with an average Spearman's rank correlation of 0.43. In addition to offline evaluation of existing interfaces, we use our unsupervised objective to learn an interface from scratch: we randomly initialize the interface, have the user attempt to perform their desired tasks using the interface, measure the mutual information score, and update the interface to maximize mutual information through reinforcement learning. We evaluate our method through a small-scale user study with 12 participants who perform a 2D cursor control task using a perturbed mouse, and an experiment with one expert user playing the Lunar Lander game using hand gestures captured by a webcam. The results show that we can learn an interface from scratch, without any user supervision or prior knowledge of tasks, with less than 30 minutes of human-in-the-loop training.

First Contact: Unsupervised Human-Machine Co-Adaptation via Mutual Information Maximization
Siddharth Reddy, Sergey Levine, Anca D. Dragan
University of California, Berkeley
(sgr,svleving, anca] ancaleeereeeey.eeo
Abstract
How can train assistive human- machane interface (e.g., an electromyograppy- based limb prosthesis) to translate a user's 's raw command signals into the actions of robot computer when there is prior mapping, cannot ask the user for supervision in the form of action labels or reward feedback, and we do not have prior knowledge of the tasks the user is trying accomplish? The key idea in this paper is that, regardless of the task, when an interface is more intuitive, the user's commands are less noisy. We formalize this idea as completely unsupervised objective for optimizing interfaces: the mutual information between the user's command signals and the induced state transitions the environment. To evaluate whether this mutual information score can distinguish between effective and ineffective interfaces, we conduct a large-scale observational study on 540K examples of users operating various keyboard and eye gaze interfaces for typing, controlling simulated robots, and playing video games. The results show that our mutual information scores are predictive of the ground-truth task completion metrics in variety of domains, with an average Spearman's 's rank correlation of p 0.43. In addition to offline evaluation of existing interfaces, we use our unsupervised objective to learn an interface from scratch: we randomly initialize the interface, have the user attempt to perform their desired tasks using the interface, measure the mutual information score, and update the interface to maximize mutual information through reinforcement learning. We evaluate a small-scale user study with 12 participants who perform 2D cursor control task using perturbed mouse, and an experiment with expert user playing the Lunar Lander game using hand gestures captured by a webcam. The results show that learn interface from scratch, without any user supervision prior knowledge of tasks, with less than 30 minutes of human-in-ther -oop training.
method through
1 Introduction
Imagine communicating with an intelligent extraterrestrial for the first time. They are trying to get us to do something (c.g., synthesize novel molecule and build space elevator out of it), but we do not know what that task is, and we cannot understand their language. They might speak, write, wave their limbs at us in a video, or send us messages through some other modality. They might even be watching how we act on their messages, and adapting the content of their messages to our behavior. How do translate their messages into the actions they want to take
To make progress on this problem in the absence of (known) alien signals, we study the related problem of helping humans communicate their intent to machines via arbitrary command signals; for example, controlling robotic arm through a brain-computer interface [1. When designing
interfaces, we typically either (a) use human ingenuity to design an interface that is
Code, data, and vidcos available at atlttpp::/sstee..ge http::/sssieeeegggee.eoe hww.g//....oo//eee////aaappattaa
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

P.Ni Rooi))
Tasdf is intuitive for the user
User's command HoMH
Interface's action T(a; Scc X²)
Environment state
Tasdf "a"
St
X
a
S1+1
"baman" St
X
a
"banana"
S
pi Rutae)
I(s1, x1)
I(Xe, 8141/Ee
Ta unintuitive
"
[(s) + II
Ad "I"
T(xi, (5t, i]]
"banan"
"bananf"
(a)
(b)
(a) The user is trying to perform a task (c.g., land quadrotor), which the interface cannot directly observe. The user communicates with the interface æ by observing the environment state then providing command signal x, (e.g,, the audio of voice command). The interface æ observes the state s; and command x,, then takes an action a, that causes the next state 5t+1- We assume that the user partially adapts their command x, to any given interface т, and that the extent of this user adaptation depends on the intuitiveness of the interface. We search for an interface that maximizes the rate of information transfer from user to the environment (orange) and from the environment the user (blue), which formalize the mutual information Ixx,, (8t, St-1)). (b) The more intuitive the interface (e.g.. a QWERTY keyboard where the 'a', 's', 'd', and 'f' keys are arranged in that order), the less noisy the user's commands (c.g., keypresses). The less intuitive the interface (c.g., randomly- randomly-permuted permuted keyboard where "asdf" is scrambled to "dfas"), " the noisier the user.
Figure 1:
intuitive for a human to operate; (b) get direct supervision on how commands should map to actions, so we can train the interface through supervised learning; or (c) get reward feedback on the system's performance, so that we can improve the interface through reinforcement learning (see Sec. D for prior work). Unfortunately, none of this is possible with alien users: we do know what intuitive for them; we cannot ask them to perform specific tasks so that we can collect labeled data for supervised learning; we do not know what tasks they are trying to accomplish; and we cannot understand any feedback they might give on the quality of our actions. Existing approaches are also inconvenient for human users who would like to perform sehhly-personalizedd tasks using traditionnl interfaces. such controlling vehicle through microphone (2), but do not want to be interrupted and asked for supervision or feedback
Without any source of supervision or prior knowledge about the user's desired tasks, it seems impossible to design good interface. Recent work proposes optimizing for the user's ability to control their environment (3. This is useful, but there tend to be many different interfaces that enable the user to influence the environment, and most of them are unintuitive to operate (e.g., compare the QWERTY keyboard to a keyboard with randomly-pemmn permuted keybindings). How do we capture what intuitive, not only when the user a human, but also when the user is an alien whose prior over 'natural' interfaces may not overlap with those of human? The key idea (and assumption) in this paper: the more intuitive the interface, the less noisy the user's commands (Fig. [D). Thus, a good interface should not only maximize the user's ability to influence the environment, but also minimize the entropy of the user's 's commands. We show that this corresponds to maximizing the mutual information between the user's command and the induced state transition (Sec. 2.1
Of course, we cannot optimize this mutual information objective without interacting with the user. What we thus propose is a co-adaptation process, wherein we start with a random interface, the user learns about as they use it, we measure the mutual information, and update the interface to maximize the mutual information through reinforcement learning. As the interface improves, the user's ability to complete their desired tasks using the interface also increases. We call this method the mutual informutioonmmmimiziin interface (MIMI).
The van is under voice command,' [Ng] explains. 'I removed the steering- Betering-wheel-aee wheel-and-pedee interface because I found verbal commands more convenient. This is why will sometimes make unfamiliar sounds with my voice- am controlling the vehicle's systems...Ng makes a yapping sound, and the van pulls out onto the frontage road....gg laughs sharply, like distant ack-ack, and the van almost swerves off the Ng sings a little song. A robot arm unfolds itself from the ceiling of the van, crisply yanks the vial from her hand, swings it around, and holds it in front of video camera into the dashboard." [

---

Our primary contribution is an unsupervised mutual information objective for evaluating interfaces that can be used even when the ground -oouud-truthha reward function for the user's desired task is unknown. To evaluate whether this unsupervised objective can be used to distinguish between effective and ineffective interfaces, we analyze offline datasets of users operating various interfaces in five different domains from prior work, and measure the correlation between the ground-truth rewards for each task and our unsupervised mutual information scores (Sec. S. The results show that, in each domain, the interface with the largest mutual information score also has the largest ground-truth reward (Fig. D in the appendix). We also contribute the MIMI algorithm for training an interface to understand human commands from scratch, without prior knowledge of the user's desired tasks or explicit feedback from the user. We evaluate MIMI through an online user study with 12 participants who use a perturbed mouse to perform a 2D cursor control task (Sec. 5.2) The results show that, in under 30 minutes of numan-in-the-logy training, MIMI learns an interface that is intuitive for users to operate and enables users to reach their goals more quickly than random interface. We also showease MIMI's ability to scale to more complex tasks and command modalities through a study with one expert user (the first author) who uses hand gestures to play the Lunar Lander game (Sec. 5.3))
2 An Unsupervised Mutual Information Objective for Evaluating Interfaces
In our problem setting, users cannot directly act in the environment, and must rely on interface to take actions for them (e.g., due to a motor impairment, or because they are remotely operating space robots, because they are aliens communicating with humans over long distances). The interface does not have direct access to the user's desired task, but can observe command signals that the user provides to the interface (e.g., through a webcam or microphone). We formulate the command-to- action translation problem as contextual Markov decision process (CMDP) 4 Each observation decomposed into two variables: the environment state S, (e.g., the position and orientation of robot arm) and the user's command signal X₃ (e.g., webcam image of their eye gaze). The interface (a (s,, xt) takes an action a, given the state s, and command X₃. We do not assume access to the user's s desired task (which represents the 'context' in the CMDP), the space of desired tasks, the ground-truth reward function R(s,, az), or the state transition dynamics. We approach this problem by defining a surrogate reward function that correlates positively with the ground-truth reward. We define this surrogate to be the information rate of the interface, which we formalize as the mutual information between the user's command x; and the induced state transition (St, Sr+1.. Fig. outlines our method.
2.1 Characterizing the Information Rate of the Interface
Viewing our system through the lens of information theory 5 d. the user interacts with the en- vironment through a noisy communication channel that mediates perception (blue in Fig. D and control (orange in Fig. I When the environment sends message s; to the user, the user's internal decision -aaiing process noisily converts the message S; into a command x,. When the user sends a message x; the interface, the interface converts the message x, into an action a, I Taaa|s, (ar[st,,)) and the environment converts the action a, into a next state S141 ~ p(s (st, a,) following unknown dynamics. In this section, we derive the information rate of this channel, and show how maximizing it enables the user to perform their desired tasks.
The amount of information that is transmitted from the user to the environment can be characterized as the conditional mutual information I(x,, Se+1 (s:) (orange in Fig. D. Maximizing this term enables the user to influence the next state 8 by modulating their command x,- However, there can be many different interfaces that achieve an cqually-high value of Z(x,, S (s:). For example, consider typing QWERTY keyboard. Randomly permuting the keybindings would preserve the value I(x,, )s since there would still be a one-to-one mapping between physical keys and typed characters, but the resulting layout would probably not be intuitive to use QWERTY. In order to maximize the intuitiveness of the interface, we need to consider the amount of information that gets transmitted in the other direction: from the environment the user.
The amount of information that gets transmitted from the environment to the user can be characterized as the mutual information I(s:, x,) (blue in Fig. I. The main assumption in our method is that maximizing I(s,, xt), which can be rewritten as H(x;) - H(x; (s;), leads to an intuitive interface. The idea behind penalizing the conditional entropy H (x{((s)) is that, when the user operates an
3

---

Algorithm MIMI-EVALUATEE(T
1: D
2: while [D|< do
3:
X₁ ~ >>pp(X]|s;i))
user gives command that is partially adapted to interface b interface takes action S(+1 p(s |s,, a²) environment transitions to next state (following unknown dynamics) D + D U {(s,, X,, St+1))
4:
a, ~ π
5:
6:
7: Split D into training set rrain and validation set vval
8:
,²,4"" +- arg max 4.4 TUURA (Ø, 10; Duinn)
D optimize MI lower bound in Equ.
2
9: Return /TUBA (Ø)" ;1;;; Deal)
return optimized MI lower bound evaluated on validation set
unintuitive interface. they provide noisier commands Xr given the current state S, (similar to the maximum causal entropy model of noisily-rational user behavior 77. This is because an unintuitive interface can confuse users, cause them to provide random commands when they are under time pressure to perform tasks, or cause them to keep trying different commands to see which ones will trigger their desired actions. On the other hand, an intuitive interface makes it easy for the user to determine which command will induce their desired next state, leading more deterministic user behavior (Fig. [p). Penalizing the conditional entropy H (x,(ss) alone is not enough, however. For example, consider an interface that minimizes the conditional entropy to zero by frustrating the user and causing them to always provide the same command, constant 'no op' signal). Hence, we must also maximize the marginal entropy Hxx)), which leads to an interface that encourages the user to provide wide distribution of commands overall (e.g.. by enabling the user to visit variety of states).
matter the state or desired task (e.g.,
Summing the two mutual information terms, we get the total amount of information that the user sends and receives, which simplifies to the mutual information between the user's command x; and the induced state transition (S2, $141),
R(x) & I(st, xt) I(x;, s1+1
= (H((x)) -H(xx(s))) (H((x)ss)) (((x:(s.
H(xx) - (x(x//se Sc+1)
I(xi, (Sc,
(1)
2.2 Estimating Mutual Information Rewards
Computing the mutual information reward R(x) exactly would require taking an expectation with respect to the state transition dynamics of the environment as well the user's policy for giving commands, both of which are unknown. Hence, we estimate the mutual information by collecting samples (st, X1, S141) of the user operating the interface Alg. I outlines this sample- based evaluation of R()).
To estimate Z(x,, (St, ((41)) from the samples in a dataset D in our experiments, we use the TUBA estimator 9. In particular, we fit "statistics network" TG and variational parameters ag to maximize the mutual information lower bound,
(To(..0.811))) (0s(01,1112) + ag((s - (2)
Σ TrUBA(&, (5,
81, EXAD
where x is sampled uniformly at random from the dataset D. Intuitively, T3 is discriminator that learns to distinguish between realistic pairs (Xe, (St, 8141) and unrealistic pairs (x, (St, 8t+1)). In experiments, represent T's and ag each as a feedforward neural network that outputs scalar.
Two practical issues emerge when applying the standard TUBA estimator to our problem setting. First, we can only collect a small amount of data ' to evaluate a given interface during our user studies, which can lead to high-variance estimates the mutual information. To address this issue, we adopt the following approach from prior work on data-efficient mutual information estimation IC:: instead of using the final training loss JTUBa as our mutual information estimate, we use the validation loss (line 9 in Alg. D] Second, we must avoid substantial wall-clock delays to the user

---

while we optimize the mutual information lower bound in line 8 of Alg. Typically, in order to accurately estimate the mutual information, one must optimize the lower bound to convergence. This can take many steps of stochastic gradient descent, and blocks the user from proceeding to the next episode. In our setting, however, we find that optimizing to convergence unnecessary. The key idea is that, since we ultimately use the mutual information estimates as rewards to compare interfaces, only the relative values of the estimates matter to us. Hence, we only take IK gradient steps (with a small batch size of 64) to fit the estimator in all our experiments. We find that, even after only IK steps, the mutual information estimates already tend to correlate positively with the ground-truth task rewards (Fig. 2
Learning an Interface through Mutual Information Maximization
In principle, one could represent the interface a in any manner (e.g., as a deep neural network) and any reinforcement learning algorithm to optimize MIMI-E (AMLATEALLATT(TT.... In our experiments, parameterize the interface as a linear model My. where are the weights and biases, and maximize MIMI-EVALUATE(G using an off-the-shelf Bayesian optimization algorithm (scikit-optimizn based on Gaussian process regression [2] Initially, we randomly sample the interface parame- ters 8g have the user attempt perform their desired tasks using the interface Té, for 10 episodes, and compute the mutual information reward HLIMI-EVALAATE(TTA
Given all pairs of interface parame- ters and corresponding mutual information rewards ((0,, MIMI-EVALUATI (T.. }) evaluated so far, fit Gaussian process regression model that predicts the mutual information reward given the interface parameters, and select the next interface 0p+1 by optimizing one of three acquisition functions expected improvement, lower confidence bound, or probability of improvement-chhee uniformly at random. Details in Appendix A.2
4 Related Work
Developing an assistive human-machine interface from scratch requires translating the unknown language of commands into actions. Translating unknown languages is a broad area of research, encompassing linguistics [13, cryptanalysis [14,, and deep neural network interpretability [IS [6 17]. Prior work on multi-agent reinforcement learning proposes inductive biases that use mutual information maximization promote positive signalling and listening in cooperative tasks [ 19] This prior work studies the emergence of communication in autonomous agents, whereas MIMI is intended for the unsupervised learning of co-adaptive user interfaces. Our problem setting also differs in that we cannot train the speaker agent (i.e., the human), so we must craft an objective for the listener agent (i.e., the interface) that indirectly promotes informative signaling from the speaker.
The literature on adaptive interfaces spans multiple fields, including brain-computer interfaces 23 speech interfaces
20 21
24 25 26 natural language interfaces 27
29
30
electronic
musical instruments B1I, and robotic telcoperation interfaces
32
35
36
In particular,
there is substantial prior work on camann-machne co-adaptation
77
3
39
40
42
43). In
contrast to this prior work, MIMI does not require knowledge of the user's s desired tasks or direct
supervision. 44 45 propose unsupervised co-adaptation method that uses principal component analysis to fit an interface that maps commands to actions, but requires that the interface be a linear mapping, and it does not necessarily maximize the intuitiveness of the interface.
Prior work has trained reinforcement learning agents to optimize implicit user feedback contained in EEG signals 46) peripheral pulse measurements 47 facial expressions [48] 49 and clicks [50)] These approaches typically assume access to dataset of user observations (c.g.. facial expressions) and explicit rewards (c.g., labels that indicate positive or negative emotion) in order to train a reward model that infers rewards from user observations. Other prior work on instan----tth--oog RL assumes MIMI, in contrast,
direct access to explicit user feedback during the RL phase [5 32 52 does not require access to explicit rewards at any time.
Mutual information maximization has previously been used to acquire diverse, complex behaviors through reinforcement learning the absence of extrinsic rewards 54 55
56
57
58
59
600
These prior methods aim to train autonomous agent, whereas MIMI aims to train user interface. Furthermore, MIMI differs in that it learns from a user who adaptively provides commands to the interface that try to induce the user's s desired next states, whereas in these prior methods, the 'command
5

---

signal' is a latent code that is randomly sampled from a fixed distribution. As a consequence, the latent skill space discovered by these prior methods is not necessarily intuitive for a user to control, or even human-interpretabbe
Our mutual information objective in Eqn. is related empowerment 61,, which measures the channel capacity between an agent's actions and the environment state. Empowerment has previously been used as an intrinsic motivation for autonomous reinforcement learning agents [62]. and an
auxiliary objective for assistive agents that preserve a user's ability to reach various states [ Our objective differs in that it measures not only the user's ability influence state transitions, but also the intuitiveness of the interface.
Our implicit model of the user's noisy rationality in Sec. 2.1 is similar to the widely-used maximum causal entropy model of user behavior 77]. but departs from it in that we assume the user partially adapts their behavior to match the interface, and that the extent of this user adaptation depends on the intuitiveness of the interface.
Experimental Evaluation
Our experiments focus on measuring MIMI's effectiveness at evaluating interfaces, learning interfaces from scratch, and scaling interface optimization to complex tasks and interface modalities where there is no clear, intuitive solution. We aim to answer the following questions. Q1 (Sec. 1. Can MIMI's mutual information score distinguish between effective and ineffective interfaces for variety of users, interface modalities, environments, and tasks? Q2 (Sec. 5:2): Can we learn an interface from scratch in completely unsupervised manner by maximizing mutual information rewards with MIMI? Q3 (Sec. 5.3) Does unsupervised human-in-the-lod complex interface modalities and tasks? To answer Q1, conduct an observational study on five datasets from prior work. To answer Q2, we conduct a user study with 12 participants who use perturbed mouse perform a 2D cursor control task. To answer Q3, we study how MIMI can be used to enable an expert user (the first author) to play the Lunar Lander game through hand gestures captured by a webcam.
reinforcement learning scale to more
5.1 Offline Evaluation of Existing Interfaces
In this experiment, we aim to evaluate whether the mutual information score in MIMI can distinguish between effective and ineffective interfaces for a wide variety of users, interface modalities, environ- ments, and tasks (Q1). We take data from prior work on adaptive interfaces in which the ground-truth rewards were measured, and check whether MIMI's unsupervised evaluation of those interfaces correlates with the true reward that users received when performing tasks via those interfaces. We examine data from four prior works: X2T 63,, ASHA 64 shared autonomy via deep reinforcement learning (SAwDRL) 65 and internal-to- -eerrallt--rall dynamics transfer (ISQL) [66] In each of these prior works, users were asked to operate at least two different interfaces: a non-adaptive baseline interface, and adaptive interface that learns to assist users over time. In the X2T experiments, participants used their eye gaze (a 728-dimensional command signal x, from their webcam) to select 1 of 8 buttons screen in order type phrase. In the ASHA experiments, participants used their eye gaze to control a 7-DoF simulated robotic arm to either flip a light switch or reach for a bottle. In the SAvDRL and ISQL experiments, participants used the directional keys a keyboard to play the Lunar Lander game 67). The appendix describes each domain in detail. The data from these prior user studies totals over 540K timesteps.
For each domain, each user, and each experimental condition (c.g., non-adaptive vs. adaptive interface), measure our mutual information score by averaging over 10 different random seeds used to fit the estimator in Eqn. The ground-truth reward function for each domain defined in Appendix A.3 The results in Fig. show that, in out of 5 domains, MIMI's mutual information score for an interface is predictive of the ground-trui -ruth task reward a user is able to obtain using that interface, with an average Spearman's 's rank correlation coefficient of p 0.43. Furthermore, in each domain, the interface with the largest mutual information score also has the largest ground-trute reward (Fig. 5 in the appendix).
In analyzing the data from the SAvDRL experiments, we initially encountered an unexpected result: strong negative correlation between our mutual information scores and the ground-truth rewards

---

Typing with Eye Gaze
Robotic Centrol with Eye Gaze
Lunar Lander with Keyboard
X2T (p 0.45)
ASIM Switch ip 0.50)
Internal to Real Dynomics Trandfer (o 0.111
2.4
Non Aduptiss (After XX1)
(b)
0.50
With Caplict
Salka Pummnn
1.3
Kenohdaption
0.35
0.45
0.40-
0.30
0.25
0.30-
Natoddapltee
(a)
5.28
0.25
(c)
0.35
0.6
-55
-1.3
-23
-20
-15
-33
0.
True Reward (Class Ficatio Accumeo))
True Reward
Roward
Shared
Shared Autonomy via Deep RL Ip 0.43)
ASHA Bottle (p 0.69)
7ht. 1%, Ri-23 vs. True Reward
2.4
Human
Coploit
Aan-Adaptive
0.2
0.20
5%% Diyanl
0.15
-12-
-2A
(d)
0.05
(e)
02
(f)
104
3.0
0.024
0.006
6.608
0.014
Time Offeat a
Thu Reward
Lunse Lander with boybourd
Lunae Lander with Keybward
Rebotic Control with Eye Geze
Figure 2: In 4 out of domains, our mutual information score (y-axis) is predictive of the ground-truth reward (x-axis) that a user is able to obtain with a given interface. (a) In X2T, ground-trun rewards correspond to the classification accuracy of the interface, which predicts the user's desired button (1 of 8) given the user's eye gaze signals. There were three conditions in the X2T experiments: the user first operated the non adaativee baseline interface, then operated the adaptive X2T interface, then returned to operating the non-adaptive baseline interface. There were 12 participants the user study, yielding a total 12 = 36 data points in the scatter plot (one for cach user in each condition). Spearman's 's rank correlation coefficient = 0.45 hetween the ground-truth rewards and the estimated mutual information scores. (b) Ground-truth rewards penalize distance from the robot end effector to the position of the target switch. (c) In the internal-to-reat dynamics transfer data, there were only two conditions: the user playing the Lunar Lander game on their own, and with assistance. (d) the SAvDRL data, the rank correlation between the ground-truth reward and the mutual information score depends on the time offset A in the generalized mutual information objective I(x,, (st, St+a)) )). (e) Following the results in (d), we rewards penalize crashing, and give bonus for landing between the flags. (f) Ground-truth rewards give a bonus for opening the door in front of the
set the time offset A to the maximum episode length of 10 Ground-
desired bottle and for reaching the desired bottle. As in (e), we set the time offset to the maximum episode length.
The reason is that, in the SAvDRL experiments, the assistant tends help the user play the Lunar Lander game by preventing them from crashing, which necessarily involves ignoring large fraction of the user's commands. This leads to a low mutual information between the user's command X; and the one-step state transition (s,, St+1). However, even though the assistant appears to reduce the user's influence over the next state, it actually increases the user's influence over later states the episode by preventing the user from crashing immediately. This leads us to propose a generalized mutual information objective, I xx, (8,, S))))) in which the time offset A is a hyperparameter than can be set to a value other than A = 1.
(Fig. f)).
Fig. 2 shows how modifying the time offset A in the generalized mutual information objective I(x;, (St, St+a)) )) affects the correlation between the mutual information scores and the ground-truth rewards in the SAvDRL experiments: for low values of A (including the default value of A 1), there is negative correlation, whereas for sufficiently high values of A (i.e., the maximum episode length of 10 steps), there is positive correlation. In Figures 2 and 2 we set A to the maximum episode length, so that the MIMI objective Z(x,, (St, s) always measures mutual information with respect to the final state sT of each episode. In general, choosing the value of A requires prior knowledge of the timescale of the user's desired influence over the system (i.e., does the user's command indicate what should happen at the next timestep, or what should happen by the end of the episode?). This is the primary limitation on the generality of our method.
7

---

User Study:: 2D Curser Contral with Perturbed Mouse
Mutual Information Reward
Tiee Roward Mutuel infermciin Rewerd up 0.81)
-3.07 (a)
(b)
-3.00
-0,00
-0.00
1.0
-0.12
-3.22
-113
Oreele HMM (Dusa)
HMM (Oura)
-314
do
250
152
200
63
185
15a
300
255
Namber of Online Training to sertes
Namber of Online Traiking Spledde
Tranioaa (Pg. Derace toragee
Emergent interfaces
<50 Training Episodes
>150 Treining Episodes
120
(d)
220 315
(8)
Figure 3: (a) Through online training. MIMI (orange) leamss an interface that substantially outperforms baseline interfaces with random parameters (gray), and approaches the performance of an oracle agent that always moves straight to target (green). (b) As expected, MIMI indeed maximizing mutual information rewards through human-in-the-liee in-the-loopop RL. (c) Each point in the scatter plot represents a distinct user operating distinct interface. The ground-truth task rewards (negative average distance to target) and the mutual information rewards are highly positively correlated (Spearman's 's rank correlation A polar histogram of the final interface parameter 0 that MIMI converges to, for each of the 12 users. Whereas the random baseline samples 0 from a uniform distribution over perturbation angles [0, 2Tr). MIMI converges to a highly non-uniform distribution over 0. In particular, MIMI tends to converge to one of two interfaces: no perturbation of the user's 's mouse (9 z 0), or inversion of the user's mouse (0 R т). (e-g) In this center-out cursor control task, the user initially tends to move in curved trajectories that wander away from the target. After 150 episodes of online training, the user tends move straight lines to the target.
0.87), as in the offline experiments in Sec. 5.1 (d)
5.2 User Study: 2D Cursor Control with Perturbed Mouse
In the previous experiment, we showed that MIMI's mutual information score correlates positively with the ground-trum task rewards in a variety of offline datasets (Q1). However, not clear from this result alone that maximizing mutual information rewards will yield an interface that also performs well in terms of the ground-truth task rewards (Q2). To answer Q2, we conduct a small-scale user study with 12 participants who use perturbed mouse perform simple 2D cursor control task. target position (Fig. b). The ground-truth reward is the average negative distance target throughout the episode. The user's s 2D mouse position command is used to control the 2D velocity of the cursor (details in Appendix A.3) The interface ιa applies rotation to the 2D mouse position to get the 2D velocity (i.e., f consists of a single parameter: the rotation angle). The initial interface To, is a random rotation, and is usually difficult for the user to operate (see scattered trajectories in Fig. B Users struggle to anticipate and adjust for the rotation, which leads to them initially moving away from the target, oscillating around the target they approach it because they keep steering in the wrong direction, or getting stuck on the boundaries of the environment because they forget how to steer away from the boundary.
The goal of the task is to drive cursor from the center of the screen to a randomly-sampi
Each of the 12 participants completed two phases of experiments: A and B. In phase A, they operate 5 interfaces with randolly-sampled parameters 0 for 10 episodes each. In phase B, they operate an adaptive interface optimized by MIMI (Sec. B Each interface proposed by MIMI is evaluated for 10 episodes, and we halt the experiment after at most 30 interfaces have been evaluated, yielding total of at most 300 episodes per participant. To avoid the confounding effect of overall user improvement or fatigue
time, counterbalance the order of phases A and B. Details in Appendix A.
The results in Fig. 3 show that, even though MIMI starts from scratch with a random interface and only aims to maximize mutual information rewards (blue), it enables users to achieve substantially higher ground-truth rewards (orange) than the random baseline interfaces (gray). Fig. 31 shows that MIMI tends to converge to one of two interfaces: moving the cursor in the same direction as the
8

---

Lunari Lander with Hand Gestures
Random intrriis (Bassiio))
(ount
Sucoen Dale
Stat spieds
Right thnsewe
Rigk engiee
Lest moin
2.0
54
100
150
250
Example of learned interface
X(,)
-0.23
0.97
-1
1
Xr.2
0.46
0.34
1
0.39
Xr.3
Lett mase
MaR
Main
Laft man
Xt.4]
[Main engine, Lateral thrusters]
fingertip tip, normalized to
Figure 4: MIMI can learn an effective mapping from hand pose commands to thruster actions in the Lunar Lander game. During the initial training episodes when the interface is randomly sampled, the interface can require uncomfortable hand contortions in order trigger certain actions, have very sharp threshold between poses that trigger one action intuitive interface (one episode illustrated above) that enables occasional successful landings in challenging game where constantly crashing is the default behavior. Success rates are averaged over 3 different random seeds, and smoothed using moving average with window size of 10 episodes. Hand tracking is performed using webcam and MediaPipe 68 See Fig. iin the appendix for plots of the mutual information reward. Videos available at htttttss///iteeee googse.oool.....oooooooooo.oo... www.woww/cwcceecc http://vww//caaaaaaataaaaaaaaaaa coa.aptaaaaaptnnnnnonccoca
different action. Towards the end of training. MIMI learns comfortable,
mouse, or rotating the mouse direction 180 degrees (which tends to be easy for users to understand and invert; see the appendix).
5.3 Lunar Lander with Hand Gestures
In the previous experiment, we showed that MIMI is capable of learning an interface from scratch through numan-in-the-loop RL with the mutual information objective, albeit simple task and interface modality (2D cursor control via 2D mouse commands). In this section, we demonstrate expert user (the first
that this result extends author) playing the Lunar Lander game (67 using hand gestures. To win the game, the user must land between the flags without crashing. The user's hand pose command is a 4D signal that consists of the distances from the thumb tip to each of the four other fingertips. The Lunar Lander game has 2D continuous action space that controls the main engine and lateral thrusters (details Appendix A.3). The interface Ta is linear transformation from the 4D hand pose command to the 2D thruster action space (i.c., 9 consists of 8 parameters). The initial interface Te, is random linear function, and is usually difficult for the user to operate (e.g., because it requires making difficult hand gestures, or never fires the lateral thrusters). Lunar Lander challenging game, even when played with joystick or keyboard. What makes more challenging in this setting is that, unlike 2D cursor control, where there is obvious prior mapping that moves the cursor in the direction of the mouse without any rotation, there is no obvious mapping from hand gestures to thruster actions that the user would have mind.
complex task and interface modality (Q3):
The results in Fig. show that, after 200 episodes of online training, MIMI learns an interface that enables the expert user maneuver the lander, usually avoid crashing, and occasionally land successfully. The interface enables the user to fire the right thruster by touching their index finger their thumb, fire the left thruster and main engine by touching their ring finger their thumb, and fire the main engine by slightly curling all fingers. The interface parameter values 0 learned through MIMI would have been difficult determine in advance, given uncertainty about the range of comfortable hand pose commands, and the non-obvious intuitiveness of using individual fingers for lateral thrusters vs. all fingers for the main engine. Interestingly, the final interface does not enable the user to fire the left thruster in isolation (i.e., all gestures that fire the left thruster also fire the main engine simultaneously). This turns out to okay, since the user wants to descend, and alternating between firing the left thruster and main engine together then firing the right thruster without the

---

main engine is reasonable strategy for descending without tilting. This would have been difficult to anticipate in advance, but nevertheless emerged through co-adaptation.
6 Discussion
We presented proof of concept that, through unsupervised human-in- buman-in-thh--oon in-the-loop reinforcement learning. we can learn an assistive interface from scratch in less than 30 minutes without any explicit user feedback or prior knowledge of the user's desired tasks. Our experiments show that, for a variety of users (12 participants in our user study, and 12 in each of the offline evaluations), command modalities (high-dimensional eye gaze and hand gestures, and low-dimensional keyboard commands), environments (typing, simulated robotic control, playing video game, and cursor control), and tasks (flipping light switches and reaching for bottles with the same simulated robotic arm). MIMI's mutual information objective can be used to rank interfaces without observing ground-truth rewards for the user's desired task.
6.1 Limitations
MIMI is limited by the fact that the correlation between the mutual information objective I(x,, (st, S1+2)) and the ground-truth reward function can depend on the time offset A (Sec. 5.1. Since we do not assume access to ground-truth rewards, choosing the value of the hyperparameter A requires prior knowledge of the timescale of the user's desired control over the system (i.c., whether the user wants to be able control what happens next, or what happens by the end of the episode).
The main limitation of our experiments that the largest interface only has 8 parameters, due to constraints the duration of a user study and the data efficiency of the Bayesian optimization algorithm we use for RL. One direction for future work is to instantiate MIMI with a data-efficient deep RL algorithm that scales to learning high-dimensional policies, such as REDQ 669 or policy evaluation networks 70
6.2 Future Work
Another direction for future work is evaluating MIMI real-world systems in which explicit user feedback is sparse or unavailable, such as brain-computer interfaces for users with total locked-in syndrome. MIMI could also be helpful for designing creative tools for artists and musicians where the 'task' or "ground-truhh reward function' is inherently difficult to specify, such as a tool that helps users navigate the high-dimensional latent space of generative model of images using physical gestures and virtual reality 771. an instrument that provides users with a playable, now-dimensional latent space of music audio 72 or a tool that helps designers carve out discriminable gestures from low-dimensional embeddings
73
There may be cases where the user can observe the state St, but the interface cannot (e.g., the user can look around with their eyes, but the interface not equipped with a camera), A useful extension of MIMI would extract implicit feedback from the stream of command signals x, alone, without assuming access to the states Sy.
Biological neurons maximize the mutual information between their inputs and outputs through local, unsupervised Dearning-knowe as the infomax principle 774. Ideas from recent work on greedy, self-supervised representation learning based on the infomax principle [55 could be useful for scaling MIMI to deep neural network interfaces.
7
Acknowledgments
This work was supported by Berkeley Existential Risk Initiative, NSF IIS-1651843, ARL DCIST CRA W91 Weill Neurohub, and NSF CAREER. Thanks to Natasha Jaques for giving feedback on this project during its early stages and pointing us to related work. Thanks to lauren, Quantum1000 guillefix, and the Transhumanists in VR community for many stimulating discussions about MIMI, human-computer interfaces, and multi-agent learning. Thanks to anonymous NeurlPS reviewers eRaa, phLR, and evLC for their constructive feedback.
10

---

References
[1] Katharina Muelling, Arun Venkatraman, Jean-Sebastien Valois, John Downey, Jeffrey Weiss, Shervin Javdani, Martial Hebert, Andrew B Schwartz, Jennifer L. Collinger, and J Andrew Bagnell, Autonomy infused teleoperation with application to BCI manipulation. arXiv preprint
arXiv:1503.05451, 2015.
[2] Neal Stephenson. Snow Crash. 2003.
[3] Yuqing Du, Stas Tiomkin, Emre Kiciman, Daniel Polani, Pieter Abbeel, and Anca Dragan. AvE: Assistance via empowerment. arXiv preprint arXiv:2006. 2020.
[4] Assaf Hallak, Dotan Di Castro, and Shie Mannor. Contextual Markov decision processes. arXiv preprint
2015.
[5] Claude Elwood Shannon. A mathematical theory of communication. The Bell System Technical Journal, 1948.
[6] David J C MacKay. Information theory, inference and learning algorithms. 2003.
[7] Brian Ziebart, Andrew Bagnell, and Anind Dey. Modeling interaction via the principle of maximum causal entropy. In International Conference on Machine Learning, 2010.
[8] David Agakov and Felix Barber. The IM algorithm: variational approach information maximization. Neural Information Processing Systems, 2004.
[9] Ben Poole, Sherjil Ozair, Aäron van den Oord, Alexander Alemi, and George Tucker. On variational lower bounds of mutual information. In NeurIPS Workshop on Bayesian Deep Learning, 2018
[10] Xiao Lin, Indranil Sur, Samuel A Nastase, Ajay Divakaran, Uri Hasson, and Mohamed R Amer. Data-efficient mutual information neural estimator. arXiv preprint 1rXiv:1905.03319 2019.
[11] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, Prettenhofer. R. Weiss, V. Dubourg. J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 2011.
[12] Carl Edward Rasmussen and Christopher K. Williams. Gaussian Processes for Machine Learning. 2006.
[13] John Chadwick. The decipherment of Linear B. 1967.
[14] Abraham Sinkov and Todd Feil. Elementary eryptanalysis. 2009.
[15] Jacob Andreas, Anca Dragan, and Dan Klein. 17X1v:1704.06960, 2017.
Translating neuralese. arXiv preprint
[16] Chris Olah, Arvind Satyanarayan, Johnson, Shan Carter, Ludwig Schubert, Katherine Ye, and Alexander Mordvintsev. The building blocks of interpretability. Distill, 2018.
[17] Nick Cammarata, Shan Carter, Gabriel Goh, Chris Olah, Michael Petrov, Ludwig Schu- bert, Chelsea Voss, Ben Egan, and Swee Kiat Lim. Thread: Circuits. Distill, 2020.
http::/ddistill.ub//0//iiiiii
[18] Tom Eccles, Yoram Bachrach, Guy Lever, Angeliki Lazaridou, and Thore Graepel. Bi- ases for emergent communication in multi-agent reinforcement learning. arXiv preprint arXiv: 19/2.099/000666 2019.
[19] Natasha Jaques, Angeliki Lazaridou, Edward Hughes, Caglar Gulcehre, Pedro Ortega, DJ Strouse, Joel Z Leibo, and Nando De Freitas. Social influence as intrinsic motivation for multi-agent deep reinforcement learning. In International Conference on Machine Learning, 2019.

---

[20] Vikash Gilja. Paul Nuyujukian, Cindy A Chestek, John P Cunningham, M Yu Byron, Joline M Fan, Mark M Churchland, Matthew T Kaufman, Jonathan C Kao, Stephen I Ryu, et al. A high-performance neural prosthesis enabled by control algorithm design. Nature neuroscience, 2012.
[21] Siddharth Dangi, Amy I. Orsborn, Helene G Moorman, and Jose M Carmena. Design and analysis of closed-loop decoder adaptation algorithms for brain-machine interfaces. Neural computation, 2013.
[22] Josh S Merel, Roy Fox, Tony Jebara, and Liam Paninski. multi-agent control framework for co-adaptation in brain-computer interfaces, Neural Information Processing Systems, 2013,
[23] Siddharth Dangi Suraj Gowda, Helene G Moorman, Amy L Orsborn, Kelvin So, Maryam Shanechi, and Jose M Carmena. Continuous closed-loop decoder adaptation with recursive maximum likelihood algorithm allows for rapid performance acquisition in brain-machine interfaces. Neural computation, 2014.
[24] Josh Merel, David Carlson, Liam Paninski, and John P Cunningham. Neuroprosthetic decoder training imitation learning. arXiv preprint 1rXiv:1511.044 2015.
[25] Gopala K Anumanchipalli, Josh Chartier, and Edward F Chang, Speech synthesis from neural decoding of spoken sentences. Nature, 2019.
[26] Francis R Willett, Donald T Avansino, Leigh R Hochberg, Jaimie M Henderson, and Krishna V Shenoy. high-performance brain-to-tex communication via imagined handwriting. bioRxiv, 2020.
[27] Sida I Wang, Percy Liang, and Christopher D Manning. Learning language games through interaction. arXiv preprint arXiv: 1606002444 2016.
[28] Siddharth Karamcheti, Dorsa Sadigh, and Percy Liang. Learning adaptive language interfaces through decomposition. arXiv preprint arXiv:2010.05/19, 2020.
[29] Sidney Fels and Geoffrey Hinton. Glove-Talk: A neural network interface between a data-glove and a speech synthesizer. IEEE Transactions on Neural Networks, 1993.
[30] Sidney S Fels and Geoffrey E Hinton. Glove-Talk II-a neural- interface which maps gestures to parallel formant speech synthesizer controls. IEEE Transactions Neural Networks, 1997.
[31] Andy Hunt and Marcelo M Wanderley. Mapping performer parameters to synthesis engines. Organised Sound, 2002.
[32] Patrick M Pilarski, Michael R Dawson, Thomas Degris, Farbod Fahimi, Jason P Carey, and Richard S Sutton. Online human training of a myoelectric prosthesis controller via actor-critic reinforcement learning. In IEEE International Conference on Rehabilitation Robotics, 2011.
[33] Alexander Broad, Todd David Murphey, and Brenna Dee Argall. Learning models for shared control of buman-machine systems with unknown dynamics. In Robotics: Science and Systems, 2017.
[34] Siddharth Reddy, Anca Dragan, and Sergey Levine. Shared autonomy via deep reinforcement learning. arXiv preprint 1002:::1800.017..0 2018.
[35] Charles Schaff and Matthew R Walter. Residual policy learning for shared autonomy. arXiv preprint 07X5v.2004.055097, 2020.
[36] Hong Jun Jeon, Dylan P Losey, and Dorsa Sadigh. Shared autonomy with learned latent actions. arXiv preprint arXii:2005.0032 10, 2020.
[37] Justin Sanchez, Babak Mahmoudi, Jack DiGiovanna, and Jose C Principe. Exploiting co-adaptation for the design of symbiotic neuroprosthetic assistants. Neural Networks, 2009.
12

---

[38] Paolo Gallina, Nicola Bellotto, and Massimiliano Di Luca. Progressive co-adaptation in human- machine interaction. In International Conference on Informatics in Control, Automation and Robotics, 2015.
[39] Stefanos Nikolaidis, David Hsu, and Siddhartha Srinivasa. Human-robot mutual adaptation in collaborative tasks: Models and experiments. International Journal of Robotics Research, 2017.
[40] Mathilde Couraud, Daniel Cattaert, Florent Paclet, Pierre-Yver Oudeyer, and Aymar De Rugy. Model and experiments optimize co-adaptation in simplified myoelectric control system. Journal of Neural Engineering, 2018.
[41] Stefan K Ehrlich and Gordon Cheng. Human-agent co-adaptation using error-related potentials. Journal of Neural Engineering, 2018.
[42] Stefan K Ehrlich and Gordon Cheng. A computational model of human decision making and learning for assessment of co-adaptation in neuro-adaptive human-robot interaction. In IEEE International Conference on Systems, Man and Cybernetics, 2019.
[43] Su Kyoung Kim, Elsa Andrea Kirchner, Lukas SchloBmüller, and Frank Kirchner. Errors in human-robot interactions and their effects on robot learning. Frontiers in Robotics and AI, 2020.
[44] Dalia De Santis, Patrycja Dzialecka, and Ferdinando A Mussa-Ivaldi. Unsupervised coad- aptation of an assistive interface to facilitate sensorimotor learning of redundant control. In International Conference on Biomedical Robotics and Biomechatronics, 2018.
[45] Dalia De Santis. A framework for optimizing co-adaptation in mocy-machine interfaces. Frontiers in Neurorobotics, 2021.
[46] Duo Xu, Mohit Agarwal, Faramarz Fekri, and Raghupathy Sivakumar. Accelerating reinforce- ment learning agent with EEG-based implicit human feedback. arXiv preprini arXiv:2006.16490 2020.
[47] Daniel McDuff and Ashish Kapoor. Visceral machines: Reinforcement learning with intrinsic physiological rewards. In International Conference on Learning Representations, 2019.
[48] Natasha Jaques, Jennifer McCleary, Jesse Engel, David Ha, Fred Bertsch, Rosalind Picard, and Douglas Eck. Learning via social awareness: Improving deep generative sketching model with facial feedback. arXiv preprint /802v04802..4533 2018
[49] Yuchen Cui, Qiping Zhang, Alessandro Allievi, Peter Stone, Scott Nickum, and W Bradley Knox. The empathic framework for task learning from implicit human feedback. arXiv preprint arXiv:2009. 13649, 2020.
[50] Filip Radlinski and Thorsten Joachims. Evaluating the robustness of learning from implicit feedback. arXiv preprint cs/0605036, 2006.
[51] W Bradley Knox and Peter Stone. Interactively shaping agents via human reinforcement: The TAMER framework. In International Conference Knowledge Capture, 2009.
[52] James MacGlashan, Mark K Ho, Robert Loftin, Bei Peng. David Roberts, Matthew E Taylor, and Michael L Littman. Interactive learning from policy- human feedback. arXiv preprint arXiv: (001: 1701.06049, 2017.
[53] Dorsa Sadigh, Anca Dragan, Shankar Sastry, and Sanjit Seshia. Active preference-based learning of reward functions. In Robotics: Science and Systems, 2017.
[54] Christian Daniel, Gerhard Neumann, and Jan Peters. Hierarchical relative entropy policy search. In Artificial Intelligence and Statistics, 2012.
[55] Karol Gregor, Danilo Jimenez Rezende, and Daan Wierstra. Variational intrinsic control. arXiv preprint 1rXiv:1611. 161.007777 2016.
[56] Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity all you need: Learning skills without reward function. arXiv preprint a0X.v.180..06070, 2018.
13

---

[57] David Surde-Farley, Tom Van de Wiele, Tejas Kulkarni, Catalin Ionescu, Steven Hansen, and Volodymyr Mnih. Unsupervised control through non- pan-maaaetrii discriminative rewards. arXiv preprint arXiv: arXiv:/811. 1359, 2018.
[58] Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamics- aware unsupervised discovery of skills. arXiv preprint arXiv: /9071900700.5.0 2019.
[59] Steven Hansen, Will Dabney, Andre Barreto, Tom Van de Wiele, David Warde-Fariee Farley, and Volodymyr Mnih. Fast task inference with variational intrinsic successor features, arXiv preprint urXiv: 05030, 2019.
[60] Michael Laskin, Hao Liu, Xue Bin Peng, Denis Yarats, Aravind Rajeswaran, and Pieter Abbeel. CIC: Contrastive intrinsic control for unsupervised skill discovery. arXiv preprint arXiv:2202.00161, 2022.
[61] Alexander Klyubin, Daniel Polani, and Chrystopher Nehaniv. Empowerment: A universal agent-centric measure of control. In IEEE Congress on Evolutionary Computation, 2005.
[62] Shakir Mohamed and Danilo Jimenez Rezende. Variational information maximisation for intrinsically motivated reinforcement learning. arXiv preprint
08731, 2015.
[63] Jensen Gao, Siddharth Reddy, Glen Berseth, Nicholas Hardy, Nikhilesh Natraj. Karunesh Ganguly, Anca Dragan, and Sergey Levine. X2T: Training an x-to-text typing interface with online learning from user feedback. International Conference on Learning Representations, 2021.
[64] Sean Chen, Jensen Gao, Siddharth Reddy, Glen Berseth, Anea Dragan, and Sergey Levine. ASHA: Assistive teleoperation via human-in-the-loog, reinforcement learning. arXiv preprint arXiv:2202.02465. 2022.
[65] Siddharth Reddy, Anca D. Dragan, and Sergey Levine. Shared autonomy via deep reinforcement learning. In Arxiv /802.0/744, 2018
[66] Siddharth Reddy, Anca D. Dragan, and Sergey Levine. Where do you think you're 're going?: Inferring beliefs about dynamics from behavior. Arxiv 1805.08016 2018.
[67] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. OpenAl Gym. arXiv preprint a606v: 0/000000044 2016
[68] Fan Zhang, Valentin Bazarevsky, Andrey Vakunov, Andrei Tkachenka, George Sung. Chuo-Ling Chang, and Matthias Grundmann. MediaPipe hands: On-device real-time hand tracking. arXiv preprint arXiv:2006. 200::2006.022/44 2020.
[69] Xinyue Chen, Che Wang, Zijian Zhou, and Keith Ross. Randomized ensembled double Q- learning: Learning fast without a model. arXiv preprint arXiv:2101.05982 2021.
[70] Jean Harb, Tom Schaul, Doina Precup, and Pierre-Lue Bacon. Policy evaluation networks. arXiv preprint arXiv:2002. 1833, 2020.
[71] Gray Crawford. Developing embodied familiarity with hyperphysical phenomena. 2019.
[72] Adam Roberts, Jesse Engel, Colin Raffel, Ian Simon, and Curtis Hawthomne. Music Creating a palette for musical scores with machine learning, 2018.
[73] Marius Mihai Rusu, Svenja Yvonne Schött, John H Williamson, Albrecht Schmidt, and Roderick Murray-Smith. Low-dimensional embeddings for interaction design. Advanced Intelligent Systems, 2022.
[74] Ralph Linsker. colf-organization in perceptual network.
Computer, 1988.
[75] Sindy Löwe, Peter O' CoCnnor, and Bastiaan Veeling. Putting an end to end-to-end: Gradient- isolated learning of representations. Neural Information Processing Systems, 2019.
14

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proof's of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any assets either in the supplemental material as URL? [Yes]
(d) Did you discuss whether and how consent obtained from people whose data you're 'rrre using/curating? [Yes]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [Yes]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [Yes]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [Yes]
15


Reviews:
[{"review_id": "xXJqWdOPnXw", "reviewer": "Reviewer_eRaa", "summary": "This paper aims to solve the problem of human-machine interface with supervision or prior knowledge about the user’s desired tasks. This is challenging due to the large space of interface designs. This paper proposes a reinforcement-based interface update method by maximizing the mutual information between the user’s commands and the induced state transition, where an underlying key idea is that “the more intuitive the interface, the less noisy the user’s commands.” The main contribution is the design of mutual information objective/reward for enhancing interfaces. The experiment section also shows several studies to demonstrate the effectiveness of the updated interfaces and application examples. ", "questions": "- It would be great why algorithm 1 uses a validation set for the return computation. Further, in this work, the authors can obtain a small amount of data. In general, the size of the validation set will be smaller than that of the training set. How does it affect the performance?\n\n- Figure 2 requires clarifications. First, it is hard to see the correlation between mutual information and true reward in Fig. 2(a)-(c). Although Fig. 2(e) and (f) show the Spearman correlation coefficient (i.e., 0.43 and 0.69), it is not clear how much the value is significant to say the two values of information are correlated. \n\n- Second, Fig. 2(d) shows a large number of time offsets that may result in very long time gaps between observed states. It should clarify how it affects the data collection, estimator update, etc considering the real-time human-machine interaction. \n\n- Any comparison with baselines will be helpful in understanding and also support the proposed method.  \n", "limitations": "N/A", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The main strength of this paper is that the intuitive design of mutual information rewards considering the user’s influence on the next state and its opposite. \n\nThe major weakness of this paper is there is no comparison with baselines. It is unclear whether the given dataset/problem is easy to design interface. Further, it is still hard to clearly confirm the correlation between the mutual information and true reward over results, particularly in Fig. 2. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "A-K-7CV8TEl", "reviewer": "Reviewer_evLC", "summary": "This paper presents a reinforcement learning algorithm that uses mutual information as a proxy reward for achieving understanding between a computer with a randomly perturbed UI and a human trying to accomplish a task using that UI, without knowing how it is perturbed.  The only feedback (signal) to the algorithm is the entropy of the humans movements/inputs where it is assumed that the human would be making more random/chaotic inputs in a sense proportional to how unexpectedly (unintuitively) the game’s state changed with respect to the action taken by the user.  The main evaluation is with a space ship control game where the actual motion of the ship was perturbed/offset initially by some random theta degrees.  The algorithm eventually learns to reduce theta either to zero or -180 degrees so that it exactly follows the human’s command.  Only at these two stable points does the human stop acting so randomly in response to the state changes, since the state changes are, apparently and acceptable reaction to the action (so further exploration by the human is not required).      \n\nAs the authors state: “the key idea  in this paper is that, regardless of the task, when an interface is more intuitive,\nthe user’s commands are less noisy.”  They formalize this “noise” as “the mutual information between\nthe user’s command signals and the induced state transitions in the environment.”  Their work is in line with recent literature in user empowerment where the machine does not assume or infer the users’ goals but only uses cues that indicate the user’s satisfaction with the result as a reward upon which to learn in this human-in-the-loop system.\n\n", "questions": "Out of curiosity did you consider any other metrics of noise assessment, e.g. standard deviation of responses, length of time of “attempted corrective” responses, etc. and how did these compare to mutual information?\n\nWas there anything special about you users, were they graduate students in computer science, friends, co-workers? Were they motivated in some way to complete the task?   I can imagine the “average” user simply abandoning a malfunctioning interface.  \n", "limitations": "This contribution is limited to the space of adaptive interfaces where the correct settings for the user’s task cannot be known in advance but where the parameters of the controlling algorithm have the ability to adapt themselves to a more optimal setting.  The assumptions are reasonable but not exhaustively studied.  The process of adaptation is comparatively slow compared to other methods, so a stronger motivating use case would have been better.  The method could have been more exhaustively compared versus other human-in the loop methods such as NLP or facial expression analysis or simply giving the user an explicit “performance feedback” channel on the algorithm’s performance (e.g. 1 to 10 as a keyboard input) and evaluated with respect to speed of convergence and user satisfaction (e.g. if could take longer yet users might prefer it as more natural.   \n\n", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The key strength of the paper is that it is interesting and the idea that users will express frustration with a poorly performing interface by random frantic exploration of potential alternative commands that might work seems plausible and is supported by the authors first study that shows that their mutual information pseudo-reward is correlated with interfaces that adapt to meet the users expectations (become more intuitive)  As the authors state: “the key idea  in this paper is that, regardless of the task, when an interface is more intuitive,\nthe user’s commands are less noisy.”  The authors give reasonable proof of the fact that with adaptive interfaces, users do seem to exhibit the behavior of taking more noisy actions when the resulting effects in the environment (state changes) are not what would be intuitively expected.  The authors do include a user study with a simple adaptive interface that shows that their mutual information pseudo-reward can drive convergence in this case.  The authors also show that a knowledgeable user can drive adaptation using the pseudo-reward.\n\nAs the author states there has been a number of attempts to find “natural” reward signals in the real world when interacting with humans whether these be through NLP, BCI (EEG), facial expressions and physiological signals.  None of these are as easy to obtain as the “noise” in the input that these authors leverage, making this perhaps the most useable of all prior proposed methods in the field.\n\nThe weakness of the paper is that the use case is not immediately compelling.  The first contact scenario is a good framing for this, however unlikely, but in reality, this is a more theoretical method for machines to learn preferences from humans.  The more compelling use case is probably in applying this as a feedback mechanism to search engines or to personalize the “fit” of something like exoskeletons or prosthetic limbs.  The user studies are not exhaustive, but I would not expect that for a contribution that is fundamentally about the algorithm.  It is laudable that the authors disclosed that the singular subject in the feasibility study was the first author and  while on the one hand it could be argued that having such an informed participant as an evaluator made the task “easier” for the machine (potentially the author knew to proportionally vary the randomness of the responses in a way that would facilitate convergence) it is not entirely to be dismissed especially in considering a case where a person was trying to “fit” or “train” a machine intentionally knowing how the machine learned through randomness.  The authors show that this is a naturalistic response in general, so intensifying it intentionally would not be entirely out of line with user behaviour.\n\nAlso, I believe the term “co-adaptive” in the title is a bit misleading as you do mention that the algorithm has no ability to adapt the behavior of the human.  I believe you are saying that the human naturally seems to adapt their behavior in response to the algorithms “correctness” and that you are relying on the existence of this adaptation otherwise you pseudo reward would not work, but maybe clarifying this a bit would be good.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "-AoZEgXtyuW", "reviewer": "Reviewer_phLr", "summary": "This paper presents a method for building an adaptive user interface in an unsupervised manner. The key idea is to use the mutual information between the user input and the state transition. Based on this idea, the authors propose an algorithm to maximize the mutual information lower bound from a small amount of user operation data. The proposed approach is evaluated using five existing dataset, real-world user study data, and another expert user demonstration.", "questions": "- I would like a more detailed explanation and discussion on the last point I raised above. Are there any results that suggest that the proposed method can complete the desired task more quickly or with greater user satisfaction than a scenario in which users continue to use a non-adaptive interface and become proficient with it?", "limitations": "- There is an experimental discussion on interfaces where the proposed method cannot be successfully applied, which is a good point for the paper.\n- Although the study involved numerous user experiments, there was no specific mention of a process for obtaining consent from participants or for ethical review.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strength\n- The unsupervised user adaptation task setting is interesting, and the idea of using mutual information content seems reasonable.\n- Experimental evaluations have been conducted from multiple perspectives, demonstrating at a minimum the effectiveness of the proposed method.\n\n### Weaknesses\n- While offline evaluation using existing data makes sense, evaluating the actual use of adaptively changing interfaces with this approach is not necessarily sufficient. The cursor control task seems a bit easier than the \"unknown\" interface that this study assumes. Multiple user experiments on diverse interfaces with similar or greater participants would increase the method's reliability. The author's demonstration is somewhat unreliable as experimental evidence.\n- As for the method's claim that it does not require prior knowledge about the desired task, it does not appear to be strictly substantiated, as there are no corresponding experiments.\n- Considering the purpose of the study, what should be shown is the advantage of the system compared to the user's solo adaptation to the system. However, the experiment mainly only compares the initial state and after adaptation, and there is no discussion of what happens after the participants use the baseline interface for an extended period.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["Responsible Research Practice (e.g., IRB, documentation, research ethics)"]}, "extra_text": {}}]

Author rebuttal:

Decision: Accept