X2T: Training an X-to-Text Typing Interface with Online Learning from User Feedback
Authors: Jensen Gao
Siddharth Reddy
Glen Berseth
Nicholas Hardy
Nikhilesh Natraj
Karunesh Ganguly
Anca Dragan
Sergey Levine
Venue: ICLR 2021 Poster (2021)
Keywords: reinforcement learning
human-computer interaction

Abstract: We aim to help users communicate their intent to machines using flexible, adaptive interfaces that translate arbitrary user input into desired actions. In this work, we focus on assistive typing applications in which a user cannot operate a keyboard, but can instead supply other inputs, such as webcam images that capture eye gaze or neural activity measured by a brain implant. Standard methods train a model on a fixed dataset of user inputs, then deploy a static interface that does not learn from its mistakes; in part, because extracting an error signal from user behavior can be challenging. We investigate a simple idea that would enable such interfaces to improve over time, with minimal additional effort from the user: online learning from user feedback on the accuracy of the interface's actions. In the typing domain, we leverage backspaces as feedback that the interface did not perform the desired action. We propose an algorithm called x-to-text (X2T) that trains a predictive model of this feedback signal, and uses this model to fine-tune any existing, default interface for translating user input into actions that select words or characters. We evaluate X2T through a small-scale online user study with 12 participants who type sentences by gazing at their desired words, a large-scale observational study on handwriting samples from 60 users, and a pilot study with one participant using an electrocorticography-based brain-computer interface. The results show that X2T learns to outperform a non-adaptive default interface, stimulates user co-adaptation to the interface, personalizes the interface to individual users, and can leverage offline data collected from the default interface to improve its initial performance and accelerate online learning.

Published as a conference paper at ICLR 2021
X2T: TRAINING AN X-TO-TEXT TYPING INTERFACE WITH ONLINE LEARNING FROM USER FEEDBACK
Jensen Gao, Siddharth Reddy, Glen Berseth University of California, Berkeley sgr wwweeeeeeeeeeeeeee..
Nicholas Hardy, Nikhilesh Natraj University of California, San Francisco
Karunesh Ganguly
Anca D. Dragan, Sergey Levine University of California, Berkeley
University of California, San Francisco karunesh. gangulyeucsf.
ABSTRACT
We aim to help users communicate their intent to machines using flexible, adaptive interfaces that translate arbitrary input into desired actions. In this work, we focus on assistive typing applications which a user cannot operate a keyboard, but can instead supply other inputs, such webcam images that capture eye gaze or neural activity measured by a brain implant. Standard methods train a model on a fixed dataset of user inputs, then deploy static interface that does not learn from its mistakes; in part, because extracting an error signal from user behavior can be challenging. We investigate simple idea that would enable such interfaces to improve over time, with minimal additional effort from the user: online learn- ing from user feedback on the accuracy of the interface's actions. In the typing domain, we leverage backspaces as feedback that the interface did not perform the desired action. We propose an algorithm called x-lo-texi (X2T) that trains a pre- dictive model of this feedback signal, and uses this model to fine-tune any existing, default interface for translating user input into actions that select words or charac- ters. We evaluate X2T through small-scale online user study with 12 participants who type sentences by gazing at their desired words, a large-scale observational study handwriting samples from 60 users, and pilot study with one partic- ipant using an sectrrcorticogrrahy-baaaes show that X2T learns outperform a non-adaptive default interface, stimulates user co-adaptation to the interface, personalizes the interface to individual users, and can leverage offline data collected from the default interface improve its initial performance and accelerate online learning.
brain-compune interface. The results
INTRODUCTION
Recent advances in user interfaces have enabled people with sensorimotor impairments to more effectively communicate their intent machines. For example, Ward al. (2000) enable users to type characters using an eye gaze tracker instead of a keyboard, and Willett et al. (2020) enable a paralyzed human patient to type using brain implant that records neural activity. The main challenge in building such interfaces is translating high- dige-dimeniional, continuous user input into desired actions. Standard methods typically calibrate the interface on predefined training tasks for which expert demonstrations are available, then deploy the trained interface. Unfortunately, this does not enable the interface to improve with use or adapt to distributional shift in the user inputs.
In this paper, we focus on the problem of assistive typing: helping user select words or characters without access to a keyboard, using eye gaze inputs (Ward et al., 2000); handwriting inputs (see Figure 7 in the appendix), which can be easier to provide than direct keystrokes (Willett al., 2020); or inputs from an electrocorticooraapyybbee based brain implant (Leuthardt et al., 2004; Silversmith et al., 2020). To enable any existing, default interface to continually adapt to the user, train model using online learning from user feedback. The key insight is that the user provides feedback on the interface's 's actions via backspaces, which indicate that the interface did not perform the desired

---

Published as a conference paper at ICLR 2021
User list of words and system's 's current prediction of desired word
ECoG brain-computer interface
records user's 's neural activity
U
the
Anonymized
Anonymized user
user
User input X
u
Backspace
Predicted word
π
or continuc
u
Binary feedback
(e.g.. from button press)
Reward 7
Our interface
Reward model
Default interface
Input-action-reward
π (u|x) X Pp
1 u)
log data
D + D (xx,,,))
Train reward model
Figure I: We formulate assistive typing as human-in-the-loop decision-making problem, in which inter- face observes user inputs (e.g.. neural activity measured by brain implant) and performs actions (e.g.. word selections) on behalf of the user. We treat backspace as feedback from the user that the interface performed the wrong action. By training model online predict backspaces, continually improve the interface.
action in response given input. By learning from this naturally-occurrin feedback signal instead of an explicit label, we do not require any additional effort from the user to improve the interface. Furthermore, because our method applied top of the user's default interface, our approach is complementary to other work that develops state-of-the-ard, domain -pencfi methods for problems like gaze tracking and handwriting recognition. Figure 1 describes our algorithm: we initialize our model using offline data generated by the default interface, deploy our interface as augmentation the default interface, collect online feedback, and update our model.
We formulate assistive typing an online decision-making problem, in which the interface receives observations of user inputs, performs actions that select words or characters, and receives a reward signal that is automatically constructed from the user's backspaces. To improve the default inter- face's actions, we neural network reward model that predicts the reward signal given the user's input and the interface's action. Upon observing user input, interface uses the trained reward model to update the prior policy given by the default interface to a posterior policy conditioned on optimality, then samples action from this posterior (see Figure 1). We call this method x-to-text (X2T), where refers to the arbitrary type of user input; e.g., eye gaze or brain activity.
Our primary contribution is the X2T algorithm for continual learning of communication interface from user feedback. We primarily evaluate X2T through an online user study with 12 participants who webcam-based gaze tracking system select words from a display. To ablation experiments that would be impractical in the online study, we also conduct an observational study with 60 users who use tablet and stylus draw pictures of individual characters. The results show that X2T quickly learns to map input images of eye gaze or character drawings to discrete word or character selections. By learning from online feedback, X2T improves upon default interface that is only trained once using supervised learning and, as a result, suffers from distribution shift (e.g., caused by changes in the user's head position and lighting over time in the gaze experiment). X2T automatically overcomes calibration problems with the gaze tracker by adapting the mis- calibrations over time, without the need for explicit re-calibration. Furthermore, X2T leverages offline data generated by the default interface accelerate online learning, stimulates co-adaptation from the user the online study, and personalizes the interface to the handwriting style of each user in the observational study.
2

---

Published as a conference paper at ICLR 2021
2 LEARNING TO INFER INTENT FROM USER INPUT
In our problem setting, the user cannot directly perform actions; e.g., due to a sensorimotor impair- ment. Instead, the user relies on an assistive typing interface, where the user's intended action is inferred from available inputs such as webcam images of eye gaze or handwritten character draw- ings. As such, we formulate assistive typing 2008; Yue & Joachims, 2009; Li et al., 2010; Lan & Baraniuk, 2016; Gordon et al., 2019). At each timestep, the user provides the interface with context X, where X' is the set of possible user inputs (c.g., webcam images). The interface then performs an action u E 24 where U is the set of possible actions (e.g., word selections). We assume the true reward function is unknown, since the user cannot directly specify their desired task (e.g., writing an email or filling out a form). Instead of eliciting a reward function explicit reward signal from the user, we automatically construct reward signal from the user's backspaces. The key idea is to treat backspaces as feedback on the accuracy of the interface's s actions.
contextual bandit problem (Langford & Zhang,
Our approach to this problem is outlined in Figure 1. We aim to minimize expected regret, which, in our setting, is characterized by the total number of backspaces throughout the lifetime of the interface. While a number of contextual bandit algorithms with lower regret bounds have been proposed in prior work (Lattimore Szepesvári, 2020), we use simple strategy that works well in our experiments: train a neural network reward model to predict the reward given the user's input and the interface's s action, and select actions with probability proportional to their predicted optimality. Our approach is similar to prior work on deep contextual nulti-armed bandits (Collier & Llorens, 2018) and NeuralUCB (Zhou et al., 2019), except that instead of using Thompson sampling or UCB to balance exploration and exploitation, we use simple, stochastic policy.
2.1 MODELING USER BEHAVIOR AND FEEDBACK
Unlike in the standard multi-armed bandit framework, we do not get to observe an extrinsic reward signal that captures the underlying task that the user aims to perform. To address this issue, we infer rewards from naturally-occurrigg user behavior. In particular, in the assistive typing setting, we take advantage of the fact that we can observe when the user backspaces; i.c., when they delete the most recent word or character typed by the interface. To infer rewards from backspaces, we make two assumptions about user behavior: (1) the user can perform a backspace action independently of our interface (e.g.. by pressing button); (2) the user tends to backspace incorrect actions; and (3) the user does not tend to backspace correct actions. Hence, we assign a positive reward to actions that were not backspaced, and assign zero reward to backspaced actions. Formally, let E {0, } denote this reward signal, where r = 0 indicates an incorrect action and r = 1 indicates a correct action.
2.2 TRAINING THE REWARD MODEL TO PREDICT FEEDBACK
In order to perform actions that minimize expected regret- i.e., the total number of backspaces over time - we need to learn a model that predicts whether or not the user will backspace a given action in given context. To do so, we learn reward model pe [r|x, u). where pe is a neural network and 0 are the weights. Since the reward r E {0, 1} can only take on one of two values, Pa is binary classifier. We train this binary classifier on a dataset D of input-action-reward triples (x, r, In particular, we fit the model pe by optimizing the maximum-likelihood objective; i.e., the binary cross-entropy loss (see Equation 2 in the appendix).
Since X2T learns from human -n- thop feedback, the amount of training data is limited by how frequently the user operates the interface. To reduce the amount of online interaction data needed to train the reward model, we use offline pretraining. We assume that the user already has access to some default interface for typing. We also assume access to an offline dataset input-action pairs generated by the user and this default interface. We assign zero rewards to the backspaced actions and positive rewards to the non- non-backspaced actions in this offline dataset, and initially train reward model to predict these rewards given the user's inputs and the default interface's 's actions. Thus, when X2T is initially deployed, the reward model has already been trained on the offline data, and requires less online interaction data reach peak accuracy.
3

---

Published as a conference paper at ICLR 2021
Algorithm 1 Xeto (X2T)
Require म. Oinit
default interface, pretrained reward model parameters
while true do
X Puser (x)
user gives input > interface performs action infer reward from user feedback D store online input-action-reward data D update reward model w/SGD
u ~ π (u(x) po(r = 1|x, u) (u)()
r < 0 if user backspaces else
D < [x, (x,,x)
4- 0+ Vo Vo ((xu))) log
2.3 USING THE REWARD MODEL TO SELECT ACTIONS
Even with offline pretraining, the initial reward model may not be accurate enough for practical use. To further improve the initial performance of our interface at the onset of online training, we combine our reward model po (r|r,, u) with the default interface #(u(x). We assume that is stochastic policy and that we can evaluate it on specific inputs, but do not require access to its implementation. We set our policy π (u|x) p(u x/ r 1) to be the probability of an action conditional on optimality, following the control-as-inference framework (Levine, 2018). Applying Bayes' theorem, we get p(u|x,r" rullx,r=1 1) xx p(r = 1|x, u ()))((((x)) The first term is given by our reward model pe, and the second term is given by the default interface. Combining these, we get the policy
=
xt <u|x) po (r 1|x, u (u|x).
(1)
This decomposition of the policy improves the initial performance of our interface at the onset of online training, and guides exploration for training the reward model. It also provides a framework for incorporating a language model into our interface, as described in Section 4.3.
Our x-to-text (X2T) method is summarized in Algorithm 1. In the beginning, we assume the user has already been operating the default interface मे for some time. In doing so, they generate an 'offline' dataset that we use to train the initial reward model parameters Oniit- When the user starts using X2T, our interface π already improves upon the default interface गे by combining the default interface with the initial reward model via Equation 1. As the user continues operating online data is used to maintain or improve the accuracy of the reward model. At each timestep. the user provides the interface with input x ~ Puser (x). Although standard contextual bandit methods assume that the inputs x are i.i.d., we find that X2T performs well even when the inputs are correlated due to user adaptation (see Section 4.2) or the constraints of natural language (see Section 4.4). The interface then uses the policy in Equation to select an action u. We then update the reward model Pe. by taking one step of stochastic gradient descent to optimize the maximum-likelitee objective in Equation 2. Appendix A.1 discusses the implementation details.
interface, the resulting
3 RELATED WORK
Prior methods for training interfaces with supervised learning typically collect a dataset of input- action pairs, then fit model that predicts actions given inputs. These methods tend to either assume access to ground-truth action labels from the user (Anumanchipalli et al., 2019; Wang et al., 2016; Karamcheti al., 2020), or assume the user always intends to take the optimal action (Gilja al., 2012: Dangi et al., 2013; 2014; Merel et al., 2015). Unfortunately, the user may not always be able to provide ground-truth action labels. Furthermore, order for the system compute optimal actions, the user must be instructed to perform specific calibration tasks for which the optimal policy already known. These calibration tasks may not be representative of the tasks that the user intends to perform. This can lead to a distribution mismatch between the inputs that the model is trained on during the calibration phase, and the inputs that the model is evaluated on test time when the user performs their desired tasks. Standard methods address this problem by periodically repeating the calibration process (Willett et al., 2020; Proroković et al., 2020), which can be time-consuming, disruptive, and requires assumptions about when and how frequently to re-calibrate. X2T overcomes these issues by continually learning from user feedback on tasks that naturally arise as the user types, rather than imposing separate training and test phases.
Extensive prior work on text entry systems (MacKenzie & Tanaka-Ishii 2010) enables users to type using eye gaze (Ward al., 2000), Braille (Oliveira et al., 2011 20, gestures (Jones et al., 2010), and
4

---

Published as a conference paper at ICLR 2021
palm keyboards for wearable displays (Wang et al., 2015). X2T differs in that it enables the user to type using arbitrary inputs like eye gaze or handwriting, rather than a fixed type of imput that restricts the flexibility of the system and must be anticipated in advance by the system designer.
X2T trains a typing interface through reinforcement learning (RL) with human-in- -n-the-ooop feedback instead of explicit reward function. COACH (MacGlashan et al., 2017; Arumugam al., 2019), TAMER (Knox& & Stone, 2009; Warnell et al., 2017), and preference learning (Sadigh et al., 2017; Christiano et al., 2017) also train an RL agent without access to extrinsic rewards, but require ex- plicit user feedback on the agent's behavior. X2T differs in that it learns from naturally-occurrigg feedback, which requires no additional effort from the user to train the agent. Other prior work trains RL agents from implicit signals, such as electroencephallogrphy (Xu et al., 2020), peripheral pulse measurements (McDuff & Kapoor, 2019), facial expressions (Jaques et al., 2017; Cui et al., 2020), and clicks in web search (Radlinski & Joachims, 2006). X2T differs in that it trains an interface that always conditions on the user's input when selecting an action, rather than an autonomous agent that ignores user input after the training phase. Furthermore, X2T focuses on the assistive typing domain, where, to the best of our knowledge, backspaces have not yet been used to train an interface through RL. Related work on assistive robotic teleoperation interfaces proposes human-in momnan-n--the-ooee tnophe-loop RL methods that minimally modify user actions (Pilarski et al.. 201 I; Broad et al., 2017; Reddy et al., 201& 8; Schaff & Walter, 2020; Du al., 2020; Jeon et al., 2020). X2T differs in that it learns to operate on arbitrary types of user inputs, instead of assuming the user provides suboptimal actions.
4 EXPERIMENTAL EVALUATION
We seek to answer the following questions: Q1 (Sec. 4.1): Does X2T improve with use and learn to outperform a non-adaptive interface? Q2 (Sec. 4.2): Does the user adapt to the interface while the interface adapts to the user? Q3 (Sec. 4.3): Does X2T personalize the interface different input styles? Q4 (Sec. 4.4): Do offline pretraining and an informative prior policy accelerate online learn- ing? Q5 (Sec. 4.4): Does online learning improve the interface beyond the initialization provided by offline pretraining? Q6 (Sec. 4.5): Can X2T improve the accuracy of a brain-computer interface for typing? To answer Q1-2, we run user study with 12 participants who use webcam images of their eyes to type via gaze (see Figure 8). To answer Q3-5, we conduct an observational study with prerecorded images of handwritten characters drawn by 60 users with a tablet and stylus (see Figure 7). To answer Q6, we conduct pilot study with one participant using an electrocorticographyy based brain-computer interface. In our experiments, we use default interfaces that are not necessarily the state of the art in gaze tracking or handwriting recognition, but are instead chosen to test the hy- potheses in Q1-6. Appendix A. describes the experiment design in detail.
4.1 ADAPTING THE INTERFACE TO THE USER
In this experiment, we aim to test X2T's ability to improve over time, relative to a non-adaptive, default interface. To that end, we formulate gaze-based word selection task in which we display a list of words to the user (see Figure 8), ask them to look at their desired word, record an image from their webcam that captures their eye gaze, and predict their desired word. To measure objective performance, we ask the user to type specific goal sentences. To simplify the experiment, we restrict the action space U = {1, 3, ..., 8} the eight buttons on the sereen, and always assign the next word in the sentence to one of those eight buttons (see Figure 8).
We evaluate (1) a default interface that uses iTrrcker (Krafka et al., 2016) estimate the user's 2D gaze position on the screen and select the nearest button, and (2) X2T. We calibrate the default interface once at the beginning of each experimental condition for each user, by asking the user to look at each of the eight buttons one by one, recording 20 eye image samples for each button in 2 cycles, and training 2D gaze position estimator using the i'Tracker method. After calibration, the default interface stays fixed, and does not adapt to the user within the session; in part, because we do not know in advance which word the user intends to select, so we cannot automatically harvest the necessary paired data of gaze images and targets to re-calibrate the interface. Instead of periodically interrupting the user to gather new paired data and re-calibrate, X2T continually adapts to the user throughout the session, following Algorithm 1. Appendix A.1 describes the implementation of the default interface and X2T in further detail. We measure the performance of each method using the ground-truth accuracy of action selections. To access ground-truth actions for calculating accuracy,
5

---

Published as a conference paper at ICLR 2021
Online User Study: Typing with Eye Gaze
(a)
Comiletae becurar or 0e (2 Methed)
(c)
Action Prediction Accuracy
0.8
0.7
0.6
05
0.5
X2T [Our Method)
Default Interface (Baseline)
50
100
150
200
0,3 0.4
0.
0.6
0.7
0.5 6,9 3.0
Dellatk mncricee securate
Number User Interactions
Accuracy
lcacorere
An online user study with 12 participants in the gaze tracking domain that addresses Q1: does X2T improve with and learn outperform non-adaptive interface? (a) X2T predicts the user's intended action more accurately than the default interface, and the gap between the two methods grows over time. We smooth the curves using a moving average with a window size of 20 interactions, and measure standard error across the 12 users. (b) X2T improves the performance of 10 out of the 12 users, and the improvement from X2T is smaller when the default interface already performs well. Each orange circle represents one of the 12 users. The dashed gray line shows defaull-equivalent performance, and the dotted orange lines show the difference between X2T and default performance. Per-user accuracy is averaged across 250 interactions. (c) shown the screenshot in Figure 8, the user is shown a display of eight words arranged in a circle. Here, we plot the default interface's 2D gaze position estimates given user inputs intended to select the bottom-mov button. The widely-scattered 2D estimates show that action prediction is particularly hard for this button, perhaps because the user's eyes tend be more obscured when they are looking down. By training a reward model on user feedback, X2T helps the interface recover from incorrect gaze estimates.
Figure 2:
we instruct the user try select a specified word from the list displayed on their screen. Note that this instruction is not an essential component of X2T, and is used purely to evaluate objective performance in this experiment.
The results in panel (a) of Figure 2 show that at the onset of online learning, both X2T and the default interface predict the user's intended action with the same accuracy, but quickly diverge. The default interface's performance degrades over time, potentially due to distribution shift in the user's inputs caused by changes head position, background lighting, and other visual conditions. In contrast, X2T maintains the interface's strong initial performance throughout the experiment, by continually updating the reward model to reflect changes the user's 's inputs. Panel (b) shows that X2T significantly improves the performance of 10 out of 12 participants, and that there are dimin- ishing returns to X2T when the default interface already performs well. We ran a one-way repeated measures ANOVA the action prediction accuracy dependent measure from the default and X2T conditions, with the presence of X2T as factor, and found that f(1, the appendix corroborate these results: users reported feeling
17. 23, p < 0.01. The
subjective evaluations in Table that X2T selected the words they wanted and improved over time more than the default interface. Panel (c) qualitatively illustrates how X2T helps the interface recover from incorrect 2D gaze posi- tion estimates: each green 'x' shows that even when the default interface estimates gaze position far from the intended button, which would normally cause an incorrect action prediction, the reward model can adjust the overall action prediction back to the correct button via Equation 1. We also find that X2T performs well even when the user's s feedback slightly noisy: the user backspaces mistakes, and does not backspace correct actions, in 98.6% of their interactions.
4.2 USER ADAPTATION TO THE INTERFACE
In the previous experiment, we tested X2T's ability to adapt the interface to the user. Prior work on human-machine co-adaptation (Taylor et al., 2002: Carmena, 2013; Shenoy & Car- mena, 2014: Nikolaidis et al., 2017) and the evolution of communication protocols between hu- mans (Hawkins al., 2020) suggests that an adaptive interface may not only learn to assist static user, but even stimulate the user to adapt their input style to the interface. periment, we investigate whether the user adapts to the gaze-based interface described in Sec- tion 4.1. To do so, we perform a counterfactual experiment: instead of training and evalu- ating X2T on inputs generated by the user while they were typing with X2T, we train and
this ex-

---

Published as a conference paper at ICLR 2021
evaluate X2T on inputs x generated by the user while they were typing with the default inter- face. During the counterfactual experiment, instead of asking the user for new inputs, we sim- ply replay the old imputs that were intended for the default interface, and automate backspaces. Co-adaptation between User and Interface
This enables us to test if the user adapted their inputs X2T, if the user provided the same distribution of inputs to both the default inter- face and X2T.
Action Prediction Accuracy
0.8
The results in Figure 3 suggest that the user does indeed adapt their input style to the in- terface, and that this user adaptation improves performance. Comparing X2T's actual perfor- mance (orange curve) to X2T's counterfactual performance on replayed user inputs that the user originally intended for the default inter- face (teal curve), we see that X2T is able to perform much better with inputs intended for X2T compared to inputs intended for the de- fault interface. From this result alone, one might infer that the user provided X2T with inputs that were more generically predictable than the imputs they provided to the default in- terface. However, by comparing the default in- terface's 'ss i performance on replayed user inputs that the user originally intended for X2T (gray curve) to X2T's performance on the same in- puts (orange curve), we see that X2T performs better than the default interface on the same in- puts. This suggests that the user's 's inputs to X2T are not merely easier to predict, but in fact adapted specifically to X2T. In other words, X2T stimulates user co-adaptation to the inter- face.
0.7
0.6
0.4
X2T (Our Method)
X2T on Inputs Intended for Default Interface Default Interface on Inputs Intended for X2T
0.3
150
200
Number of User Interactions
counterfactual experiment with the on- line user study data that addresses Q2: does the user adapt to the interface while the interface adapts to the user? Training and evaluating X2T on user inputs orig- inally intended for the default interface leads worse performance than doing so on user inputs intended for X2T (orange vs. teal). Evaluating the default inter- face on inputs intended for X2T Icads to worse perfor- mance than evaluating X2T the same inputs (orange vs. gray). These results suggest that the user adapts their input style specifically to X2T, and that this user co-adaptation improves performance. We smooth the curves using moving average with window size of 20 interactions, and measure standard error across the 12 users.
Figure
4.3 PERSONALIZING THE INTERFACE FOR DIFFERENT USERS
In this experiment, we demonstrate X2T's s ability to operate on a different type of user input: draw- ings of characters (see Figure 7 in the appendix); which, for some users, can be easier to provide than direct keystrokes (Willett et al., 2020). We also investigate to what extent X2T learns person- alized interface that is uniquely adapted to the individual user. To that end, we analyze handwriting samples from 60 users, collected through a tablet and stylus, from the UJI Pen Characters Database (Llorens et al., 2008). Each sample consists of a sequence of 2D positions that traces the trajectory of the user's pen tip they draw known character. We conduct observational study with this data by sampling goal sentences, replaying a randomly-selected handwriting sample of the user's desired next character from the goal sentence, and treating each drawing as the user input x; akin to the replay experiment in Section 4.2. This observational study is equivalent to an online study, except that it does not permit user co-adaptation to the interface since it replays logged user inputs, and assumes that the feedback signal is not noisy when automating backspaces. To test X2T's ss ro- bustness to noise and distributional shift in the user's input, we perturb the replayed pen tip positions by adding Brownian noise (see Figure 7 in the appendix).
We evaluate (1) a default interface trained classify handwritten EMNIST characters (Cohen et al., 2017), and (2) X2T. We intentionally train the default interface on EMNIST images in- stead of UJI Pen Characters drawings, to model real-world applications with a distribution mis- match between the training data and test data, discussed in Section 3. To address the chal- lenge of selecting from 27 possible character actions, we use a language model (Dai et al., 2019) to predict the prior likelihood of action PLM (u, uo:r-1) given the preceding characters uga-1 in the user's text field. We use this language model in both the default interface and X2T. In particular, we set (ulxx) x po(ul PLMM 11: where Pp is the EMNIST image classifier.
7

---

Published as a conference paper at ICLR 2021
Evaluation
Training
User
User 2
User 3
User 4
As in Section 4.1, the default interface stays fixed throughout the experiment and does not adapt to the user, since re-training the default interface would require interrupting the user to collect paired data.
User
.955
.033
.025
.017
User
.018
.463
.009
.061
User
.000
.002
.955
.039
User
.018
.002
.025
.993
The results in Figure 4 show that X2T signifi- cantly outperforms the default interface (orange vs. gray). We ran a one-way repeated measures ANOVA on the action prediction accuracy de- pendent measure from the default and X2T con- ditions, with the presence of X2T as a factor, and found that f(1,59) 59) 37.46, < 0.0001. Furthermore, Table shows that X2T learns an interface that is particularly suited to the user whose data it was trained on: when an interface trained on user i's data is evaluated on data from user # instead of user i, performance degrades substantially.
Table I: An observational study with 60 users in the handwriting recognition domain that addresses Q3: does X2T personalize the interface to different input styles? We measure action prediction accuracy across 1000 interactions, and randomly sample users 1-4 from the pool of 60 users. The interface trained on user i is substantially more accurate when evaluated from user than inputs user j. suggesting that the learned interface is personalized to each individual user.
inputs
4.4 ABLATION EXPERIMENTS
In this experiment, we aim to test the impor- tance of three components of X2T: offline pre- training. an informative prior policy, and online learning. As discussed in Sections 2.2 and 2.3, to improve the initial performance X2T and accelerate online learning, pretrain of- fline data collected using the default interface, and incorporate prior knowledge from the de- fault interface into the prior policy म in Equa- tion 1. Using the handwriting recognition task from Section 4.3, we conduct ablation experi- ments which we drop out each of the three components, one by one, and measure any re- sulting changes in performance. In the first con- dition, we test the effect of not performing of- fline pretraining, by initializing X2T with ran- dom weights Oiit In the second, test the effect of not incorporating prior knowledge into be
Observational Study: Hardariting Recognition
Actev medeeton Accime
con-
0.2
XET without Offilee Pretroining XET witoot Online Leariing
cOur Methodi Debutt interface (Baseline)
X2T wineut Pitor Pulicy
0.0
200
400
600
800
our policy, by setting the prior policy uniform distribution over actions. In the third, test the effect of not learning online and in- stead relying solely on offline pretraining, by freezing the reward model parameters after of- fline pretraining and not storing online data.
Figure 4: An observational study with 60 users in the handwriting recognition domain that addresses Q4-5: do offline pretraining and informative prior policy accelerate online learning, does online learning im- prove the interface beyond the initialization provided by offline pretraining? In this ablation experiment, we remove each of the three components offfine pretrain- ing, an informative prior policy, and online learning one by one, and find that each component critical for maintaining high action prediction accuracy at different stages of the experiment.
The results Figure show that offline pre- training is helpful at the onset of online learn- ing (orange vs. teal), but does not have a sub- stantial effect on performance after some online data has been collected. This is unsurprising, since leveraging offline data is only necessary when insufficient online data has been collected. Us- ing the default interface as a prior policy in Equation is critical for X2T's performance throughout the experiment (orange vs. red). This is also unsurprising, given that the default interface contains useful prior knowledge about how to interpret user inputs. Online learning is not particularly helpful the onset of the experiment (orange vs. purple), but has substantial effect on performance over time. This result shows that learning from on-policy data critical for X2T. In other words, X2T learns best when learns from its own mistakes, rather than the mistakes of the default interface.
4.5 PILOT STUDY WITH A
INTERFACE
8

---

Published as a conference paper at ICLR 2021
Pilot Study
In this experiment, demonstrate X2T's abil- ity to improve the typing accuracy of a brain- computer interface (BCI) that uses 128-
Uniform Random (Baseline)
Action Prediction Accuracy
0.5
channel chronic electrocorticography (ECoG)
implant to measure neural activity (Leuthardt et al., 2004; Silversmith et al., 2020). Each user input has 128 features, which are obtained by pre-proeessing the raw ECoG signals (Ap- pendix A.1.3 discusses the details). The results in Figure 5 show that, after offline pretraining on a dataset of 2913 input-action-rewarr tu- ples, X2T achieves 60.7% action prediction ac- curacy during a new session of 300 steps. We train default interface paired data of inputs and ground-truth actions collected before the offline data. To evaluate the default interface, we conduct a counterfactual experiment (sim- ilar those in previous sections) on the 300 steps of data from the X2T evaluation session. The default interface only achieves 46.3% ac-
0.4
0.3
0.1
0.0
X2T (Our Method)
Default Interface (Baseline)
Figure 5: A pilot study with one participant that ad- dresses Q6: can X2T improve the accuracy of a brain- computer interface (BCI) for typing? We find that X2T adapts to recent apput-action-rwall data and outper- forms a default interface that is trained via supervised learning on older, paired input-action data.
curacy. Interestingly. the default interface never predicts the top-most button action, leading to 0% recall rate for that action. This most likely due to changes the feature processing pipeline or other kinds of distributional shift in the user inputs over time. In contrast, X2T achieves 70.2% recall on the top-most button action, suggesting that it can overcome these challenges.
5 DISCUSSION
In our online user study gaze-based word selection, we show that X2T learns to outperform non-adaptive interface under minutes (including offline data collection), and that the user si- multancously adapts their input style to the interface. The observational study on handwritten char- acter recognition shows that X2T can also personalize the interface to individual Addittoldiionally, pilot study with brain-computer interface user shows that X2T improves the recall rate of one particular action from 0% to 70.2%. These experiments broadly illustrate how online learning from user feedback can be used to train human- mananne interface.
One limitation of this work is that we assume backspaces can be generated independently of X2T. This assumption restricts X2T to settings where the user can communicate binary signal; e.g., through a button press or a sip-and-puff interface. One direction for future work is to relax this assumption by training binary classifier generate feedback signals from, c.g., facial expressions or brain activity. X2T is also limited in that the benefit from using X2T to fine-tune the default interface may decrease as the default interface is improved, as suggested by the diminishing returns in Figure 2. One direction for future empirical work is to test X2T with saat--of-the-ar default in- terfaces. spite of these limitations, methods like X2T that learn from their mistakes and stimulate user co-adaptation provide a general mechanism for improving user interfaces; not only for assistive typing, but also for other domains, such as brain-computer interfaces for prosthetic limb control and augmented reality devices for visually-impi -imaarred users.
ACKNOWLEDGMENTS
Thanks to Sarah Seko, Reza Abiri, Yasmin Graham, and members of the Neural Engineering and Plasticity lab at UC San Francisco for helping to conduct the brain-computer interface experiments in Section 4.5. Thanks to members of the InterACT and RAIL labs at UC Berkeley for feedback on this project. This work supported in part by NVIDIA Graduate Fellowship, AFOSR FA9550- 17-1-0308, NSF NRI 1734633, NIH New Innovator Award [I DP2 HD087955), and UCSF Weill Institute for Neurosciences.

---

Published as a conference paper at ICLR 2021
REFERENCES
Gopala K Anumanchipalli, Josh Chartier, and Edward F Chang, Speech synthesis from neural decoding of spoken sentences. Nature, 2019.
Dilip Arumugam, Jun Ki Lee, Sophie Saskin, and Michael L Littman, Deep reinforcement learning from policy-dependent human feedback. arXiv preprint arXiv:1902.04257, 2019.
Alexander Broad, Todd David Murphey, and Brenna Dee Argall. Learning models for shared control of human-machine systems with unknown dynamics. In Robotics: Science and Systems, 2017.
Jose M Carmena. Advances in neuroprosthetic learning and control. PLoS Biology, 2013.
Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. One billion word benchmark for measuring progress in statistical language modeling. arXiv preprint 3rX.v:1312.3000 2013.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep re- inforcement learning from human preferences. In Neural Information Processing Systems, 2017.
Gregory Cohen, Saeed Afshar, Jonathan Tapson, and Andre Van Schaik. Emnist: Extending mnist to handwritten letters. In International Joint Conference on Neural Networks, 2017.
Mark Collier and Hector Urdiales Llorens. Deep contextual multi-armed bandits. arXiv preprint
1:X1::/807009000 2018.
Yuchen Cui, Qiping Zhang, Alessandro Allievi, Peter Stone, Scott Niekum, and W Bradley Knox. The empathic framework for task learning from implicit human feedback. arXiv preprint arXiv:2009. 13649, 2020.
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdi- nov. Transformer-xl: Attentive language models beyond fixed-length context. arXiv preprint 2019.
Siddharth Dangi, Amy L Orsborn, Helene G Moorman, and Jose M Carmena. Design and analysis of closed-loop decoder adaptation algorithms for brain maih-ee interfaces. Neural computation, 2013.
Siddharth Dangi Suraj Gowda, Helene G Moorman, Amy L Orsborn, Kelvin So, Maryam Shanechi, and Jose M Carmena. Continuous closed-loop decoder adaptation with recursive maximum likelihood algorithm allows for rapid performance acquisition in brain-machine interfaces. Neural computation, 2014.
Yuqing Du, Stas Tiomkin, Emre Kiciman, Daniel Polani, Pieter Abbeel, and Anea Dragan, Ave: Assistance via empowerment. arXiv preprint 20Xv:2200..1476. 2020.
Vikash Gilja, Paul Nuyujukian, Cindy Chestek, John P Cunningham, M Yu Byron, Joline M Fan, Mark M Churchland, Matthew T Kaufman, Jonathan C Kao, Stephen I Ryu, et al. A high- performance neural prosthesis enabled by control algorithm design. Nature neuroscience, 2012.
Ethan K Gordon, Xiang Meng. Matt Barnes, Tapomayukh Bhattacharjee, and Siddhartha S Srini- vasa. Adaptive robot-assisted feeding: An online learning framework for acquiring previously- unseen food items. arXiv preprint arXiv: 000800788, 2019.
Robert D Hawkins. Michael C Frank, and Noah D Goodman. Characterizing the dynamics of learning in repeated reference games. Cognitive Science, 2020.
Natasha Jaques, Shixiang Gu, Dzmitry Bahdanau, José Miguel Hermández-Lobato, Richard E Turner, and Douglas Eck. Sequence tutor: Conservative fine-tuning of sequence generation mod- cls with kl-control. International Conference on Machine Learning, 2017.
Hong Jun Jeon, Dylan P Losey, and Dorsa Sadigh. Shared autonomy with learned latent actions. arXiv preprint 20055:2005.032 2020.
10

---

Published as a conference paper at ICLR 2021
Eleanor Jones, Jason Alexander, Andreas Andreou, Pourang Irani, and Sriram Subramanian. Ges- text: accelcrometer-bate based gestural text-entry systems. In SIGCHI Conference on Human Factors in Computing Systems, 2010.
Siddharth Karamcheti, Dorsa Sadigh, and Percy Liang- Learning adaptive language interfaces through decomposition. arXiv preprint 1rXiv:2010.05199, 2020.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint TrXi::1412.69880 2014.
W Bradley Knox and Peter Stone. Interactively shaping agents via human reinforcement: The tamer framework. In International Conference on Knowledge Capture, 2009.
Kyle Krafka, Aditya Khosla, Petr Kellnhofer, Harini Kannan, Suchendra Bhandarkar, Wojeiech Matusik, and Antonio Torralba. Eye tracking for everyone. In IEEE Conference Computer Vision and Pattern Recognition, 2016.
Andrew Lan and Richard G Baraniuk. A contextual bandits framework for personalized learning action selection. In Educational Data Mining, 2016.
John Langford and Tong Zhang. The epoch-greedy algorithm for multi-armed bandits with side information. In Neural Information Processing Systems, 2008.
Tor Lattimore and Csaba Szepesvári. Bandit algorithms. Cambridge University Press, 2020.
Eric C Leuthardt, Gerwin Schalk, Jonathan R Wolpaw, Jeffrey G Ojemann, and Daniel Moran. A brain-computer interface using electrocorticcoraahic signals in humans. Journal of Neural Engineering, 2004.
Sergey Levine. Reinforcement learning and control as probabilistic inference: Tutorial and review. arXiv preprint arXiv:1805.00900 2018.
Lihong Li, Wei Chu, John Langford, and Robert E Schapire. A contextual-bandit approach to personalized news article recommendation. International Conference on World Wide Web, 2010.
David Llorens, Federico Prat, Andrés Marzal, Juan Miguel Vilar, María José Castro, Juan- Carlos Amengual, Sergio Barrachina, Antonio Castellanos, Salvador Espana Boquera, Jon Ander Gómez, et al. The ujipenchars database: a pen-based database of isolated handwritten characters. In International Conference on Language Resources and Evaluation, 2008
James MacGlashan, Mark Ho, Robert Loftin, Bei Peng, David Roberts, Matthew E Taylor, and Michael L Littman. Interactive learning from pelicy-dependent human feedback. arXiv preprint arXiv: 2017.
I Scott MacKenzie and Kumiko Tanaka-Ishia Text entry systems: Mobility, accessibility, universal- ity. 2010.
Joseph G Makin, David A Moses, and Edward F Chang. Machine translation of cortical activity to text with an encoder encoder-decoder framework. Technical report, 2020.
Daniel McDuff and Ashish Kapoor. Visceral machines: Reinforcement learning with intrinsic
physiological rewards. In International Conference Learning Representations, 2019. URL https: htttps://cpenzeview wwtpp/vieneeeeeeeee//e/ee net/ wtw.leewwwnn//ouuumeeeeeee
9KC.
Josh Merel, David Carlson, Liam Paninski, and John P Cunningham. Neuroprosthetic decoder 2015.
training as imitation learning. arXiv preprint
Stefanos Nikolaidis, Yu Xiang Zhu, David Hsu, and Siddhartha Srinivasa. Human-robot mutual adaptation in shared autonomy. In ACM/IEEE International Conference on Slmman-Roba Inter- action, 2017.
João Oliveira, Tiago Guerreiro, Hugo Nicolau, Joaquim Jorge, and Daniel Gonçalves. Brailletype: unleashing braille over touch screen mobile phones. In IFIP Conference on Human- Commatt Interaction, 2011.
11

---

Published as a conference paper at ICLR 2021
Patrick M Pilarski, Michael R Dawson, Thomas Degris, Farbod Fahimi, Jason P Carey, and Richard S Sutton. Online human training of a myoelectric prosthesis controller via actor-critic reinforcement learning. In IEEE International Conference Rehabilitation Robotics, 2011.
Krsto Proroković, Michael Wand, and Jürgen Schmidhuber. Meta-leaming for recalibration of emg- based upper limb prostheses. 2020.
Filip Radlinski and Thorsten Joachims. Evaluating the robustness of learning from implicit feed- back. arXiv preprint cs/0605036, 2006.
Siddharth Reddy, Anca D Dragan, and Sergey Levine. Shared autonomy via deep reinforcement learning. arXiv preprint arXiv:1802.01744, 2018.
Dorsa Sadigh, Anca D Dragan, Shankar Sastry, and Sanjit A Seshia. Active preference-based learn- ing of reward functions, In Robotics: Science and Systems, 2017.
Charles Schaff and Matthew R Walter. Residual policy learning for shared autonomy. arXiv preprint wxXYv:2004.05097, 2020.
Krishna Shenoy and Jose M Carmena. Combining decoder design and neural adaptation brain- machine interfaces. Neuron, 2014.
Daniel B Silversmith, Reza Abiri, Nicholas F Hardy, Nikhilesh Natraj, Adelyn Tu-Chan, Edward F Chang, and Karunesh Ganguly. Plug-and-play control of a brain-computer interface through neural map stabilization. Nature Biotechnology, 2020.
Dawn Taylor, Stephen Helms Tillery, and Andrew Schwartz. Direct cortical control of 3d neuroprosthetic devices. Science, 2002.
Cheng-Yas Yao Wang, Wei-Chen Chu, Po- Po-Tsung Chiu, Min-Chich Hsiu, Yih-Harn Chiang, and Mike Y Chen. Palmtype: Using palms as keyboards for smart glasses. In International Conference on Human- Comman-Computer Interaction with Mobile Devices and Services, 2015.
Sida Wang, Percy Liang, and Christopher D Manning. Learning language games through interac- tion. arXiv preprint arXiv:1606.0244/ 2016.
David J Ward, Alan F Blackwell, and David JC MacKay. Dasher-a data entry interface using continuous gestures and language models. ACM Symposium on User Interface Software and Technology, 2000.
Garrett Warnell, Nicholas Waytowich, Vernon Lawhern, and Peter Stone. Deep tamer: Interactive agent shaping in high-dimensional state spaces. arXiv preprint arXiv: 10163, 2017.
Francis R Willett, Donald T Avansino, Leigh R Hochberg, Jaimie M Henderson, and Krishna V Shenoy. High-performance brain-to-text communication via imagined handwriting. bioRxiv, 2020.
Alan Wrench. The mocha-timit articulatory database, 1999.
Duo Xu, Mohit Agarwal, Faramarz Fekri, and Raghupathy Sivakumar. Accelerating reinforcement learning agent with eeg-based implicit human feedback. arXiv preprint arXiv:2006. 16498, 2020.
Yisong Yue and Thorsten Joachims. Interactively optimizing information retrieval systems as dueling bandits problem. In International Conference Machine Learning, 2009.
Dongruo Zhou, Lihong Li, and Quanquan Gu. Neural contextual bandits with upper confidence bound-based exploration. arXiv preprint 1rX1v:1911.04462, 2019.
12

---

Published as a conference paper at ICLR 2021
A APPENDIX
A. IMPLEMENTATION DETAILS
Stochastic gradient descent. We use Adam (Kingma & Ba, 2014) to optimize the binary cross- entropy loss described in Section 2.2,
(())
Σ
rlog (po(r 1|x, + log (1 pe 1/x, u)).
(x.u,)jDD
We set gradient norm clipping to 10 in all experiments.
Offiie pretraining. To pretrain X2T offinn data, we first train the reward model po con- vergence on the offline data, and store the learned network weights Oinit- Once a minimum of four online uppt-action-rewart triples have been collected, after every interaction, we update the reward model pa by taking one gradient step. We use the values Omit to initialize the network weights (e.g., instead of random initialization).
A.1.1 ONLINE USER STUDY: TYPING WITH EYE GAZE
Eye image features. Instead of operating directly on 224x224 images of the user's ' eyes, we use the activations of the last fully-commn colnecco layer in the iTracker model (Krafka et al., 2016) as the input x to our reward model (o x, u). When training the reward model po: freeze the iTTrackr network weights used to generate this 128-dimensional input x.
Reward model architecture and learning. We set the learning rate for Adam to 10-3, and batch size to 128. We do not perform online updates to the reward model parameters 0 while |D|< 4. In our experiments, for every input x, there is exactly one desired action u" that will result in a positive reward other words, p(r = 1|x, u) p(u= u x).. Hence, we structure the reward model for X2T as po(r= = 1|x, u) = [u(xx,, where fo is an action classifier. Furthermore, since we expect the re- ward model to learn implicitly estimate the user's gaze position order to predict actions, we directly incorporate this inductive bias into the model: we structure the action classifier as fo(u(x) x) x exp (-1lge(x) pos(u) pos(u)ll2). where go (x(x) outputs a 2D position of the user's estimated gaze, and pos(u) u) is the known 2D position of the button for action u. Note that even though (x) outputs 2D position, the parameters are still trained on the reward prediction objective in Equa- tion 2 (e.g., instead of 2D gaze position prediction objective). We represent go using feedforward neural network with one hidden layer containing 64 units, a dropout layer with a dropout rate of 0.3 between the hidden layer and the output layer, and ReLU activations. At each timestep I, record 10
1, while all other actions u o u result in a zero reward = 0. In
10 eye images {x:) ==1 at sampling rate of 10 Hz, and average our predictions over these 10 inputs. Specifically, for X2T, we set go = To Ei go (x()). For the default interface, we average the 2D gaze position estimates across the 10 samples before predicting the action whose button position is nearest to the average gaze position estimate. We initialize 6mi with the offline pretraining scheme described in Section 2.2, using 250 input-action-reward triples collected with the default interface.
Experiment design. We recruited 11 male and I female participants, with average age of 21. Each participant was provided with the rules of the task and a short practice period of 20 interactions input,
to familiarize themselves with the system. Each interaction which consisted of providing observing the interface's action, and deciding whether or not to backspace took an average of seconds. Each participant completed three phases of experiments: A. B, and C. In phase A, they operate the default interface for 250 steps, generating an offline dataset of input-action-rewaad triples that we use to initialize X2T. In phase B, they operate X2T for 250 steps. In phase C, they operate the default interface for 250 steps. To avoid the confounding effects of user learning or fatigue over time, we counterbalance the order of phase and C: six aeddoollysslletee participants completed phase B before C, and the other six participants completed phase C before B. Phase A is used solely to generate offline data to initialize X2T in phase B. One participant's 's room lighting changed substantially during phase B. Since their performance during phase A was substantially better than during phase B, we use their phase A data (instead of phase data) to measure the default interface's ss performance on this (Wrench, 1999), following prior work on speech interfaces (Makin et al., 2020). We set the same goal sentences for each user in each condition.
participant. We sample goal sentences from the MOCHA-TIMI database
13

---

Published as a conference paper at ICLR 2021
Deterministic policy. In our experiments, we find that sampling actions u (u(xx) from the stochastic policy x does not substantially improve exploration. and can in fact degrade perfor- mance by not always choosing the optimal action. This is most likely due to the use of the default interface क in Equation 1, which already provides an effective exploration mechanism. Hence, instead of randomly sampling actions, we deterministically select the highest-likelihood action:
u < arg max, πT (u(x).
A.1.2 OBSERVATIONAL STUDY: TYPING BY DRAWING CHARACTERS
Perturbing user inputs. We perturb the character drawings in the UJI Pen Characters Database by decomposing each drawing into a sequence of pen tip velocity vectors, adding Brownian noise to each velocity, then integrating over the perturbed velocities to yield a complete, perturbed draw- ing (see Figure 7 in the appendix). We compute the Brownian noise by sampling an independent Gaussian noise vector with zero mean and variance 2 10 at each timestep, and summing over these noise vectors from time to time to compute the Brownian noise vector for time The same Brownian noise vector is applied to all the velocity segments in a given drawing. Each user input x is 28x28 image of the user's complete, perturbed drawing of a given character. There are 27 possible characters in the action space U: 26 lower-case letters, and space (which we represent using the digit 7).
Reward model architecture and learning. We set the learning rate for Adam to 5 10-4, batch size to 128, pretrain on the offline data for 20 epochs, and sample actions from the stochastic policy described in Equation I (instead of the deterministic policy described in Appendix A.1.1). We also limit the size of the replay buffer D in Algorithm 1 to the latest 500 input-action-reward triples. We do not perform online updates the reward model parameters A while |D|< 100. We represent the reward model pe as a neural network with the following architecture: 28x28 input layer, 32x5x5 convolutional layer, 2x2 max pool layer, dropout layer with dropout rate of 0.5, 64x5x5 convolu- tional layer, 2x2 max pool layer, dropout layer with dropout rate of 0.3, and a fully-connected output layer. We structure the reward model as an action classifier, as in Section A.1.1. We initialize fuuit with the offline pretraining scheme described in Section 2.2, using 1000 input-action-reward triples collected with the default interface.
Experiment design. The UJI Pen Characters Database (Llorens et al., 2008) contains handwriting samples from 60 users. For each user, includes two repetitions of lowercase letters, uppercase letters, and 10 digits, for total of 1364 samples per user. In our observational study, we randomly sample target sentences, and replay user inputs that attempt to type the characters in those target automatically backspace incorrect actions; realistic modeling choice,
sentences. In the replays, given that in the online user study in Section 4.1, the user did indeed backspace mistakes, and did not backspace correct actions, in 98.6% of their interactions. As in the online user study, we each user's s data through two conditions: default and X2T. Since this is an observational study, we do not need counterbalance the order of the two conditions. For the personalization experiment in Table 1, we assign a randomly-selected, constant value to the random seed of the Brownian noise for each user in each condition. This ensures that comparisons between entry (i, i) and entries (i.j A i) in the 4x4 table are not confounded by differences in the input noise, and are only influenced by systematic differences in the users' individual handwriting styles. As in the online user study, we sample goal sentences from the MOCHA-TIMIT database (Wrench, 1999), and set the same goal sentences for each user in each condition.
Language model. We use the Transformer-XI. language model (Dai et al., 2019) specifically, the word-level version that is pretrained on the One Billion Word dataset (Chelba et al., 2013) - to compute the prior likelihood PLM (LM((u|MM::-1)) 1). To compute character-level likelihoods, we marginal- ize over the 60000 words in the language model's vocabulary that occur the most frequently in the One Billion Word training corpus, not including words with punctuation and converting upper-case to lower-case characters. We feed the previously-ty characters in the current sentence (i.e., not including the previous sentences) context to the language model.
uo:r-1)
14

---

Published as a conference paper at ICLR 2021
A.1.3 PILOT STUDY WITH BRAIN- CAAI-CCOPPTERR INTERFACE
Due to constraints on the duration and number of sessions we were able to conduct with the partici- pant, we made two simplifications to the interface: we simplified the display to use buttons instead of buttons, and we automated backspaces.
Reward model architecture and learning. We set the learning rate for Adam to 10 10-4, batch size to 128, and maximum buffer size [D] to 1000. Unlike the previous experiments, we did not use dropout. We used an 1.2 regularization constant of 0.001 for X2T, and 0.01 for the default interface. For both the default interface and reward model, we used feedforward network architecture with 3 layers of 256 hidden units each. We deterministically selected actions with the maximum conditional likelihood under the policy, instead sampling from the stochastic policy in Equation 1. When pretraining the reward model on offline data, we initialized the reward model network weights with the default interface network weights. We set the prior policy π in Equation to be a uniform prior, instead of using the default interface. To accommodate noise in the user inputs. require 4 consecutive bins of input to be mapped to the same action by the policy before executing that action; after 10 bins with string of 4 consecutive, equal actions, the majority action is executed. In the counterfactual experiment with the default interface, if we reach the end of the bins for a given step without a string of 4 consecutive, equal actions, the majority action is executed. After an action executed and reward collected, we add an input-action-rewale tuple for the input at each bin to the buffer D.
Experiment design. We conducted three experimental sessions: the first on 2/26/21, in which we collected 997 input-action-reww rewarrd samples; the second on 3/5/21, which yielded 1916 samples; and the third on 3/13/21, which yielded 1984 samples. The default interface was trained on 3686 samples (left button intended: 959, down: 916, right: 897, up: 914) recorded prior 2/26/21. There were 3 seconds of input before each word selection. The interface was paused intermittently during each session to accommodate user fatigue.
Feature processing. The ECoG signals were binned at frequency of 2 Hz. Each user input has 128 dimensions, which consist of high-bar band frequencies of raw ECoG signal recorded during each binning period. The remaining implementation details are described in Silversmith et al. (2020).
15

---

Published as a conference paper at ICLR 2021
p-value
Default Interface
X2T
The system selected the words wanted
.05
4.50
5,42
The system improved over time iippooeed at using the system over time
.05
3.17
4.75
.05
4.08
4.42
The system did not select the words wanted
.05
4.08
2.83
The system got got worse using the system time backspaced when there was mistake
time
.05
3.25
2,42
.05
3.25
2.67
.05
5.42
5.83
ignored mistakes (did not backspace them)
> .05
2.33
2.17
Table 2: Subjective evaluations from the 12 participants in the online user study. Means reported below for responses 7-point Likert scale, where Strongly Disagree, 4 Neither Disagree Agree, and 7 Strongly Agree. p-values from one-way way repeated measures ANOVA with presence X2T as a factor influencing responses.
Sensitivity to Noisy Rewards
0.800
X2T (Our Method)
0.775
Default Interface
Action Prediction Accuracy
Empirical Mislabeling Rate in Online User Study
0.750
0.725
0.700
0.675
0.650
0.625
0.600
0.00
0.05
0.10
0.15
0.20
0.25
0.30
0.35
0.40
0.45
0.50
Reward Mislabeling Rate
Figure 6: measure X2T's sensitivity to noise the reward signal, conduct counterfactual experiment with eye gaze user study data, similar to Section 4.2. In this experiment, we flip the reward to with prohahility p. which we call the reward mislabeling rate. We then train our reward model on these noisy rewards. X2T outperforms the default interface for wide range of mislabeling rates, and only performs worse than the default interface when the rewards completely random (i.e., the mislabeling rate 50%). practice, we find that users' backspaces tend follow the assumptions Section 2.1, which leads to relatively low empirical mislabeling rate of 1.4%.
16

---

Published as a conference paper at ICLR 2021
aаaaaaauumm b
666466
7 1
-
a
Handwriting samples from the UJI Pen Characters Database that have been perturbed by adding Brownian noise to pen tip velocities. The left-most column shows the true input, and each successive column shows the perturbed input after 100-timestep intervals. We used the character '7' place of the space character.
Figure 7:
User list of words
Webcam records image of user's eye gaze
11
the
W
User input
X
u
u
Backspace
Predicted word u
π
or continue
Binary feedback (c.g., from button press)
Reward r
Our interface
Reward model Default interface
Input-action-reward log data
π (u|x) x Po = 1|x, u) u)
D - DU ((x r))
Train reward model
Figure 8: An illustration of the eye gaze experiments Section 4.1
17


Reviews:
[{"review_id": "s5Qf2ALrECl", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: \nThis work presents a method for online learning of an assistive typing user interface (XT2) with implicit user feedback. User inputs for such an assistive typing interface are assumed to be in the form of eye gaze or handwritten characters. However, the implicit human feedback is assumed to be backspaces typed on a keyboard. Backspaces are used to delete words predicted by the assistive typing interface based on the user’s input. The online learning of such an interface to improve its assistive performance and adapt to the user over time is framed as a contextual bandit problem. A reward prediction network is trained to predict the use of backspaces (implicit feedback) by the user. This reward prediction network combined with the default interface policy using Baye’s theorem is used to update the policy of the typing interface. The experimental results with two user studies reveal that the presented method performs better than a non-adaptive default interface, stimulates user co-adaptation to the interface, and offline learning accelerates online learning.\n\n########################\n\n\nPros:\n- The paper is well-written and easy to follow.\n- Strong experimental results to support the idea of the work.\n- Offline pretraining of the reward model reduces the burden on online user interactions needed for the interface to be improved.\n- The presented technique is applicable to any form of user inputs and the authors test their approach with two different forms of inputs (eye gaze and handwritten characters).\n- Use of backspaces to train an interface with RL is a novel idea.\n\n########################\n\nCons:\n- The implicit feedback is assumed to be perfect and available via a keyboard. How realistic is this assumption? What happens if the feedback is noisy? Is the availability of an independent backspace action being true to the nature of the interface being truly assistive?\n- It was unclear to me while reading the paper if the implicit feedback (backspaces) are provided via a keyboard or are also predicted with the user’s input (such as with gaze tracking). I assumed its the former with subtle hints in the paper and Fig 1. It would be helpful to make this explicit for the reader in Section 2.1 (as part of assumption 1).\n- This approach does not model the temporal effects of learning. Practically, modeling the problem as an MDP would be more realistic versus contextual bandits.\n\n########################\n\nReason for score: \nThis simple approach is presented with clarity and supported with well-reported experiments (including several ablative analyses). Some minor issues in the writing could be improved but overall the idea is well-presented and well-evaluated.\n\n########################\n\nQuestions during rebuttal:\n- Sec 4.1: Is the user study conducted with the interface type variable being varied within-subjects (i.e. each user uses both default and X2T)? From the experimental results, it seems this is the case but it is not explicitly stated. If this is so, is the order in which users attempt to use the two interfaces (default and X2T) counterbalanced?\n- Please address other questions raised as part of the Cons section and other feedback.\n\n########################\n\nSome typos and other feedback:\n- Section 2,  paragraph 1, sentence 2: “… relies on an assistive typing interface to infer the user’s intended action from available inputs …” -> “… relies on an assistive typing interface, where the user’s intended action is inferred from available inputs …” \n- Algorithm 1: Consider defining what p_user(x) is. Are you making any assumptions on such a model of human user inputs (such as random sampling as suggested in Sec 4.3)? \n- Consider citing a recent work on learning from imperfect implicit user feedback such as facial expressions): Cui, Y., Zhang, Q., Allievi, A., Stone, P., Niekum, S., & Knox, W. B. (2020). The EMPATHIC Framework for Task Learning from Implicit Human Feedback. Conference on Robot Learning (CoRL), 2020.\n- Even though compared to prior work, this work does not assume access to ground truth action labels from the user provided to the interface, it does assume access to ground truth backspace actions. It would be beneficial to emphasize this in paragraph 1 of Section 3.\n- It only becomes clear to me by Section 3 what is meant by “handwriting as an input” and how it can be an assistive input modality. A reference to Appendix Figure 5 early on the introduction, along with highlighting this can be an easier mode of user input versus typing on a keyboard, would be helpful. \n- Section 4, paragraph 1: References to the subsection numbers can be made when stating the evaluation questions. For example: Q1 (Sec 4.1): Does X2T improve with use and learn to outperform a non-adaptive interface? The questions are well-framed and very clear though!\n- Section 42: One of the insights presented for the presented method is that the XT2 interface can learn to automatically overcome calibration issues with the gaze tracker, thus the interface adapts to the mis-calibrations over time without the need for recalibration, even though external conditions would require a recalibration for better eye gaze prediction. This should be highlighted in the introduction as well.\n- Section 4.3: Isn’t p_LM(u) conditioned on the preceding characters of the text seen so far? Would p_LM(u|t) be a better representation?\n- Section 4.4, last paragraph: Consider reversing the order of the first two results presented (they are in opposite order to the questions posed in the previous paragraph).\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "h9t_ZCvCd0Z", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "SUMMARY:\nThe authors propose a simple algorithm for using online learning from implicit human feedback to improve systems that operate in the contextual bandit setting. The main idea is to capture the presence/absence of corrective actions and use this information to infer a reward signal that the system can use to make decisions later. The proposed method is instantiated for text-entry tasks as a system called XT2, and the authors perform extensive empirical evaluation that seems to show that the system is very successful.\n\n\nSTRENGTHS:\n\t(S1) The authors present a simple-yet-seemingly-powerful algorithm for building adaptive interfaces from implicit user feedback.\n\t(S2) The proposed algorithm can easily leverage existing interfaces and therefore provides an interesting path to improving all kinds of existing systems.\n\t(S2) The authors instantiate the algorithm in a compelling use-case scenario -- non-traditional text entry -- and demonstrate seemingly compelling results empirically.\n\n\nWEAKNESSES:\n\t(W1) It's unclear as to whether or not the evaluation in the major plots (\"[Number of User Interactions] vs. [Action Prediction Accuracy]\") is meaningful. In particular, it's not clear what data was used to compute the reported prediction accuracy. It would seem that this is computed over the prediction *for that time step*, but then more detail is needed on the datasets in order to determine whether or not the task somehow got easier for XT2 over time instead of it actually learning.\n\t(W2) Some of the experimental results are confusing. For example, what is the explanation for the position of the red circles in Figure 2c? There seems to be very little intuitive reasoning behind where red vs. green appears -- has XT2 really learned such a complicated and sensitive decision surface? If yes, why should we \"trust\" this decision surface? In analyzing 2c closely, it seems like a nearest-neighbor classifier would have done just as well.\n\t(W3) While I greatly appreciate the authors making clear their assumptions about backspaces in Section 2.1, the authors did not justify why their first assumption (\"the user can perform a backspace action independently of our interface\") is valid. It would seem to me that, in situations where XT2 is necessary, that such an assumption may not actually be valid.\n\t(W4) In Section 2.2, the authors state that they \"assign a positive reward to actions that were not backspaced,\" but this choice is also not justified. It seems to me that there could be cases in which the user simply accepts an incorrect action from the system without providing the feedback.\n\n\nRECOMMENDATION STATEMENT:\nWhile I have listed a number of weaknesses above, overall I really like the paper. The algorithm is simple and can build upon existing interfaces as opposed to simply replacing them. Moreover, the algorithm is built to leverage data that is easy to collect, and therefore it seems feasible that similar systems could be deployed for a number of applications. That said, the current paper is lacking in the description of the experimental results and justifications for some of the assumptions.\n\n\nQUESTIONS FOR AUTHORS:\n\t(Q1) How was accuracy computed for the \"[Number of User Interactions] vs. [Action Prediction Accuracy]\" figures?  How did the authors ensure that the task didn't somehow get easier regardless of system learning?\n\t(Q1) What justification do the authors have for their first assumption about backspaces (i.e., that they can be performed independently -- and, presumably, reliably -- of the XT2 interface).\n\t(Q2) What justification do the authors have for their implicit assumption that no backspace means positive reward?\n\t(Q3) While Table 1 is convincing that XT2 adapts to individual users, it also brings up the question as to how a model trained on _all_ user data would do. Do the authors have the ability to comment on this?\n\n\nMINOR COMMENTS:\n\t(MC1) I find it odd that the system is named \"x-to-text\" but abbreviated \"XT2.\" Should it not be \"X2T\" since one would replace the \"to\" with \"2\" (\"x 2 text\" -> \"X2T\")? I'm not taking any issue with the current name if this is how the authors meant it, rather just wondering if this is a typo.", "extra_scores": {}, "extra_text": {}}, {"review_id": "UaliB7vuNvC", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents Machine Learning based approaches that map inputs to desired actions with applications to assistive interfaces. The paper leverages user feedback to improve the predictive model to perform desired actions. The paper conducts user studies that gauge the impact of the proposed interface in contrast to a non-adaptive baseline interface. The problem is well motivated and the paper is well written.\n\nSome questions/comments:\n\n[1] What the paper refers to as “implicit” feedback, i.e., in the context of the paper, the backspace command is actually explicit. It is clear that the assumptions enforced by the paper indicate that a backspace means a strongly negative signal, and anything that has not been backspaced to be the right action. This can, for e.g., be used for training an appropriate reward regressor. This feedback however is incomplete, because, we tend to observe the feedback only for the actions presented by the interface.\n\n[2] This paper attempts to solve a problem that is highly non-trivial when the cardinality of actions grows large, for e.g., the English language vocabulary. It is unclear how one can utilize a standard yes/no feedback using a backspace key to effectively learn a highly accurate policy that presents us with the next action (for example, a word) given the context. The paper however doesn’t present a discussion surrounding these challenges, which for me is a major shortcoming of this paper.\n\n[3] Naturally, some of the assumptions considered with regards to the use of backspaces is pretty strong. The feedback model assumed by the paper is essentially noiseless in that the user always does the right thing with regards to presenting feedback on good/bad actions. The paper doesn’t validate how errors in the use of the backspace key manifest themselves in the performance of the policy learning step.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper describes a cool application of online learning from bandit feedback -- creating personalized, adaptive typing interfaces for users with sensorimotor impairments. The problem is well-motivated -- the interface can observe users' gaze (e.g. via a webcam image), predict a character as an action, and bandit feedback can be collected by observing whether users use the backspace key after the interface's action. Prior work showed that gaze-to-text can be less burdensome than typing, but this can quickly become untrue the more mistakes the interface makes. So, the goal is to personalize the interaction policy so that it makes fewer mistakes than the default interaction policy trained using a fixed dataset of expert demonstrations.

The high-point of the paper is the empirical user study with 12-60 participants -- the study convincingly demonstrates that indeed a simple bandit algorithm can improve over the default interface; moreover, users exhibit intriguing co-adaptation patterns with the adaptive interfaces. These findings may prove to be an interesting point for future studies in user co-adaptation.

The low-point of the paper is its algorithmic development. There is a vast literature on bandit/RL algorithms, and incorporating human feedback into their operation (the paper rightly cites TAMER, COACH, etc.) but it is very unclear why any one of these algorithms could not be used for the paper's application. COACH (human feedback gives an explicit view of the action's advantage -- which in the contextual bandit setting exactly matches the paper's assumptions) seems particularly appropriate. Although the algorithm proposed in the paper is simple, how applicable is it in any other context? how does it compare to COACH/etc.? when should we prefer this algorithm over others? Furthermore, given that X2T trains a reward model from observed user-behavior, a natural baseline would use an epsilon-greedy strategy (fraction of the time, pick actions greedily according to the reward model) -- this might isolate the benefit of the approximately Boltzmann exploration being conducted on top of the reward estimates in Eqn 2. Finally, since X2T trains a reward model per user it could be particularly informative to visualize what the models have learned to illustrate qualitatively how X2T is personalizing across its user base.

The paper could have a much bigger impact if the authors can figure out some creative way to enable the broader research community to work on this problem domain. A testbed or environment (like RecSim for content recommendation https://github.com/google-research/recsim) with configurable but realistic reward models could allow researchers to test several bandit algorithms, MDP vs CB formulations, other ways to interpret user feedback etc. 
Decision: Accept (Poster)