X2T: Training an X-to-Text Typing Interface with Online Learning from User Feedback

1 INTRODUCTION . Recent advances in user interfaces have enabled people with sensorimotor impairments to more effectively communicate their intent to machines . For example , Ward et al . ( 2000 ) enable users to type characters using an eye gaze tracker instead of a keyboard , and Willett et al . ( 2020 ) enable a paralyzed human patient to type using a brain implant that records neural activity . The main challenge in building such interfaces is translating high-dimensional , continuous user input into desired actions . Standard methods typically calibrate the interface on predefined training tasks for which expert demonstrations are available , then deploy the trained interface . Unfortunately , this does not enable the interface to improve with use or adapt to distributional shift in the user inputs . In this paper , we focus on the problem of assistive typing : helping a user select words or characters without access to a keyboard , using eye gaze inputs ( Ward et al. , 2000 ) ; handwriting inputs ( see Figure 7 in the appendix ) , which can be easier to provide than direct keystrokes ( Willett et al. , 2020 ) ; or inputs from an electrocorticography-based brain implant ( Leuthardt et al. , 2004 ; Silversmith et al. , 2020 ) . To enable any existing , default interface to continually adapt to the user , we train a model using online learning from user feedback . The key insight is that the user provides feedback on the interface ’ s actions via backspaces , which indicate that the interface did not perform the desired action in response to a given input . By learning from this naturally-occurring feedback signal instead of an explicit label , we do not require any additional effort from the user to improve the interface . Furthermore , because our method is applied on top of the user ’ s default interface , our approach is complementary to other work that develops state-of-the-art , domain-specific methods for problems like gaze tracking and handwriting recognition . Figure 1 describes our algorithm : we initialize our model using offline data generated by the default interface , deploy our interface as an augmentation to the default interface , collect online feedback , and update our model . We formulate assistive typing as an online decision-making problem , in which the interface receives observations of user inputs , performs actions that select words or characters , and receives a reward signal that is automatically constructed from the user ’ s backspaces . To improve the default interface ’ s actions , we fit a neural network reward model that predicts the reward signal given the user ’ s input and the interface ’ s action . Upon observing a user input , our interface uses the trained reward model to update the prior policy given by the default interface to a posterior policy conditioned on optimality , then samples an action from this posterior ( see Figure 1 ) . We call this method x-to-text ( X2T ) , where x refers to the arbitrary type of user input ; e.g. , eye gaze or brain activity . Our primary contribution is the X2T algorithm for continual learning of a communication interface from user feedback . We primarily evaluate X2T through an online user study with 12 participants who use a webcam-based gaze tracking system to select words from a display . To run ablation experiments that would be impractical in the online study , we also conduct an observational study with 60 users who use a tablet and stylus to draw pictures of individual characters . The results show that X2T quickly learns to map input images of eye gaze or character drawings to discrete word or character selections . By learning from online feedback , X2T improves upon a default interface that is only trained once using supervised learning and , as a result , suffers from distribution shift ( e.g. , caused by changes in the user ’ s head position and lighting over time in the gaze experiment ) . X2T automatically overcomes calibration problems with the gaze tracker by adapting to the miscalibrations over time , without the need for explicit re-calibration . Furthermore , X2T leverages offline data generated by the default interface to accelerate online learning , stimulates co-adaptation from the user in the online study , and personalizes the interface to the handwriting style of each user in the observational study . 2 LEARNING TO INFER INTENT FROM USER INPUT . In our problem setting , the user can not directly perform actions ; e.g. , due to a sensorimotor impairment . Instead , the user relies on an assistive typing interface , where the user ’ s intended action is inferred from available inputs such as webcam images of eye gaze or handwritten character drawings . As such , we formulate assistive typing as a contextual bandit problem ( Langford & Zhang , 2008 ; Yue & Joachims , 2009 ; Li et al. , 2010 ; Lan & Baraniuk , 2016 ; Gordon et al. , 2019 ) . At each timestep , the user provides the interface with a context x ∈ X , where X is the set of possible user inputs ( e.g. , webcam images ) . The interface then performs an action u ∈ U , where U is the set of possible actions ( e.g. , word selections ) . We assume the true reward function is unknown , since the user can not directly specify their desired task ( e.g. , writing an email or filling out a form ) . Instead of eliciting a reward function or explicit reward signal from the user , we automatically construct a reward signal from the user ’ s backspaces . The key idea is to treat backspaces as feedback on the accuracy of the interface ’ s actions . Our approach to this problem is outlined in Figure 1 . We aim to minimize expected regret , which , in our setting , is characterized by the total number of backspaces throughout the lifetime of the interface . While a number of contextual bandit algorithms with lower regret bounds have been proposed in prior work ( Lattimore & Szepesvári , 2020 ) , we use a simple strategy that works well in our experiments : train a neural network reward model to predict the reward given the user ’ s input and the interface ’ s action , and select actions with probability proportional to their predicted optimality . Our approach is similar to prior work on deep contextual multi-armed bandits ( Collier & Llorens , 2018 ) and NeuralUCB ( Zhou et al. , 2019 ) , except that instead of using Thompson sampling or UCB to balance exploration and exploitation , we use a simple , stochastic policy . 2.1 MODELING USER BEHAVIOR AND FEEDBACK . Unlike in the standard multi-armed bandit framework , we do not get to observe an extrinsic reward signal that captures the underlying task that the user aims to perform . To address this issue , we infer rewards from naturally-occurring user behavior . In particular , in the assistive typing setting , we take advantage of the fact that we can observe when the user backspaces ; i.e. , when they delete the most recent word or character typed by the interface . To infer rewards from backspaces , we make two assumptions about user behavior : ( 1 ) the user can perform a backspace action independently of our interface ( e.g. , by pressing a button ) ; ( 2 ) the user tends to backspace incorrect actions ; and ( 3 ) the user does not tend to backspace correct actions . Hence , we assign a positive reward to actions that were not backspaced , and assign zero reward to backspaced actions . Formally , let r ∈ { 0 , 1 } denote this reward signal , where r = 0 indicates an incorrect action and r = 1 indicates a correct action . 2.2 TRAINING THE REWARD MODEL TO PREDICT FEEDBACK . In order to perform actions that minimize expected regret – i.e. , the total number of backspaces over time – we need to learn a model that predicts whether or not the user will backspace a given action in a given context . To do so , we learn a reward model pθ ( r|x , u ) , where pθ is a neural network and θ are the weights . Since the reward r ∈ { 0 , 1 } can only take on one of two values , pθ is a binary classifier . We train this binary classifier on a dataset D of input-action-reward triples ( x , u , r ) . In particular , we fit the model pθ by optimizing the maximum-likelihood objective ; i.e. , the binary cross-entropy loss ( see Equation 2 in the appendix ) . Since X2T learns from human-in-the-loop feedback , the amount of training data is limited by how frequently the user operates the interface . To reduce the amount of online interaction data needed to train the reward model , we use offline pretraining . We assume that the user already has access to some default interface for typing . We also assume access to an offline dataset of input-action pairs generated by the user and this default interface . We assign zero rewards to the backspaced actions and positive rewards to the non-backspaced actions in this offline dataset , and initially train our reward model to predict these rewards given the user ’ s inputs and the default interface ’ s actions . Thus , when X2T is initially deployed , the reward model has already been trained on the offline data , and requires less online interaction data to reach peak accuracy . Algorithm 1 X-to-Text ( X2T ) Require π̄ , θinit . default interface , pretrained reward model parameters while true do x ∼ puser ( x ) . user gives input u ∼ π ( u|x ) ∝ pθ ( r = 1|x , u ) π̄ ( u|x ) . interface performs action r ← 0 if user backspaces else 1 . infer reward from user feedback D ← D ∪ { ( x , u , r ) } . store online input-action-reward data θ ← θ +∇θ ∑ ( x , u , r ) ∼D log ( pθ ( r|x , u ) ) . update reward model w/SGD 2.3 USING THE REWARD MODEL TO SELECT ACTIONS . Even with offline pretraining , the initial reward model may not be accurate enough for practical use . To further improve the initial performance of our interface at the onset of online training , we combine our reward model pθ ( r|x , u ) with the default interface π̄ ( u|x ) . We assume that π̄ is a stochastic policy and that we can evaluate it on specific inputs , but do not require access to its implementation . We set our policy π ( u|x ) = p ( u|x , r = 1 ) to be the probability of an action conditional on optimality , following the control-as-inference framework ( Levine , 2018 ) . Applying Bayes ’ theorem , we get p ( u|x , r = 1 ) ∝ p ( r = 1|x , u ) p ( u|x ) . The first term is given by our reward model pθ , and the second term is given by the default interface . Combining these , we get the policy π ( u|x ) ∝ pθ ( r = 1|x , u ) π̄ ( u|x ) . ( 1 ) This decomposition of the policy improves the initial performance of our interface at the onset of online training , and guides exploration for training the reward model . It also provides a framework for incorporating a language model into our interface , as described in Section 4.3 . Our x-to-text ( X2T ) method is summarized in Algorithm 1 . In the beginning , we assume the user has already been operating the default interface π̄ for some time . In doing so , they generate an ‘ offline ’ dataset that we use to train the initial reward model parameters θinit . When the user starts using X2T , our interface π already improves upon the default interface π̄ by combining the default interface with the initial reward model via Equation 1 . As the user continues operating our interface , the resulting online data is used to maintain or improve the accuracy of the reward model . At each timestep , the user provides the interface with input x ∼ puser ( x ) . Although standard contextual bandit methods assume that the inputs x are i.i.d. , we find that X2T performs well even when the inputs are correlated due to user adaptation ( see Section 4.2 ) or the constraints of natural language ( see Section 4.4 ) . The interface then uses the policy in Equation 1 to select an action u . We then update the reward model pθ , by taking one step of stochastic gradient descent to optimize the maximum-likelihood objective in Equation 2 . Appendix A.1 discusses the implementation details .

Summary: The authors propose a simple algorithm for using online learning from implicit human feedback to improve systems that operate in the contextual bandit setting. The main idea is to capture the presence/absence of corrective actions and use this information to infer a reward signal that the system can use to make decisions later. The proposed method is instantiated for text-entry tasks as a system called XT2, and the authors perform extensive empirical evaluation that seems to show that the system is very successful.