X2T: Training an X-to-Text Typing Interface with Online Learning from User Feedback

1 INTRODUCTION . Recent advances in user interfaces have enabled people with sensorimotor impairments to more effectively communicate their intent to machines . For example , Ward et al . ( 2000 ) enable users to type characters using an eye gaze tracker instead of a keyboard , and Willett et al . ( 2020 ) enable a paralyzed human patient to type using a brain implant that records neural activity . The main challenge in building such interfaces is translating high-dimensional , continuous user input into desired actions . Standard methods typically calibrate the interface on predefined training tasks for which expert demonstrations are available , then deploy the trained interface . Unfortunately , this does not enable the interface to improve with use or adapt to distributional shift in the user inputs . In this paper , we focus on the problem of assistive typing : helping a user select words or characters without access to a keyboard , using eye gaze inputs ( Ward et al. , 2000 ) ; handwriting inputs ( see Figure 7 in the appendix ) , which can be easier to provide than direct keystrokes ( Willett et al. , 2020 ) ; or inputs from an electrocorticography-based brain implant ( Leuthardt et al. , 2004 ; Silversmith et al. , 2020 ) . To enable any existing , default interface to continually adapt to the user , we train a model using online learning from user feedback . The key insight is that the user provides feedback on the interface ’ s actions via backspaces , which indicate that the interface did not perform the desired action in response to a given input . By learning from this naturally-occurring feedback signal instead of an explicit label , we do not require any additional effort from the user to improve the interface . Furthermore , because our method is applied on top of the user ’ s default interface , our approach is complementary to other work that develops state-of-the-art , domain-specific methods for problems like gaze tracking and handwriting recognition . Figure 1 describes our algorithm : we initialize our model using offline data generated by the default interface , deploy our interface as an augmentation to the default interface , collect online feedback , and update our model . We formulate assistive typing as an online decision-making problem , in which the interface receives observations of user inputs , performs actions that select words or characters , and receives a reward signal that is automatically constructed from the user ’ s backspaces . To improve the default interface ’ s actions , we fit a neural network reward model that predicts the reward signal given the user ’ s input and the interface ’ s action . Upon observing a user input , our interface uses the trained reward model to update the prior policy given by the default interface to a posterior policy conditioned on optimality , then samples an action from this posterior ( see Figure 1 ) . We call this method x-to-text ( X2T ) , where x refers to the arbitrary type of user input ; e.g. , eye gaze or brain activity . Our primary contribution is the X2T algorithm for continual learning of a communication interface from user feedback . We primarily evaluate X2T through an online user study with 12 participants who use a webcam-based gaze tracking system to select words from a display . To run ablation experiments that would be impractical in the online study , we also conduct an observational study with 60 users who use a tablet and stylus to draw pictures of individual characters . The results show that X2T quickly learns to map input images of eye gaze or character drawings to discrete word or character selections . By learning from online feedback , X2T improves upon a default interface that is only trained once using supervised learning and , as a result , suffers from distribution shift ( e.g. , caused by changes in the user ’ s head position and lighting over time in the gaze experiment ) . X2T automatically overcomes calibration problems with the gaze tracker by adapting to the miscalibrations over time , without the need for explicit re-calibration . Furthermore , X2T leverages offline data generated by the default interface to accelerate online learning , stimulates co-adaptation from the user in the online study , and personalizes the interface to the handwriting style of each user in the observational study . 2 LEARNING TO INFER INTENT FROM USER INPUT . In our problem setting , the user can not directly perform actions ; e.g. , due to a sensorimotor impairment . Instead , the user relies on an assistive typing interface , where the user ’ s intended action is inferred from available inputs such as webcam images of eye gaze or handwritten character drawings . As such , we formulate assistive typing as a contextual bandit problem ( Langford & Zhang , 2008 ; Yue & Joachims , 2009 ; Li et al. , 2010 ; Lan & Baraniuk , 2016 ; Gordon et al. , 2019 ) . At each timestep , the user provides the interface with a context x ∈ X , where X is the set of possible user inputs ( e.g. , webcam images ) . The interface then performs an action u ∈ U , where U is the set of possible actions ( e.g. , word selections ) . We assume the true reward function is unknown , since the user can not directly specify their desired task ( e.g. , writing an email or filling out a form ) . Instead of eliciting a reward function or explicit reward signal from the user , we automatically construct a reward signal from the user ’ s backspaces . The key idea is to treat backspaces as feedback on the accuracy of the interface ’ s actions . Our approach to this problem is outlined in Figure 1 . We aim to minimize expected regret , which , in our setting , is characterized by the total number of backspaces throughout the lifetime of the interface . While a number of contextual bandit algorithms with lower regret bounds have been proposed in prior work ( Lattimore & Szepesvári , 2020 ) , we use a simple strategy that works well in our experiments : train a neural network reward model to predict the reward given the user ’ s input and the interface ’ s action , and select actions with probability proportional to their predicted optimality . Our approach is similar to prior work on deep contextual multi-armed bandits ( Collier & Llorens , 2018 ) and NeuralUCB ( Zhou et al. , 2019 ) , except that instead of using Thompson sampling or UCB to balance exploration and exploitation , we use a simple , stochastic policy . 2.1 MODELING USER BEHAVIOR AND FEEDBACK . Unlike in the standard multi-armed bandit framework , we do not get to observe an extrinsic reward signal that captures the underlying task that the user aims to perform . To address this issue , we infer rewards from naturally-occurring user behavior . In particular , in the assistive typing setting , we take advantage of the fact that we can observe when the user backspaces ; i.e. , when they delete the most recent word or character typed by the interface . To infer rewards from backspaces , we make two assumptions about user behavior : ( 1 ) the user can perform a backspace action independently of our interface ( e.g. , by pressing a button ) ; ( 2 ) the user tends to backspace incorrect actions ; and ( 3 ) the user does not tend to backspace correct actions . Hence , we assign a positive reward to actions that were not backspaced , and assign zero reward to backspaced actions . Formally , let r ∈ { 0 , 1 } denote this reward signal , where r = 0 indicates an incorrect action and r = 1 indicates a correct action . 2.2 TRAINING THE REWARD MODEL TO PREDICT FEEDBACK . In order to perform actions that minimize expected regret – i.e. , the total number of backspaces over time – we need to learn a model that predicts whether or not the user will backspace a given action in a given context . To do so , we learn a reward model pθ ( r|x , u ) , where pθ is a neural network and θ are the weights . Since the reward r ∈ { 0 , 1 } can only take on one of two values , pθ is a binary classifier . We train this binary classifier on a dataset D of input-action-reward triples ( x , u , r ) . In particular , we fit the model pθ by optimizing the maximum-likelihood objective ; i.e. , the binary cross-entropy loss ( see Equation 2 in the appendix ) . Since X2T learns from human-in-the-loop feedback , the amount of training data is limited by how frequently the user operates the interface . To reduce the amount of online interaction data needed to train the reward model , we use offline pretraining . We assume that the user already has access to some default interface for typing . We also assume access to an offline dataset of input-action pairs generated by the user and this default interface . We assign zero rewards to the backspaced actions and positive rewards to the non-backspaced actions in this offline dataset , and initially train our reward model to predict these rewards given the user ’ s inputs and the default interface ’ s actions . Thus , when X2T is initially deployed , the reward model has already been trained on the offline data , and requires less online interaction data to reach peak accuracy . Algorithm 1 X-to-Text ( X2T ) Require π̄ , θinit . default interface , pretrained reward model parameters while true do x ∼ puser ( x ) . user gives input u ∼ π ( u|x ) ∝ pθ ( r = 1|x , u ) π̄ ( u|x ) . interface performs action r ← 0 if user backspaces else 1 . infer reward from user feedback D ← D ∪ { ( x , u , r ) } . store online input-action-reward data θ ← θ +∇θ ∑ ( x , u , r ) ∼D log ( pθ ( r|x , u ) ) . update reward model w/SGD 2.3 USING THE REWARD MODEL TO SELECT ACTIONS . Even with offline pretraining , the initial reward model may not be accurate enough for practical use . To further improve the initial performance of our interface at the onset of online training , we combine our reward model pθ ( r|x , u ) with the default interface π̄ ( u|x ) . We assume that π̄ is a stochastic policy and that we can evaluate it on specific inputs , but do not require access to its implementation . We set our policy π ( u|x ) = p ( u|x , r = 1 ) to be the probability of an action conditional on optimality , following the control-as-inference framework ( Levine , 2018 ) . Applying Bayes ’ theorem , we get p ( u|x , r = 1 ) ∝ p ( r = 1|x , u ) p ( u|x ) . The first term is given by our reward model pθ , and the second term is given by the default interface . Combining these , we get the policy π ( u|x ) ∝ pθ ( r = 1|x , u ) π̄ ( u|x ) . ( 1 ) This decomposition of the policy improves the initial performance of our interface at the onset of online training , and guides exploration for training the reward model . It also provides a framework for incorporating a language model into our interface , as described in Section 4.3 . Our x-to-text ( X2T ) method is summarized in Algorithm 1 . In the beginning , we assume the user has already been operating the default interface π̄ for some time . In doing so , they generate an ‘ offline ’ dataset that we use to train the initial reward model parameters θinit . When the user starts using X2T , our interface π already improves upon the default interface π̄ by combining the default interface with the initial reward model via Equation 1 . As the user continues operating our interface , the resulting online data is used to maintain or improve the accuracy of the reward model . At each timestep , the user provides the interface with input x ∼ puser ( x ) . Although standard contextual bandit methods assume that the inputs x are i.i.d. , we find that X2T performs well even when the inputs are correlated due to user adaptation ( see Section 4.2 ) or the constraints of natural language ( see Section 4.4 ) . The interface then uses the policy in Equation 1 to select an action u . We then update the reward model pθ , by taking one step of stochastic gradient descent to optimize the maximum-likelihood objective in Equation 2 . Appendix A.1 discusses the implementation details .

Summary: This work presents a method for online learning of an assistive typing user interface (XT2) with implicit user feedback. User inputs for such an assistive typing interface are assumed to be in the form of eye gaze or handwritten characters. However, the implicit human feedback is assumed to be backspaces typed on a keyboard. Backspaces are used to delete words predicted by the assistive typing interface based on the user’s input. The online learning of such an interface to improve its assistive performance and adapt to the user over time is framed as a contextual bandit problem. A reward prediction network is trained to predict the use of backspaces (implicit feedback) by the user. This reward prediction network combined with the default interface policy using Baye’s theorem is used to update the policy of the typing interface. The experimental results with two user studies reveal that the presented method performs better than a non-adaptive default interface, stimulates user co-adaptation to the interface, and offline learning accelerates online learning.