Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper presents Machine Learning based approaches that map inputs to desired actions with applications to assistive interfaces. The paper leverages user feedback to improve the predictive model to perform desired actions. The paper conducts user studies that gauge the impact of the proposed interface in contrast to a non-adaptive baseline interface. The problem is well motivated and the paper is well written.

Some questions/comments:

[1] What the paper refers to as “implicit” feedback, i.e., in the context of the paper, the backspace command is actually explicit. It is clear that the assumptions enforced by the paper indicate that a backspace means a strongly negative signal, and anything that has not been backspaced to be the right action. This can, for e.g., be used for training an appropriate reward regressor. This feedback however is incomplete, because, we tend to observe the feedback only for the actions presented by the interface.

[2] This paper attempts to solve a problem that is highly non-trivial when the cardinality of actions grows large, for e.g., the English language vocabulary. It is unclear how one can utilize a standard yes/no feedback using a backspace key to effectively learn a highly accurate policy that presents us with the next action (for example, a word) given the context. The paper however doesn’t present a discussion surrounding these challenges, which for me is a major shortcoming of this paper.

[3] Naturally, some of the assumptions considered with regards to the use of backspaces is pretty strong. The feedback model assumed by the paper is essentially noiseless in that the user always does the right thing with regards to presenting feedback on good/bad actions. The paper doesn’t validate how errors in the use of the backspace key manifest themselves in the performance of the policy learning step.

review_title: Interesting problem motivation.

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: Summary: 
This work presents a method for online learning of an assistive typing user interface (XT2) with implicit user feedback. User inputs for such an assistive typing interface are assumed to be in the form of eye gaze or handwritten characters. However, the implicit human feedback is assumed to be backspaces typed on a keyboard. Backspaces are used to delete words predicted by the assistive typing interface based on the user’s input. The online learning of such an interface to improve its assistive performance and adapt to the user over time is framed as a contextual bandit problem. A reward prediction network is trained to predict the use of backspaces (implicit feedback) by the user. This reward prediction network combined with the default interface policy using Baye’s theorem is used to update the policy of the typing interface. The experimental results with two user studies reveal that the presented method performs better than a non-adaptive default interface, stimulates user co-adaptation to the interface, and offline learning accelerates online learning.

########################


Pros:
- The paper is well-written and easy to follow.
- Strong experimental results to support the idea of the work.
- Offline pretraining of the reward model reduces the burden on online user interactions needed for the interface to be improved.
- The presented technique is applicable to any form of user inputs and the authors test their approach with two different forms of inputs (eye gaze and handwritten characters).
- Use of backspaces to train an interface with RL is a novel idea.

########################

Cons:
- The implicit feedback is assumed to be perfect and available via a keyboard. How realistic is this assumption? What happens if the feedback is noisy? Is the availability of an independent backspace action being true to the nature of the interface being truly assistive?
- It was unclear to me while reading the paper if the implicit feedback (backspaces) are provided via a keyboard or are also predicted with the user’s input (such as with gaze tracking). I assumed its the former with subtle hints in the paper and Fig 1. It would be helpful to make this explicit for the reader in Section 2.1 (as part of assumption 1).
- This approach does not model the temporal effects of learning. Practically, modeling the problem as an MDP would be more realistic versus contextual bandits.

########################

Reason for score: 
This simple approach is presented with clarity and supported with well-reported experiments (including several ablative analyses). Some minor issues in the writing could be improved but overall the idea is well-presented and well-evaluated.

########################

Questions during rebuttal:
- Sec 4.1: Is the user study conducted with the interface type variable being varied within-subjects (i.e. each user uses both default and X2T)? From the experimental results, it seems this is the case but it is not explicitly stated. If this is so, is the order in which users attempt to use the two interfaces (default and X2T) counterbalanced?
- Please address other questions raised as part of the Cons section and other feedback.

########################

Some typos and other feedback:
- Section 2,  paragraph 1, sentence 2: “… relies on an assistive typing interface to infer the user’s intended action from available inputs …” -> “… relies on an assistive typing interface, where the user’s intended action is inferred from available inputs …” 
- Algorithm 1: Consider defining what p_user(x) is. Are you making any assumptions on such a model of human user inputs (such as random sampling as suggested in Sec 4.3)? 
- Consider citing a recent work on learning from imperfect implicit user feedback such as facial expressions): Cui, Y., Zhang, Q., Allievi, A., Stone, P., Niekum, S., & Knox, W. B. (2020). The EMPATHIC Framework for Task Learning from Implicit Human Feedback. Conference on Robot Learning (CoRL), 2020.
- Even though compared to prior work, this work does not assume access to ground truth action labels from the user provided to the interface, it does assume access to ground truth backspace actions. It would be beneficial to emphasize this in paragraph 1 of Section 3.
- It only becomes clear to me by Section 3 what is meant by “handwriting as an input” and how it can be an assistive input modality. A reference to Appendix Figure 5 early on the introduction, along with highlighting this can be an easier mode of user input versus typing on a keyboard, would be helpful. 
- Section 4, paragraph 1: References to the subsection numbers can be made when stating the evaluation questions. For example: Q1 (Sec 4.1): Does X2T improve with use and learn to outperform a non-adaptive interface? The questions are well-framed and very clear though!
- Section 42: One of the insights presented for the presented method is that the XT2 interface can learn to automatically overcome calibration issues with the gaze tracker, thus the interface adapts to the mis-calibrations over time without the need for recalibration, even though external conditions would require a recalibration for better eye gaze prediction. This should be highlighted in the introduction as well.
- Section 4.3: Isn’t p_LM(u) conditioned on the preceding characters of the text seen so far? Would p_LM(u|t) be a better representation?
- Section 4.4, last paragraph: Consider reversing the order of the first two results presented (they are in opposite order to the questions posed in the previous paragraph).

review_title: Official Blind Review #3

final_score: rating: 7: Good paper, accept
review_content: SUMMARY:
The authors propose a simple algorithm for using online learning from implicit human feedback to improve systems that operate in the contextual bandit setting. The main idea is to capture the presence/absence of corrective actions and use this information to infer a reward signal that the system can use to make decisions later. The proposed method is instantiated for text-entry tasks as a system called XT2, and the authors perform extensive empirical evaluation that seems to show that the system is very successful.


STRENGTHS:
	(S1) The authors present a simple-yet-seemingly-powerful algorithm for building adaptive interfaces from implicit user feedback.
	(S2) The proposed algorithm can easily leverage existing interfaces and therefore provides an interesting path to improving all kinds of existing systems.
	(S2) The authors instantiate the algorithm in a compelling use-case scenario -- non-traditional text entry -- and demonstrate seemingly compelling results empirically.


WEAKNESSES:
	(W1) It's unclear as to whether or not the evaluation in the major plots ("[Number of User Interactions] vs. [Action Prediction Accuracy]") is meaningful. In particular, it's not clear what data was used to compute the reported prediction accuracy. It would seem that this is computed over the prediction *for that time step*, but then more detail is needed on the datasets in order to determine whether or not the task somehow got easier for XT2 over time instead of it actually learning.
	(W2) Some of the experimental results are confusing. For example, what is the explanation for the position of the red circles in Figure 2c? There seems to be very little intuitive reasoning behind where red vs. green appears -- has XT2 really learned such a complicated and sensitive decision surface? If yes, why should we "trust" this decision surface? In analyzing 2c closely, it seems like a nearest-neighbor classifier would have done just as well.
	(W3) While I greatly appreciate the authors making clear their assumptions about backspaces in Section 2.1, the authors did not justify why their first assumption ("the user can perform a backspace action independently of our interface") is valid. It would seem to me that, in situations where XT2 is necessary, that such an assumption may not actually be valid.
	(W4) In Section 2.2, the authors state that they "assign a positive reward to actions that were not backspaced," but this choice is also not justified. It seems to me that there could be cases in which the user simply accepts an incorrect action from the system without providing the feedback.


RECOMMENDATION STATEMENT:
While I have listed a number of weaknesses above, overall I really like the paper. The algorithm is simple and can build upon existing interfaces as opposed to simply replacing them. Moreover, the algorithm is built to leverage data that is easy to collect, and therefore it seems feasible that similar systems could be deployed for a number of applications. That said, the current paper is lacking in the description of the experimental results and justifications for some of the assumptions.


QUESTIONS FOR AUTHORS:
	(Q1) How was accuracy computed for the "[Number of User Interactions] vs. [Action Prediction Accuracy]" figures?  How did the authors ensure that the task didn't somehow get easier regardless of system learning?
	(Q1) What justification do the authors have for their first assumption about backspaces (i.e., that they can be performed independently -- and, presumably, reliably -- of the XT2 interface).
	(Q2) What justification do the authors have for their implicit assumption that no backspace means positive reward?
	(Q3) While Table 1 is convincing that XT2 adapts to individual users, it also brings up the question as to how a model trained on _all_ user data would do. Do the authors have the ability to comment on this?


MINOR COMMENTS:
	(MC1) I find it odd that the system is named "x-to-text" but abbreviated "XT2." Should it not be "X2T" since one would replace the "to" with "2" ("x 2 text" -> "X2T")? I'm not taking any issue with the current name if this is how the authors meant it, rather just wondering if this is a typo.
review_title: An attractive and practical algorithm for improving user interfaces

Meta-review: The paper describes a cool application of online learning from bandit feedback -- creating personalized, adaptive typing interfaces for users with sensorimotor impairments. The problem is well-motivated -- the interface can observe users' gaze (e.g. via a webcam image), predict a character as an action, and bandit feedback can be collected by observing whether users use the backspace key after the interface's action. Prior work showed that gaze-to-text can be less burdensome than typing, but this can quickly become untrue the more mistakes the interface makes. So, the goal is to personalize the interaction policy so that it makes fewer mistakes than the default interaction policy trained using a fixed dataset of expert demonstrations.

The high-point of the paper is the empirical user study with 12-60 participants -- the study convincingly demonstrates that indeed a simple bandit algorithm can improve over the default interface; moreover, users exhibit intriguing co-adaptation patterns with the adaptive interfaces. These findings may prove to be an interesting point for future studies in user co-adaptation.

The low-point of the paper is its algorithmic development. There is a vast literature on bandit/RL algorithms, and incorporating human feedback into their operation (the paper rightly cites TAMER, COACH, etc.) but it is very unclear why any one of these algorithms could not be used for the paper's application. COACH (human feedback gives an explicit view of the action's advantage -- which in the contextual bandit setting exactly matches the paper's assumptions) seems particularly appropriate. Although the algorithm proposed in the paper is simple, how applicable is it in any other context? how does it compare to COACH/etc.? when should we prefer this algorithm over others? Furthermore, given that X2T trains a reward model from observed user-behavior, a natural baseline would use an epsilon-greedy strategy (fraction of the time, pick actions greedily according to the reward model) -- this might isolate the benefit of the approximately Boltzmann exploration being conducted on top of the reward estimates in Eqn 2. Finally, since X2T trains a reward model per user it could be particularly informative to visualize what the models have learned to illustrate qualitatively how X2T is personalizing across its user base.

The paper could have a much bigger impact if the authors can figure out some creative way to enable the broader research community to work on this problem domain. A testbed or environment (like RecSim for content recommendation https://github.com/google-research/recsim) with configurable but realistic reward models could allow researchers to test several bandit algorithms, MDP vs CB formulations, other ways to interpret user feedback etc. 
Decision: Accept (Poster)