Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper aims to solve the problem of human-machine interface with supervision or prior knowledge about the user’s desired tasks. This is challenging due to the large space of interface designs. This paper proposes a reinforcement-based interface update method by maximizing the mutual information between the user’s commands and the induced state transition, where an underlying key idea is that “the more intuitive the interface, the less noisy the user’s commands.” The main contribution is the design of mutual information objective/reward for enhancing interfaces. The experiment section also shows several studies to demonstrate the effectiveness of the updated interfaces and application examples. 
strengths_and_weaknesses: The main strength of this paper is that the intuitive design of mutual information rewards considering the user’s influence on the next state and its opposite. 

The major weakness of this paper is there is no comparison with baselines. It is unclear whether the given dataset/problem is easy to design interface. Further, it is still hard to clearly confirm the correlation between the mutual information and true reward over results, particularly in Fig. 2. 

questions: - It would be great why algorithm 1 uses a validation set for the return computation. Further, in this work, the authors can obtain a small amount of data. In general, the size of the validation set will be smaller than that of the training set. How does it affect the performance?

- Figure 2 requires clarifications. First, it is hard to see the correlation between mutual information and true reward in Fig. 2(a)-(c). Although Fig. 2(e) and (f) show the Spearman correlation coefficient (i.e., 0.43 and 0.69), it is not clear how much the value is significant to say the two values of information are correlated. 

- Second, Fig. 2(d) shows a large number of time offsets that may result in very long time gaps between observed states. It should clarify how it affects the data collection, estimator update, etc considering the real-time human-machine interaction. 

- Any comparison with baselines will be helpful in understanding and also support the proposed method.  

limitations: N/A
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper presents a method for building an adaptive user interface in an unsupervised manner. The key idea is to use the mutual information between the user input and the state transition. Based on this idea, the authors propose an algorithm to maximize the mutual information lower bound from a small amount of user operation data. The proposed approach is evaluated using five existing dataset, real-world user study data, and another expert user demonstration.
strengths_and_weaknesses: ### Strength
- The unsupervised user adaptation task setting is interesting, and the idea of using mutual information content seems reasonable.
- Experimental evaluations have been conducted from multiple perspectives, demonstrating at a minimum the effectiveness of the proposed method.

### Weaknesses
- While offline evaluation using existing data makes sense, evaluating the actual use of adaptively changing interfaces with this approach is not necessarily sufficient. The cursor control task seems a bit easier than the "unknown" interface that this study assumes. Multiple user experiments on diverse interfaces with similar or greater participants would increase the method's reliability. The author's demonstration is somewhat unreliable as experimental evidence.
- As for the method's claim that it does not require prior knowledge about the desired task, it does not appear to be strictly substantiated, as there are no corresponding experiments.
- Considering the purpose of the study, what should be shown is the advantage of the system compared to the user's solo adaptation to the system. However, the experiment mainly only compares the initial state and after adaptation, and there is no discussion of what happens after the participants use the baseline interface for an extended period.
questions: - I would like a more detailed explanation and discussion on the last point I raised above. Are there any results that suggest that the proposed method can complete the desired task more quickly or with greater user satisfaction than a scenario in which users continue to use a non-adaptive interface and become proficient with it?
limitations: - There is an experimental discussion on interfaces where the proposed method cannot be successfully applied, which is a good point for the paper.
- Although the study involved numerous user experiments, there was no specific mention of a process for obtaining consent from participants or for ethical review.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents a reinforcement learning algorithm that uses mutual information as a proxy reward for achieving understanding between a computer with a randomly perturbed UI and a human trying to accomplish a task using that UI, without knowing how it is perturbed.  The only feedback (signal) to the algorithm is the entropy of the humans movements/inputs where it is assumed that the human would be making more random/chaotic inputs in a sense proportional to how unexpectedly (unintuitively) the game’s state changed with respect to the action taken by the user.  The main evaluation is with a space ship control game where the actual motion of the ship was perturbed/offset initially by some random theta degrees.  The algorithm eventually learns to reduce theta either to zero or -180 degrees so that it exactly follows the human’s command.  Only at these two stable points does the human stop acting so randomly in response to the state changes, since the state changes are, apparently and acceptable reaction to the action (so further exploration by the human is not required).      

As the authors state: “the key idea  in this paper is that, regardless of the task, when an interface is more intuitive,
the user’s commands are less noisy.”  They formalize this “noise” as “the mutual information between
the user’s command signals and the induced state transitions in the environment.”  Their work is in line with recent literature in user empowerment where the machine does not assume or infer the users’ goals but only uses cues that indicate the user’s satisfaction with the result as a reward upon which to learn in this human-in-the-loop system.


strengths_and_weaknesses: The key strength of the paper is that it is interesting and the idea that users will express frustration with a poorly performing interface by random frantic exploration of potential alternative commands that might work seems plausible and is supported by the authors first study that shows that their mutual information pseudo-reward is correlated with interfaces that adapt to meet the users expectations (become more intuitive)  As the authors state: “the key idea  in this paper is that, regardless of the task, when an interface is more intuitive,
the user’s commands are less noisy.”  The authors give reasonable proof of the fact that with adaptive interfaces, users do seem to exhibit the behavior of taking more noisy actions when the resulting effects in the environment (state changes) are not what would be intuitively expected.  The authors do include a user study with a simple adaptive interface that shows that their mutual information pseudo-reward can drive convergence in this case.  The authors also show that a knowledgeable user can drive adaptation using the pseudo-reward.

As the author states there has been a number of attempts to find “natural” reward signals in the real world when interacting with humans whether these be through NLP, BCI (EEG), facial expressions and physiological signals.  None of these are as easy to obtain as the “noise” in the input that these authors leverage, making this perhaps the most useable of all prior proposed methods in the field.

The weakness of the paper is that the use case is not immediately compelling.  The first contact scenario is a good framing for this, however unlikely, but in reality, this is a more theoretical method for machines to learn preferences from humans.  The more compelling use case is probably in applying this as a feedback mechanism to search engines or to personalize the “fit” of something like exoskeletons or prosthetic limbs.  The user studies are not exhaustive, but I would not expect that for a contribution that is fundamentally about the algorithm.  It is laudable that the authors disclosed that the singular subject in the feasibility study was the first author and  while on the one hand it could be argued that having such an informed participant as an evaluator made the task “easier” for the machine (potentially the author knew to proportionally vary the randomness of the responses in a way that would facilitate convergence) it is not entirely to be dismissed especially in considering a case where a person was trying to “fit” or “train” a machine intentionally knowing how the machine learned through randomness.  The authors show that this is a naturalistic response in general, so intensifying it intentionally would not be entirely out of line with user behaviour.

Also, I believe the term “co-adaptive” in the title is a bit misleading as you do mention that the algorithm has no ability to adapt the behavior of the human.  I believe you are saying that the human naturally seems to adapt their behavior in response to the algorithms “correctness” and that you are relying on the existence of this adaptation otherwise you pseudo reward would not work, but maybe clarifying this a bit would be good.


questions: Out of curiosity did you consider any other metrics of noise assessment, e.g. standard deviation of responses, length of time of “attempted corrective” responses, etc. and how did these compare to mutual information?

Was there anything special about you users, were they graduate students in computer science, friends, co-workers? Were they motivated in some way to complete the task?   I can imagine the “average” user simply abandoning a malfunctioning interface.  

limitations: This contribution is limited to the space of adaptive interfaces where the correct settings for the user’s task cannot be known in advance but where the parameters of the controlling algorithm have the ability to adapt themselves to a more optimal setting.  The assumptions are reasonable but not exhaustively studied.  The process of adaptation is comparatively slow compared to other methods, so a stronger motivating use case would have been better.  The method could have been more exhaustively compared versus other human-in the loop methods such as NLP or facial expression analysis or simply giving the user an explicit “performance feedback” channel on the algorithm’s performance (e.g. 1 to 10 as a keyboard input) and evaluated with respect to speed of convergence and user satisfaction (e.g. if could take longer yet users might prefer it as more natural.
review_title: null

Meta-review: metareview: The paper describes an approach to learning an adaptive user interface (i.e., mapping raw inputs to the agent's actions) in an unsupervised way via reinforcement learning. The goal is to learn interfaces that are intuitive for the user, with the supposition that the user's inputs become less noisy as the interface becomes more intuitive. To that end, the proposal is to use the mutual information between the raw input provided by the user and the resulting state transitions as a reward proxy. The approach is evaluated on a series of control and typing domains as well as a small-scale user study involving a cursor control task.

The paper was reviewed by three researchers who read the author response an discussed the paper with the AC. The reviewers agree that the problem of adapting a user interface in an unsupervised way is interesting and the proposed use of mutual information for adaptation is sensible and interesting. The reviewers initially raised concerns about the absence of a compelling use case and the experimental evaluations, notably the lack of appropriate baselines (Reviewer eRaa) and inadequate experiments (Reviewers eRaa and phLr). The authors made a concerted effort to address most of the reviewers' concerns, which included experiments conduced on the cursor domain using the alternative method suggested by Reviewer eRaa. However, the authors did not address the experimentation issues raised by Reviewer phLr, who finds that the paper lacks experimental evidence for some of the claims being made. As it stands, the paper doesn't show that the interface that can be achieved with this approach is truly intuitive. Making such a claim requires comparative experiments with appropriate baseline interfaces and more detailed user analyses. As such a detailed set of user studies may out-of-scope for a conference-length algorithms paper focused on the use of mutual information as a reward proxy for interface learning, the claims in the paper should be revisited.
confidence: Less certain
Decision: Accept