Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper looks at the setting where the goal is to learn a policy without access to a reward from the environment, with access only to feedback. Previous work (Xie et al (2021)) assumes that the feedback does not have the action. This paper removes that assumption and proposes a new contrastive approach that can integrate feedback with actions to learn a policy. Through different empirical evaluations (toy MNIST task, and large-scale synthetic eval on OpenML), the authors show how this approach can learn from feedback that has the actions taken by the agent.
strengths_and_weaknesses: Strengths

- The paper removes some assumptions introduced in previous works to bring algorithms closer to being realised in realistic environments.
- Learning from non-scalar rewards and feedback is an important direction and this work makes progress towards making this possible.
- The paper has the appropriate ablations and model comparisons.

Weaknesses

I am not very familiar with the relevant literature in the subfield. As someone from a related but outside the subfield, here are some potential weaknesses. My main concerns lie with the writing of the paper.

- Paper is not clearly written. The introduction does not motivate the central problem well. I would have liked to see a few examples where previous methods fail, but the paper is able to bridge the gap. The authors add these to the conclusion, but it might be useful to add examples to the introduction to motivate the problem. Section 3 is especially difficult to understand as there is a lot of notation with little intuition.
- The empirical evaluations aren’t described well. I had to refer to Xie et al. (2021) to get a better idea of what the evaluations look like. Similarly, the previous paper also does a much better job of describing and motivating different applications.
- The empirical evaluations are limited; assuming very simple policies and representations. These would be difficult to realize in realistic settings that the authors use as examples.
- The intuition behind the feedback signal for the MNIST classification signal is not clear.
- The empirical evaluations seem detached from potential practical applications.
- Feedback is often noisy; the authors do not consider this scenario.
- There is a lack of discussion of why a CB setting is better than an MDP formulation for the tasks proposed.
questions: How does this work relate to learning from interaction works in robotics / HRI and human-in-the-loop learning?
limitations: The authors’ discuss the potential negative impact of the work well. Limitations of the approach and testing might be lacking; see weaknesses.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper studies the problem of interaction-grounded learning, where an agent takes an action a given context x, and receives a feedback vector y in return. A reward (to be optimized) is also generated from the environment, but not revealed to the learning process. The goal is to learn a policy that optimizes reward without explicitly observing reward from the environment, by learning a mapping from Y to the space of rewards, and learning a value function mapping X and A to [0, 1]. An existing challenge of this setting is that IGL fails when y contains information about the executed action a. This paper proposes a contrastive learning approach that allows IGL to succeed even when y contains information about a. Experiments on several RL benchmarks show improvements over IGL assuming conditional independence of y given a, nearing performance of contextual bandit (which assumes full access to reward).
strengths_and_weaknesses: Strengths:

* Paper is relatively clear and easy to follow. As someone who doesn't have much practice with reading theory-heavy papers (unfortunately this paper is a few steps out of my area), I found it easy to follow. I found the information-theoretic arguments intuitive.
* Main contributions have potential broad significance (e.g., for applications as discussed in the introduction)

--- 

Weaknesses:

* See assumptions below -- discussion on unstated assumptions (if any) and their limits on real-world applications would be nice to have.
* I found the format of Figure 2 somewhat confusing; to me the y-axis of the graphs seems to imply some kind of sequential process
questions: Questions about some other assumptions that seem to be made in the paper:
* Does this algorithm / setting assume a discrete action space? What happens if the action space is continuous?
* Does the setting assume that y is deterministic given (x, a)? It seems like no, given the experiments.
* Is there an assumption that there exists an optimal \Psi, that can map any (y, a) to the correct reward? I suppose if y is deterministic given r, then yes. But it seems like that's not the case given the experiments.
* Does the algorithm require executing all actions in all states in the batch data D? If so, seems like another assumption that might break in real-world settings.
* In Table 1, action-exclusive feedback is just the image of the reward, as in Figure 2?
* Images y in 5.1 are randomly chosen from the dataset?
limitations: See above about limitations / assumptions for applications to real-world settings.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper explores a straightforward reformulation of a newly introduced formalism — Interaction-Grounded Learning (IGL) — in which an agent must learn to act optimally in an environment only given access to some context, an action space, and per-step feedback that is some function of a latent reward function. 

Whereas prior work formalizes the feedback component of IGL as conditionally independent of the context and language given the hidden rewards, this work (1) shows that this prior conditional independence assumption is too strong, leading to pathological failure modes when “action” information is present in the feedback, (2) presents a new formulation with less restrictive conditional independence assumptions that condition feedback on both latent reward AND executed action, and (3) presents a learning algorithm for deriving good policies given the new formulation and feedback. 

Experiments on a modified MNIST task where the (context, action) are (image, label) respectively, and high-dimensional feedback is given as some image y where the digit shown is taken as (action + 6 * binary-reward - 3 modulo 10), and a similar construction for the various OpenML CC-18 datasets shows that learning with the proposed algorithm under the new conditional independence assumptions outperforms learning with the prior assumptions in cases where action-information is present in the feedback. 
strengths_and_weaknesses: The key strength of this paper is in its clarity; at its core it takes an existing formalism (“full conditional independence Interaction-Grounded Learning”), finds a key failure mode, and adjusts the formalism with new, less restrictive assumptions. Each of the steps taken along the way are well-motivated, and the proofs tied to the new assumptions (separability, access to a baseline policy for symmetry breaking) are clean and relevant.

Unfortunately, the weaknesses of this paper are in the evaluation, and the underlying motivation behind this work; the introduction and discussion/conclusion motivate that this type of “Interaction Grounded Learning” is **critical** for scenarios in multimodal interactive feedback in human-computer interaction and in designing brain-computer interaction interfaces. Especially for the latter, the problem of having feedback data that is tied to action information is already a key problem in fMRI information orthogonalization (line 340), and for eye tracker recalibration for ALS patients (line 341). Unfortunately, none of the existing evaluations reflect these real-world use cases, instead constructing synthetic tasks based on MNIST or open classification benchmarks — tasks that are hard to understand.

For example, it’s hard to make the leap to these well-motivated, clear tasks from the current MNIST classification experiments where feedback is an arbitrary formula of the “right label” and the “predicted action” — in the paper experiments, the feedback provided an agent is a digit with label = (action + 6 * binary-reward - 3 modulo 10). In general, it’s not clear why the action-intermingling with feedback is an action problem in real-world settings, or why other, simpler approaches for learning couldn’t learn to decouple this information.

EDIT: The new revision does address this a little bit, so I am updating my score.
questions: - Notation in (1) is a bit unclear; L(\pi, \psi) seems to indicate a loss, but you’re actually trying to maximize the difference in Values right? Might it be better to call it J(\pi, \psi)?
- On line 115, it’s not clear where the augmented data is actually coming from; how are you generating the marginal distributions \mu(x) and\mu(y)? Especially if the data coming in is online interaction data... this feels rather strong? Or am I misunderstanding?
- In general - concrete examples to ground out the front half of the paper would be greatly appreciated!

limitations: In general this paper is severely limited by its evaluation. The paper seeks to show that the initial assumptions in the interaction-grounded learning framework are prohibitive and lead to failure modes, designing a new formulation of the framework’s assumptions, as well as a new learning algorithm to fix this. However, by only evaluating synthetic, toy constructed tasks with arbitrary feedback that is not realistic, it’s not clear that this is an actual problem.

This paper would be considerably stronger if evaluated on the motivating tasks in brain-computer interfaces or human-computer interaction used in the introduction and conclusion, and evaluating real-world instances of where action-intermingled-feedback is an actual problem.
review_title: null

Meta-review: metareview: This paper addresses the problem of learning to behave optimally when actions result only in new observations but no rewards. Feedback is provided in the shape of a vector. This problem, known as IGL, has already been described in previous works which had to make the assumption that the action was not included in the feedback. This paper gets rid of this assumption and provides theoretical guarantees. 

The discussion has been quite extensive and the main issue raised by reviewers concerned the experimental setups. They were considered toy-ish and too far from a real application. Especially, the authors mentioned BCI and HCI in their intro (mainly focusing on the fact that having the action in the observation is mandatory with humans in the loop) but didn't provide experiments involving actual BCI or HCI. 

The authors tried to address this issue by providing synthetic experiments simulating BCI and fMRI. As the authors stated, the cost of real experiments in that setup would be prohibitive. 

Given the effort made by authors to provide experimental results supporting them, the algorithmic and theoretical contributions seem good enough to reach the acceptance bar. 
confidence: Certain
Decision: Accept