Interaction-Grounded Learning with Action-Inclusive Feedback
Authors: Tengyang Xie
Akanksha Saran
Dylan J Foster
Lekan P Molu
Ida Momennejad
Nan Jiang
Paul Mineiro
John Langford
Venue: NeurIPS 2022 Accept (2022)
Keywords: interaction-grounded learning
interactive machine learning
reinforcement learning
contextual bandits
human-AI interaction

Abstract: Consider the problem setting of Interaction-Grounded Learning (IGL), in which a learner's goal is to optimally interact with the environment with no explicit reward to ground its policies. The agent observes a context vector, takes an action, and receives a feedback vector, using this information to effectively optimize a policy with respect to a latent reward function. Prior analyzed approaches fail when the feedback vector contains the action, which significantly limits IGL’s success in many potential scenarios such as Brain-computer interface (BCI) or Human-computer interface (HCI) applications. We address this by creating an algorithm and analysis which allows IGL to work even when the feedback vector contains the action, encoded in any fashion. We provide theoretical guarantees and large-scale experiments based on supervised datasets to demonstrate the effectiveness of the new approach.

Interaction-Grounded Learning with Action-Inclusive Feedback
Tengyang Xie UICC tx100illinois.cm edu
Akanksha Saran"
Dylan J. Foster
Microsoft Research, NYC
Microsoft Research, New England
akanksha. sar saran@mic crosoft ft com
@yylanfosteremicroom com
Lekan Molu
Ida Momennejad Microsoft Research, NYC idamo@microsoft. com
Nan Jiang UIUC nanjiangeillinoin
Microsoft Research, NYC
wewrammml@mmicrooooooooo com
Paul Mineiro Microsoft Research, NYC pminei ro@microoff com
John Langford Microsoft Research, NYC jc jclemicrosoft.ccom
Abstract
Consider the problem setting of imeraction-Groundee Learning (IGL), which a learner's goal is to optimally interact with the environment with explicit reward to ground its policies. The agent observes a context vector, takes an action, and receives feedback vector-using this information to effectively optimize policy with respect to a latent reward function. Prior analyzed approaches fail when the feedback vector contains the action, which significantly limits IGL's success in many potential scenarios such as Brain-computer interface (BCI) or Human- computer interface (HCI) applications. We address this by creating algorithm and analysis which allows IGL to work even when the feedback vector contains the action, encoded in any fashion. We provide theoretical guarantees and large-scale experiments based on supervised datasets to demonstrate the effectiveness of the approach.
Introduction
Most real-world learning problems, such BCI and HCI problems, not tagged with rewards. Consequently, (biological and artificial) learners must infer rewards based on interactions with the environment, which reacts the learner's actions by generating feedback, but does provide any explicit reward signal. This paradigm has been previously studied by researchers [e.g., Grizou et al., 2014; Nguyen al., 2021], J, including recent formalization [Xie al., 2021b] that proposed the term Interaction-Groundee Learning (IGL).
In IGL, the learning algorithm discovers grounding for the feedback which implicitly discovers reward function. An information -thooretic impossibility argument indicates additional assumptions necessary to succeed. Xie al. [2021b] proceed by assuming the action is conditionally independent of the feedback given the reward. However, this unnatural in many settings such as neurofeedback in BCI [Katyal et al., 2014; Mishra and Gazzaley. 2015: Debettencourt al., 2015; Muñoz-Moldes and Cleeremans, 2020; Akinola et al., 2020; Poole and Lee, 2021; Xu et al., 2021] and multimodal interactive feedback in HCI [Pantic and Rothkrantz, 2003; Vitense et al., 2003; Freeman et al., 2017; Mott et al., 2017; Duchowski, 2018; Saran et al., 2018, 2020; Zhang et al.,
"equal contribution
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

LEARNING
Neural CONTEXT
9
BANGINATION
(IMAGINATION)
(MR signal for Imagined number "9"
PERCEPTION
7
JUDGNENE
AGENT ACTION
Display a number ('7")
MISMATCHI
Neural FEEDBACK PERCEPTION JUDGMENT
1MRI signal for "perceiving? 7" + is mismatch
Figure 1: A simulated BCI interaction experiment for number- number-guessing task. The human imagines a number, which the agent has to identify through their fMRI signals. The agent displays the number identifies. The human perceives this number (action) and judges whether it match or not (reward)-thr the mixing of these two activities creates a complex feedback signal, which the learning agent has to decode through the interaction.
2020; Cui et al., 2021; Gao et al., 2021] where the action proceeds and thus influences the feedback. For example, if the feedback is an IMRI recording from a human brain implant in the parietal cortex, and the action of the algorithm is to display a number being imagined by the human (Fig. 1), the parietal cortex feedback will contain information about both perceiving the number being displayed (action) and human's 's reaction to it for match or mismatch (reward). Another example is determining if a patient in coma is happy not, and using their brain signals (feedback) to dissociate the fact that they were asked about being happy (action) and the actual thought that they are happy (reward). An HCI example is self-calibrating eye tracker used for typing on a screen by ALS patient [Mott et al., 2017; Gao et al., 2021], where the automated agent calibrates the eye gaze to a certain point on screen (action) and the human's 's gaze patterns (feedback) convey response the tracker's offset (action) as well as their satisfaction for being able to type certain key (reward). If you apply the prior IGL approach to such settings, it will fail catastrophically because the requirement of conditional independence is essential to its function. This motivates the question:
possible to do ceeracttiongrowwee learning when the feedback has the full information of the action embedded in it?
We propose a new approach solve IGL, which call action-inclusive IGL (AI-IGL), that allows the action to be incorporated into the feedback in arbitrary ways. We consider latent reward as playing the role of latent states, which can be further separated via a contrastive learning method (Section 3.1). Different from the typical latent state discovery in rich-observation reinforcement learning [e.g.. Dann al., 2018; Du al., 2019; Misra et al., 2020], the IGL setting also requires identifying the semantic meaning of the latent reward states, which is addressed by symmetry breaking procedure (Section 3.2). We analyze the theoretical properties of the proposed approach, and we prove that it is guaranteed to learn a near-optimal policy as long as the feedback satisfies a weaker context conditional independence assumption. We also empirically evaluate the proposed AI-IGL approach using large-scale experiments on Open-ML's supervised classification datasets [Bischl et al., 2021] ]. as well as a simulated BCI experiment using real human fMRI data [Ellis et al., 2020], demonstrating the effectiveness of the proposed approach (Section 5). Thus, our findings broaden the scope of applicability for IGL.
The paper proceeds as follows. In Section 2, we present the mathematical formulation for IGL. In Section 3, we present contrastive learning perspective for grounding latent reward which helps to expand the applicability of IGL. In Section 4, we state the resulting algorithm AI-IGL. We provide experimental support for the technique in Section 5 using a diverse set of supervised learning datasets and simulated BCI dataset. We conclude with discussion in Section 6.
2 Background
Interaction-Grounder Learning This paper studies the Interaction- Groundded Learning (IGL.) setting [Xie et al., 2021b], where the learner optimizes for a latent reward by interacting with the

---

environment and associating ("grounding") observed feedback with the latent reward. At each time step, the learner receives an i.id. context æ from context set A' and distribution dg. The learner then selects an action a E A from finite action set AA| = K. The environment generates latent binary reward E {0. 1} (can be either deterministic or stochastic) and feedback vector y E y conditional on (x, a),, but only the feedback vector y is revealed to the learner. In this paper, we use R(z, a) == Ezza [r] to denote the expected (latent) reward after executing action a on context æ. The space of context X' and feedback vector Y' can be arbitrarily large.
Throughout this paper, we use π E II X A(A) denote a (stochastic) policy, The expected return of policy π is defined by (TT) = ((a,))~da x [r]. The learning goal of IGL is to find the optimal policy in the policy class, xt' = argmax TEll (TT),, only from the observations of context- action-feedback tuples, (Ir; a, v, This paper mainly considers the batch setting, and we use / to denote the behavior policy. In this paper, we also introduce value function classes and decoder classes, We assume the learner has access
value function class F where F X' x A + [0, and reward decoder class v E V : / x A + [0, 1]. We defer the assumptions we made on these classes to Section 3 for clarity.
We may hope to solve IGL without any additional assumptions. However, is information- theoretically impossible without additional assumptions, even if the latent reward is decodable from (I, 0, y), as demonstrated by the following example.
Example 1 (Hardness of assumption deterministic in (x, a). In this case, the latent reward r can be perfectly decoded from y. However, the learner receives no more information than (x. a) from the (I, a, v) tuple. Thus if II contains at least 2 policies, for any environment where any IGL algorithm succeeds, we can construct another environment with the same observable statistics where that algorithm must fail.
IGL), Suppose y (x, a) and suppose the reward is
IGL with full conditional independence Example I demonstrates the need for further assumptions to succeed at IGL.. Xie et al. [2021b] proposed an algorithm that leverages the following conditional independence assumption to facilitate grounding the feedback in the latent reward.
Assumption (Full conditional independence). For arbitrary (x,, a, r. y) tuples where r and are generated conditional the context x and action a, we assume the feedback vector y is conditionally independent of context x and action a given the latent reward r, L.e. I, y.l.
Xie et al. [2021b] introduce reward decoder class 2b E V y - [0, 1] for estimating E[rly]. which leads to the decoded return Vaz, == (lz x [v))] They proved that it is possible to learn the best ı and vb jointly by optimizing the following proxy learning objective:
argmax J(T, 1b) = V( v)) - V(Taase, v) EI
where Thsse is a policy known to have low expected return. Over this paper, we use IGL (full CI) to denote the proposed algorithm of Xie al. [2021b].
3 A Contrastive-learning Perspective for Grounding Latent Reward
The existing work of interaction-groundee learning leverages the assumption of full conditional independence (Assumption 1), where the feedback vector only contains information from the latent reward. The goal of this paper is to relax these constraining assumptions and broaden the scope of applicability for IGI.. This paper focuses on the scenario where the action information is possibly embedded in the feedback vector, which is formalized by the following assumption.
Assumption 2 (Context Conditional Independence). For arbitrary (x. a. r. y) tuple where r and y are generated conditional on the context x and action a, we assume the feedback vector y is conditionally independent of context x given the latent reward r and action a. That is, x ILyla,r
Assumption 2 allows the feedback vector y to be generated from both latent reward and action differing from Assumption I which constrains the feedback vector y to be generated based only on the latent reward r. We discuss the implications at the end of this section.
3

---

3.1 Grounding Latent Reward via Contrastive Learning
In this section, we propose a contrastive learning objective for interaction -roioonddd learning, which further guides the design of our algorithm. We perform derivations with exact expectations for clarity and intuitions and provide finite-sample guarantees in Section 4.
Assumption 3 (Separability). For each à E A. there exists an (Sä wg) E F x v
such that: 1) E [va (y, a) a, r = I] E j (y, a) r - 0] = 1; 2) E Ja (x, B [fā (z, a) 0] Ay. We also assume 1 - V C V. where 1 - v = { 111-4117 : v)
Assumption 3 consists of two components. For V, it is a realizability assumption that ensures that perfect reward decoder is included in the function classes. Although this superficially appears unreasonably strong, note y is generated based upon and the realization of r; therefore this compatible with stochastic rewards. For F, it ensures the expected predicted reward conditioned on the latent reward, having value r E {0,1) and the action being a, is separable. When μ = Tunif x, (f, R)I (Ta denotes the constant policy with action a). Detailed proof of this argument can be found in Appendix A. One sufficient condition is that the expected latent reward R(-, a) has enough variance and R E F. The condition of 1 T C V can be constructed easily via standard realizable classes. That is, if 1/. E V for some classes V simply setting V <- V U(1 V satisfies Assumption 3. Note that, this construction of vp only amplifies the size of V/ by factor 2.
A5 can be lower bounded by m 4|Cov,,
Reward Prediction via Contrastive Learning We now formulate the following contrastive- learning objective for solving IGL. Suppose μla(zz) is the behavior policy. We also abuse f(z, a, y) to denote the data distribution, and /sa (a.,)) (x))), (a(y) denote the marginal distribution under action a E A. We construct an augmented data distribution for each a E A: (2, y) = Ma Pa Pa(y) (i.e., sampling and y independently from pa).
Conceptually, for each action a E A, we consider tuples (). ~ pa y) and (Z, 7) ~ 7 (2r, y)- From Assumption 3, conditioned on (ar, a) the optimal feedback decoder v/ (v. a) has mean equal to the optimal reward predictor (x, a). Therefore we might seek an (f. ei) pair which minimizes any consistent loss function, c.g., squared loss. However this is trivially achievable, c.g., by having both always predict O. Therefore we formulate a contrastive-lcaming objective, where we maximize the loss between the predictor and decoder on the augmented data distribution. For each E A. we solve the following objective
dh argmin (a)aaraa)= = E,,, [(aa( (a(v Eqg -Va (2) (J..w))FFW
In the notation of equation (2), the subscript indicates the (fa: v/) pair are optimal for action a.
Note they are always evaluated at o, which we retain as an input for compatibility with the original function classes.
Note that L is also similar to many popular contrastive losses [e.g.. Wu et al., 2018; Chen et al., 2020] especially for the spectral contrastive loss [HaoChen et al., 2021].
Ca(fa: v/a) = E,., [fa(z, a)² - 2fa a)Va(8 + (v.
2fa 2f = 2 (Rp, alla,,))aauu.0 a)] E,, [fa a)Wa a))) (spectral contrastive loss)
- En,
2 (Ep. (al
E,,, [fa a)) E,,. (v)(
(Ma(I,y) = Ma(x)) 1ho (y))
Below we show that minimizing La (fas vaa decodes the latent reward under Assumptions 2 and up to sign ambiguity. For simplicity, we introduce the notation of far and varr for any (f. 16) E F x W
far
Pr(zla,r)fxx,
Parr == E,, Pr(yla,r))hin
far and warr are the expected predicted reward of f(z, a) and decoded reward sh a) (under the behavior policy μ for fa,r) conditioned on the latent reward having value r and the action being a. Propusition 1. For any action a A, if p(alz) > 0 for all x E X and Assumption 2 and 3 hold, and let (Fa, va) be the solution of (2). Then, 1fa,1 - Sa,ol = maxxer Ifa,1 - fa.ul and | Vo,ol maxvew Wa,1 Va.ol.

---

Proof Sketch. For any policy т, we use di ==== (((z) to denote the visitation occupancy
of action a under policy и. and Pe == L, dn ( R(x, a) denote the average reward received under executing action a. Then, by the context conditional independent assumption v|r, (Assumption 2), we know
pa (f()) = (1- (1--p))aee PaDaaa + Pala,itas
Bu [f(x)] [x(1]] = (11 faokaa + (p%)) fa.1a.11 (1 P Pa) + f0,1
= La,a Da) & - (fa,1 - -a,o) - -Wan))
Therefore, separately maximizing Ifa.1 fa,ol and Ida,1 Vo,ol maximizes La vo).
3.2 Symmetry Breaking
In the last section, we demonstrated that the latent reward could be decoded in a contrastive-lcaning manner up to a sign ambiguity. The following example demonstrates the ambiguity.
Example 2 (Why do we need extra information to identify the latent reward?). In the optimal solution of objective (2), both Vo and &o == v. yield the same value. It is informatton-hheorreiaalllll difficult to distinguish which one of them is the correct solution without extra information. That is because, for any environment ENV1, there always exists a "symmetric" " environment ENV2, where: I) R(x, a) of ENV1 is identical to (1 R(x, a)) of ENV2 for all (x, a) E X x A; 2) the the conditional distribution of y)r, a in ENV1 is identical the conditional distribution y|1 r, a the ENV2 for all E A. In this example, ENV1 and ENV2 will always generate the identical distribution of feedback vector y after any (x, a) X x A. However, ENV1 and ENV2 have the exuctly opposite latent reward information.
As we demonstrate in Example 2, the learned decoder from (2) could be corresponding to symmetric pair of semantic meanings, and identifying them without extra information is information- theoretically impossible. The symmetry breaking procedure is one of the key challenges of interaction- grounded learning. To achieve symmetry breaking, we make the following assumption to ensure the identifiability of the latent reward.
Assumption 4 (Baseline Policies). For each A, there exists baseline policy Thase such that,
(a) base satisfies
Eg do(z) do(z))T(a(z)) 2 Cm
2 n, where pa Eg do da( ((lx) En do(x)) (a|z)
(b)
2 Pa hone
To instantiate Assumption in practice, we provide the following simple example of Thase that satisfies Assumption 4. Suppose Thase = Tunif (uniformly random policy), and we have "all constant policies are bad", i.... V(T,, 1a a)) 1/2 1 17 for all ā E A. Then easy verify that Cm = 1/K and Pa Tue < 1/2 7 for all ā ã A.
Note that Thare can be different probability of base selecting action (over all context x E X') is at least C.. This condition ensures that base has enough visitation to action a and makes symmetry breaking possible. Assumption 4(b) states that if we only consider the reward obtained from taking action a, Thase is known to be either "sufficiently bad" "sufficiently good". Note the directionality of the extremeness of Thsse must be known, e.g.. a policy which has a unknown reward of either 0 or is not usable. This condition follows a similar intuition as the identifiability assumption of Xie et al. [2021b, Assumption 2] and breaks the symmetry. For example, consider the ENV1 and ENV2 introduced in Example 2, EN,2 for any policy æ. To separating ENV1 and ENV2 using some policy म, P",,NVI and E2v2 require to have non-zero gap, which leads Assumption 4(b).
actions. Intuitively, Assumption 4(a) is saying that the total
P", ENV1
The effectiveness of symmetry breaking under Assumption 4 can be summarized as below: we - Σ, (x)))) (ala) (a.a) E. do(=) (a/z)
conduct the following estimation of TTsee using the learned C.. Po
If Vh can efficiently decode the latent reward, then - converges to either pa or p pa Therefore, applying Assumption 4(b) breaks the symmetry.
5

---

(b) Latent Reward in IGL under Assumption 2 (this paper)
(c) Latent Reward in IGL under
(a) Latent State in Rich- Aich-Observation RL [Misra et al., 2020]
Assumption [Xic ct al., 2021b]
Figure 2: Causal graphs of interaction interaction-rrounded learning under different assumptions well as rich-observation reinforcement Learning.
3.3 Comparison to Full CT
When have the context conditional independence, it is easy verify the failure of optimizing the original IGL objective (1.(1) by the following example.
Example 3 (Failure the original IGL objective under Assumption 2). Let &' AA (1,2.... ,0) and feedback vector is generated by y (a R(z, a)) mod 10 (we use % to denote mod in the following part). We also assume do(x) = [aase(a(z)) = 1/10 for any (x, a) E X x A and R(x, a) Tt" (a|x:) [x a). Then, we have, for any tb y - [0, (approach proposed by Xie et al. [2021b] assumes the reward decoder only takes feedback vector y as the impur).
C(T" (-
to Σ - E 1(z a)v((((aaaaa () 1)%10) 100 Σ Eee s((a + 1(x a))
Em (a)) in E.e (aa) (a)a)00
On the other hand, consider the constant policy x (alax) = (a = 1) for all x E X and decoder v2(()) = (yy = for all Y,, then,
[(M v22 = :tb -. E1e 1(a = 1)ea(((a + 1)%10) 1 Ene Em V2((a + 1(x = a))%10) (a) x2) v2() 0.9 > L(x ,)), Veh E v.
100
This implies that maximizing the original IGL objective (1) could not always converge to x² when we only have the context conditional independence.
This example indicates optimizing combined contrastive objective with single symmetry symmetry-breaking policy is insufficient to succeed in our rLylr, a case. Our current approach corresponds to optimizing contrastive objective and breaking symmetry for each action separately rather than simultancously.
3.4 Viewing Latent Reward Latent State
Our approach motivated by latent state discovery in Rich-Observation RL [Misra et al., 2020]. Figure compares the causal graphs of Rich-Observation RL, IGL with context conditional indepen- dence, and IGL with full conditional independence. In Rich-Observation RI., contrastive learning objective is used to discover latent states; whereas in IGL. contrastive learning objective is used to discover latent rewards. In this manner we view latent rewards analogously to latent states.
Identifying latent states up permutation is completely acceptable in Rich-Observations RL, as the resulting imputed MDP orders policies identically to the true MDP. However in IGL the latent states have scalar values associated with them (rewards), and identifying up permutation is not sufficient to order policies correctly. Thus, we require the additional step of symmetry breaking.
4 Main Algorithm
This section instantiates the algorithm IGL. with action-inclusive feedback, following the concept we introduced in Section 3. For simplicity, we select the uniformly random policy Tunif as behavior policy μ in this section. That choice of μ can be further relaxed using an appropriate importance weight. We first introduce the empirical estimation of C. as follows (C, is defined in Eq.(2). Ja

---

estimates the spectral contrastive loss, which corresponds to -L). For any ā E A, we define the following empirical estimation of the spectral contrastive loss:
Ja,p(f. v)) Ep (z, ay a)] (ED =a] a] ED joy, ally,a)I((a -a]. (4)
Using this definition, Algorithm instantiates version of the IGL. algorithm with action-inclusive feedback. Without loss of generality, we also assume Pa - I 1/2 for Assumption 4(b) for all a E A
in this section. The case of Tims > 1/2 for some action a can be addressed by modifying the symmetry-breaking step properly in Algorithm I.
Algorithm 1 Action-inclusive IGL (AI-IGL)
Input: Batch data D generated by f Munif- baseline policy "base"
Initialize policy лı as the uniform policy.
2: for ā E A do
3:
Obtain (fas via) by
Latent State (Reward) Discovery
va)
argmax Ja,
(5)
(.v)eFx
where (a, is defined in (4)
4: Compute Pa sas by la Tom Σ Dig The (a
Symmetry Breaking
5: if Toe
La
> then via + (1 - v.).
6: else V/2 f v'a
7: end if
8:
end for
9: Generate decoded contextual bandits dataset DcB + {(x, a, (v, p(alz)) (ar, a. E D).
Output policy (2) + CB(DcB), where CB denotes an offline contextual bandit oracle.
At a high level, Algorithm has two separate components, latent state (reward) discovery (line 3) and symmetry breaking (line 4-7), for each action in A.
Theoretical guarantees In Algorithm 1, the output policy obtained by calling an offline contextual bandits oracle (CB). We now formally define this oracle and its expected property.
Definition 1 (Offline contextual bandits oracle). An algorithm CB is called an offline contextual bandit oracle for any dataset reward determined by (Ia, a:)) and any policy class II, the policy से produced by CB(D) satisfies ECB max Ed, x (r] Ed x [r] S o11)
{z; a, Tas p(a, [x)))
(x; dos as μt, and ri is the
The notion in Definition corresponds to the standard policy learning approaches in the contextual bandits literature [e.g., Langford and Zhang, 2007; Dudik et al., 201 1; Agarwal et al., 2014], and typically leads to EçB = / Klog /'//////) We now provide the theoretical analysis of Algorithm In this paper, we use dr, v to denote the joint statistical complexity of the class of F and V. For example, the function classes are finite, we have dy.v Olog (F1191/a) and the failure probability. The infinite function classes be addressed by some advanced methods such covering number or Rademacher complexity [see, e.g., Mohri et al., 2018). The following theorem provides the performance guarantee of the output policy of Algorithm 1.
Theorem 2. Suppose Assumptions 2, 3 and hold, Dy is defined in Assumption 3, and ft be the output policy of Algorithm 1. If we have (D) 2 O
then, with high probability, K2
Imin/nAy.Ken.
- ((=)) 1
ECB
Similar to the performance of Xie et al. [2021b], the learner is guaranteed to converge in the right direction only after we have sufficient data for symmetry breaking. The dependence on K in Theorem 2 can be improved as different action has separate learning procedure. For example, if we consider F Ji x F2 .... F and V V, x V2 ..... Wk. where F, and V are independent components that is only corresponding to action (this is common setup for linear approximated
7

---

reinforcement learning approaches with discrete action space). If Fa and V. are identical copies of K separate classes, we know log |F] = K log Fa and log |V] = K log |aal which leads v K improvement.
We now provide the proof sketch of Theorem and we defer the detailed proof to Appendix A.
Proof Sketch. The proof of Theorem 2 consists of two different despoonents--sccoov latent reward and breaking the symmetry, which are formalized by the following lemma.
Lemma 3 (Discovering latent reward). Suppose Assumptions 2 and hold, and let (fa. va) be obtained by (5) Then, with high probubility, we have [Wa, t/a.al
O IF
K$dr
12
Lemma ensures that the learned decoder on Eq.(S) correctly separates the latent reward. In particular since & ranges over [0, 1], Lemma 3 ensures max (Pr(1(y, a) = r), Pr( 1 r)) (1)(1) under the behaviour policy. Thus, we can break symmetry, we can use y to generate a reward signal and reduce ordinary contextual bandit learning. The following lemma guarantees the correctness of the symmetry- -brekkng step.
Lemma 4 (Breaking symmetry). Suppose Assumption 4 holds. For any à E A, if we huve [D] 2
(min(n)s.Kem)) then, with high probability, vaa 2 va..-
dy
Combining these two lemmas above as well as the CB oracle establishes the proof of Theorem 2, and the detailed proof can be found in Appendix A.
Empirical Evaluations
In this section, we provide empirical evaluations in simulated environments created using supervised classification datasets (Sec. 5.1) and an open source fMRI simulator (Sec. 5.2). We evaluate our approach by comparing: (1) CB: Contextual Bandits with exact reward, (2) IGL (full CI): The method proposed by Xie et al. [2021b] which assumes the feedback vector contains no information about the context and action, and (3) AI-IGL: The proposed method which assumes that the feedback vector could contain information about the action but is conditionally independent of the context given the reward. Note that contextual bandits (CB) is skyline compared to both IGL (full CI) and AI-IGL, since it need not disambiguate the latent reward. All methods use logistic regression with a linear representation. At test time, each method takes the argmax of the policy. Following the practical instantiation of Assumption 4 (Section 3.2), we know if the dataset has a balanced action distribution (no action belongs to more than 50% of the samples). selecting uniformly random policy Tunif as base for all a E A satisfies Assumption 4. Therefore, in this section, our experiments are all based on the dataset with a balanced action distribution, and select Thss uniif for all a E A.
5.1 Large-scale Experiments with OpenML CC-18 Datasets
To verify that our proposed algorithm scales to a variety of tasks, we evaluate performance on more than 200 datasets from the publicly available OpenMI. Curated Classification Benchmarking Suite [Vanschoren et al., 2015; Casalicchio et al., 2019; Feurer et al., 2021; Bischl et al., 2021]. OpenML. CC-18 datasets are licensed under CC-BY license and the platform and library are licensed under the BSD (3-Clause) license". At each time step. the context xt is generated uniformly at random. The learner selects an action a, E (O...., the total number of actions available in the environment). The binary reward is the correctness of the prediction label a,. The feedback yt is a two dimensional vector (a,,Tt). Each dataset has different sample size (N) and different set of available actions (K). We sample datasets with 3 or more actions, and with a balanced action distribution (no action belongs to more than 50% of the samples) to satisfy Assumption 4. We use 90% of the data for training and the remaining 10% for evaluation. Additional details of setting up the experiment are in Appendix C. The results are averaged over 20 trials and shown in Table
, K 1} as the predicted label of xz, (where K is
Pnttos://creativecommns org http://ccmmmmoorrr/////////////a .0/ htttp:///oemsuure..rr///iiiteeee CCluse
8

---

Dataset
Daraset
Constant
CB Folicy
IGL. (fall CI)
Performance
AL-GGI Policy Accuracy (%)
Performance
Criteria
Count
Action
Accuracy (%8)
Policy Accuracy (%)
WB
WE
271
25.28
+ 2.96
57.98 5.65
15.05 2,30
0.30
0.04
35.74
1.45
0.02
K.. K2.N N 2 70000
83
22.57
= 3.14
58. = 5.04
11.91 1880
0.22 + 0.04
50.11 2.98
0.09
Table I: Results in the OpenML environments with two-dimensional action-inclusive feedback.
Average and standard error reported over 20 trials for each algorithm. "Performance w.r.t. CB' reports the ratio of an IGL method's policy accuracy over CB policy accuracy.
5.2 Experiments on Realistic IMRI Simulation
We also evaluate our approach on simulated BCI experiment, where human imagines a number and the agent faces number-decodingg task (Fig. 1). We simulate an fMRI based BCI setup, recording the IMRI BOLD (Blood Oxygenation Level Dependent) response the parictal cortex of a human brain. We have selected the parietal cortex since both numerical processing [Santens et al., 2010] and match-mismatch processing are reported to involve this region [Male et al., 2020], and in principle it could feasibly be recorded by fNIRS (with similar spatial resolution to fMRI) [Naseer and Hong, 2015]. The task requires an intelligent agent to decode the numbers the human imagines. This task is motivated by real-world examples where an implant could assist a person, user, or patient by communicating their mental states to the outside world, whether for therapeutic purposes or to control hardware or software.
The task setup consists of the following steps- (1) Human Imagination: A human participant imagines a number from a set of digits (i.e., 7, 8, 9), (2) Agent Decoding: an agent (Full CI IGL or AI-IGL) decodes the number being imagined using only the IMRI signals of the human participant, with no labels and no supervised pretraining, (3) Human Perception: the agent visually displays the number it decoded to the human, which the human brain perceives, and hence the fMRI signals would contain patterns related to the perception of the agent's s guess, (4) Human Judgment: Once the human judges whether the agent-guessed number yields a match or mismatch to the number they imagined, IMRI signals reflect the participant's 's judgment. This simple setup mimics many relevant applications of BCI with realistic simulation of fMRI signals (generalizable to the more cost-efficient fNIRS). We used a third party simulation software, offering strong test bed for the algorithms compared here.
We use an open source simulator [Ellis et al., 2020] to simulate BCI-like, interactive fMRI signals for the imagination, perception, and judgment phases described above. This simulator leverages real human fMRI data [Bejjanki et al., 2017] to simulate signals for different regions of the brain. Prior neuroscience studies have shown that the parietal cortices in the human brain (in particular the posterior parietal cortex) process numbers [Santens et al., 2010], while the frontoparietal cortices process visual attention as well as match-mismatch judgments [Male et al., 2020]. These studies, together with the low temporal resolution of IMRI/NIISS [Glover, 201 11, warrant the assumption we make here: that number perception and judgments signals can both be decoded from the same region, i.c., the parietal cortex. Therefore, per AI-IGL's theoretical assumptions, the two signals can be mixed in the brain's 'ss implicit feedback signal, here simulated as an iMRI or fNIRS signal, in arbitrary ways. Thus, selected a region of interest (ROI) in the parietal cortex to cover the mixed feedback of perceiving the numbers and making a maith-hmismacch judgments. This simulated BCI setting is precisely the sort of realistic condition that our proposed AI-IGL solution is designed to handle well. Moreover, it is quite possible to conduct this experiment in principle using fMRI, or more realistically using fNIRS which is portable, noninvasive, easy to use, and can be focused specific patch on the surface of the brain.
Each event (perception, imagination, judgment) evokes different activity in the human brain, within the same set of 4 x x 4 voxels, generating a 64-dimensional activity vector. We generate a time course of 666 trials, each of them including sequence of events where an imagination event is followed by a response from the agent, then a perception event, and a judgment event. Thus, the 666 trials include 1998 simulated brain events total. Each event is 2 seconds long, with inter-stimulus (ISI) of seconds, and a temporal resolution of 10 seconds. For each event, the simulator combines spatiotemporal noise (trial noise) and voxel activation patterns to simulate activity signals. We use the imagination signal as context (Fig. 1, Context) and a weighted average of the perception and judgement signals to simulate the brain's mixed feedback signal (Fig. 1, Feedback).
We evaluate the performance of both full CI IGL and AI-IGL on this simulated and interactive number-decoding task, under varying levels of feedback noise. The results are shown in Table 2. We

---

found that full CI IGL failed at the task, remaining at chance level guessing, However, the novel algorithm proposed here, AI-IGL, was capable of learning to dissociate the human judgment from mixed feedback signal. The significance of these findings are two-fold. First, they offer realistic approach to simulating BCI experiments order to test theoretical advances before conducting real world experiments. Second, our results further validate the theoretical advances of AI-IGL compared to the previous IGL approach for a realistic interactive brain-computer interface.
Algorithm
1% noise
5% noise
10% noise
IGL (full CI)
32.60 = 0.24
33.75 + 0.29
33.33 + 0.29
AI-IGL
89.10 + 3.58
76.60 4.09
64.25 4.16
Table Policy aceuracy for the simulated BCI experiments with and standard error reported over 20 trials for each algorithm.
feedback. Average
Discussion
We have presented a new approach to solving Interaction -rrructdionGruudeee Learning, in which an agent learns interact with the environment in the absence of any explicit reward signals. Compared to a prior solution [Xie et al., 2021b], the proposed AI-IGL approach removes the assumption of conditional independence of actions and the feedback vector by treating the latent reward as a latent state. It thereby provably solves IGL for action-inclusive feedback vectors.
By viewing the feedback containing an action-(latent) reward pair which is an unobserved latent space, we propose latent reward learning using a contrastive learning approach. This solution concept naturally connects latent state discovery in tich-observation reinforcement learning [e.g.. Dann et al., 2018; Du et al., 2019; Misra et al., 2020]. On the other hand, different from rich-observation RL, the problem of IGL also contains a unique challenge in identifying the semantic meaning of the decoded class, which is addressed by a on binary latent rewards, for which symmetry breaking is possible using one policy (per action). Breaking symmetry in more general latent reward spaces is a topic for future work. A possible negative societal impact of this work can be performance instability, especially with inappropriate use of the techniques in iissiivne applications.
procedure. In this work, we focus
Barring intentional misuse, we envision several potential benefits of the proposed approach. The proposed algorithm broadens the scope of IGL's feasibility for real-world applications. Imagine an agent being trained to interpret the brain signals of user to control a prosthetic arm. The brain's response (feedback vector) to an action is neural signal that may contain information about the action itself. This is so prevalent in neuroimaging that fMRI studies routinely use specialized techniques to orthogonalize differrnt information (e.g. action and reward) within the same signal [Momennejad and Haynes, 2012, 2013; Belghazi et al., 2018; Shah and Peters, 2020]. Another example is a continually self-calibrating eye tracker, used by people with motor disabilities such as ALS [Hansen et al., 2004; Liu et al., 2010; Mott et al., 2017: Gao et al., 2021]. A learning agent adapting to the ability of such users can encounter feedback directly influenced by the calibration correction action. The proposed approach is stepping stone on the path to solving IGL for complex interactive settings, overcoming the need for explicit rewards as well as explicit separation of action and feedback information.
Acknowledgments
The authors would like to thank Mark Rucker for sharing methods and code for computing and comparing diagnostic features of OpenML datasets. NJ acknowledges funding support from ARL Cooperative Agreement W91 NSF Data Science Research Award.
NSF CAREER award, and Adobe
References
Sarah Daniel Abdelmessih, Faisal Shafait, Matthias Reif, and Markus Goldstein. Landmarking for meta-learning using rapidminer. In RapidMiner community meeting and conference. Citeseer, 2010.
10

---

Alekh Agarwal, Daniel Hsu, Satyen Kale, John Langford, Lihong Li, and Robert Schapire. Taming the monster: A fast and simple algorithm for contextual bandits. In International Conference on Machine Learning, pages 1638-1646. PMLR, 2014.
Iretiayo Akinola, Zizhao Wang, Junyao Shi, Xiaomin He, Pawan Lapborisuth, Jingxi Xu, David Watkins-Valle Paul Sajda, and Peter Allen. Accelerated robot learning via human brain signals. In 2020 IEEE international conference on robotics and automation (ICRA), pages 3799-3805. IEEE, 2020.
Vikranth R Bejjanki, Rava Azeredo Da Silveira, Jonathan Cohen, and Nicholas Turk-Browne. Noise correlations in the human brain and their impact on pattern classification. PLoS computa- tional biology, 13(8)ee005674, 2017.
Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeswar, Sherjil Ozair, Yoshua Bengio, R. Devon Hjelm, and Aaron C. Courville. Mutual information neural estimation. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 530- 539. PMLR, 2018.
Bernd Bischl, Giuseppe Casalicchio, Matthias Feurer, Pieter Gijsbers, Frank Hutter, Michel Lang, Rafael Gomes Mantovani, Jan N. van Rijn, and Joaquin Vanschoren. Openml benchmarking suites. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurlPS Datasets and Benchmarks, 2021.
Giuseppe Casalicehio, Jakob Bossek, Michel Lang, Dominik Kirchhoff, Pascal Kerschke, Benjamin Hofner, Heidi Seibold, Joaquin Vanschoren, and Bernd Bischl. Openml: An package to connect to the machine learning platform openml. Computational Statistics, 34(3):977-99) -91777999,1111 2019.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020.
Yuchen Cui, Qiping Zhang, Brad Knox, Alessandro Allievi, Peter Stone, and Scott Niekum. The empathic framework for task learning from implicit human feedback. In Conference on Robot Learning, pages 604-626. PMLR, 2021.
Christoph Dann, Nan Jiang, Akshay Krishnamurthy, Alekh Agarwal, John Langford, and Robert E Schapire. On oracle-efficient pac rl with rich observations. Advances in neural information processing systems, 31, 2018.
Megan T Debettencourt, Jonathan D Cohen, Ray F Lee, Kenneth A Norman, and Nicholas B Turk- Browne, Closed-loop training of attention with real-time brain imaging. Nature neuroscience, 18 (3):470-475, 2015.
Simon Du, Akshay Krishnamurthy, Nan Jiang, Alekh Agarwal, Miroslav Dudik, and John Langford. Provably efficient rl with rich observations via latent state decoding. In International Conference on Machine Learning, pages 1665-1674, PMLR, 2019.
Andrew T Duchowski. Gaze-based interaction: A 30 year retrospective. Computers & Graphics, 73: 59-69, 2018
Miroslav Dudik, Daniel Hsu, Satyen Kale, Nikos Karampatziakis, John Langford, Lev Reyzin, and Tong Zhang. Efficient optimal learning for contextual bandits. In Proceedings of the Twenty- -evennhh Conference Uncertainty in Artificial Intelligence, pages 169-178, 2011.
Cameron T Ellis, Christopher Baldassano, Anna C Schapiro, Ming Bo Cai, and Jonathan Cohen. Facilitating open-science with realistic fmri simulation: validation and application. PeerJ, 8:88564, 2020.
Matthias Feurer, Jan N van Rijn, Arlind Kadra, Pieter Gijsbers, Neeratyoy Mallik, Sahithya Ravi, Andreas Müller, Joaquin Vanschoren, and Frank Hutter. Openml-python: an extensible python api for openml. Journal of Machine Learning Research, 22:1-5, 2021.
11

---

Euan Freeman, Graham Wilson, Dong- Brng-Bach Vo, Alex Ng. Ioannis Politis, and Stephen Brewster. Multimodal feedback in hei: haptics, non-speech audio, and their upplications. In The Handbook of Paalimoodal-Multiseeee Interfaces: Foundations, User Modeling, and Common Modality Combinations- voviluauttioo--uueeeee 1, pages 277-317. ACM, 2017.
Jensen Gao, Siddharth Reddy, Glen Berseth, Nicholas Hardy, Nikhilesh Natraj, Karunesh Ganguly, Anca D. Dragan, and Sergey Levine. X2T: training an x-to-text typing interface with online learning from user feedback. In 9th International Conference on Learning Representations, ICLR, 2021.
Gary H Glover. Overview of functional magnetic resonance imaging. Neurosurgery Clinics, 22(2): 133-139, 2011.
Jonathan Grizou, Inaki Iturrate, Luis Montesano, Pierre-Yve Oudeyer, and Manuel Lopes. Calibration bei based control. In Proceedings the AAAI Conference on Artificial Intelligence, volume 28, 2014.
John Paulin Hansen, Kristian Tørning, Anders Sewerin Johansen, Kenji Itoh, and Hirotaka Aoki. Gaze typing compared with input by head and hand. In Proceedings of the 2004 symposium on Eye tracking research & applications, pages 131-138, 2004.
Jeff Z HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma. Provable guarantees for self-supervised deep learning with spectral contrastive loss. Advances in Neural Information Processing Systems,
34, 2021.
Kapil D Katyal, Matthew S Johannes, Spencer Kellis. Tyson Aflalo, Christian Klaes, Timothy G McGee, Matthew P Para, Ying Shi, Brian Lee, Kelsie Pejsa, et al. A collaborative bci approach to autonomous control of prosthetic limb system. In 2014 IEEE International Conference on Systems, Man, and Cybernetics (SMC), pages 1482. IEEE, 2014.
John Langford and Tong Zhang. The epoch-greedy algorithm for contextual multi-armed bandits. Advances in neural information processing systems, 20(1):96-1, 2007.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-2324 1998.
Shuo Samuel Liu, Andrew Rawicz, Teng Ma, Cheng Zhang, Kyle Lin, Siavash Rezaei, and Eion Wu. An eye-gaze tracking and human computer interface system for people with als and other locked-in diseases. CMBES Proceedings, 33, 2010.
Ana C Lorena, Luis PF Garcia, Jens Lehmann, Marcilio CP Souto, and Tin Kam Ho. How complex is your classification problem? survey on measuring classification complexity. ACM Computing Surveys (CSUR), 52(5):1-34, 2019.
Alie G Male, Robert P O'Shea, Erich Schröger, Dagmar Müller, Urte Roeber, and Andreas Widmann. The quest for the genuine visual mismatch negativity (vmmn): Event-related potential indications of deviance detection for low-level visual features. Psychophysiology. (7(6)e13576 2020.
Jyoti Mishra and Adam Gazzaley. Closed-loop cognition: the next frontier arrives. Trends in cognitive sciences, 242-223 2015.
Dipendra Misra, Mikael Henaff, Akshay Krishnamurthy, and John Langford. Kinematic state abstrac- tion and provably efficient rich-observation reinforcement learning. In International conference machine learning, pages 6961-6971. PMLR, 2020.
Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of machine learning. MIT press, 2018
Ida Momennejad and John-Dylan Haynes. Human anterior prefrontal cortex encodes the 'what' 'nd 'when' of future intentions. Neuroimage, 611) 2012.
Ida Momennejad and John-Dylan Haynes. Encoding of prospective tasks in the human prefrontal cortex under varying task loads. Journal of Neuroscience, 134444417742211339 2013.
12

---

Martez E Mott. Shane Williams, Jacob O Wobbrock, and Meredith Ringel Morris. Improving dwell-based gaze typing with dynamic, cascading dwell times. In Proceedings the Conference on Human Factors in Computing Systems, pages 2358-2570, 2017.
the 2017 CHI human neuroscience, 9:3, 2015.
Santiago Muñoz-Moldes and Axel Cleeremans. Delineating implicit and explicit processes in neurofeedback learning. Neuroscience Biobehavioral Reviews, 118:681-688 1-88, 2020.
Noman Naseer and Keum-Shik Hong. fnirs-based brain-computer interfaces: review. Frontiers in
Khanh X Nguyen, Dipendra Misra, Robert Schapire, Miroslav Dudik, and Patrick Shafto. Interactive learning from activity description. In International Conference on Machine Learning, pages 8108 PMLR, 2021.
Maja Pantic and Leon JM Rothkrantz. Toward an sffect-sensitive multimodal numan-compuics interaction. Proceedings of the IEEE, 9109):370--139, 2003.
Benjamin Poole and Minwoo Lee. Towards intrinsic interactive reinforcement learning. arXiv preprint arXiv:21 2021.
Matthias Reif, Faisal Shafait, Markus Goldstein, Thomas Breuel, and Andreas Dengel. Automatic
classifier selection for non-experts. Pattern Analysis and Applications, 17(1):83 2014.
Seppe Santens, Chantal Roggeman, Wim Fias, and Tom Verguts. Number processing pathways in human parietal cortex. Cerebral cortex, 20(1):77 2010.
Akanksha Saran, Srinjoy Majumdar, Elaine Schaertl Short, Andrea Thomaz, and Scott Niekum. Human gaze following for human-robot interaction. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 8615-862. IEEE, 2018.
Akanksha Saran, Elaine Schaertl Short, Andrea Thomaz, and Scott Niekum. Understanding teacher gaze patterns for robot learning. In Conference on Robot Learning, pages 1247-1258. PMLR, 2020.
Rajen D) Shah and Jonas Peters. The hardness of conditional independence testing and the generalised covariance measure. The Annals of Statistics, 48(3): 1514-1554455388 2020.
Vicenę Torra, Yasuo Narukawa, and Toho Gakuen. Modeling decisions for artificial intelligence. International Journal of Intelligent Systems, 23(2):113, 2008.
Joaquin Vanschoren, Jan N van Rijn, B Bischl, G Casalicchio, M Lang, and M Feurer. Openml: networked science platform for machine learning. In ICML 2015 MLOSS Workshop, volume 2015.
Holly S Vitense, Julie A Jacko, and V Kathlene Emery. Multimodal feedback: an assessment of performance and mental workload. Ergonomics, 46(1-3):68-B)) 887, 2003.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3733-3742, 2018
Tengyang Xie, Nan Jiang, Huan Wang. Caiming Xiong, and Yu Bai. Policy finetuning: Bridg- ing sample-efficient offline and online reinforcement learning. Advances in neural information processing systems, 34, 2021a.
Tengyang Xie, John Langford, Paul Mineiro, and Ida Momennejad. Interaction-grounded learning. In International Conference on Machine Learning, pages 11444-11423, PMLR, 2021b.
Duo Xu, Mohit Agarwal, Ekansh Gupta, Faramarz Fekri, and Raghupathy Sivakumar. Accelerating reinforcement learning using eeg-based implicit human feedback. Neurocomputing, 460:139-155. 2021.
Ruohan Zhang, Akanksha Saran, Bo Liu, Yifeng Zhu, Sihang Guo, Scott Niekum, Dana Ballard, and Mary Hayhoe. Human gaze assisted artificial intelligence: review. In IJCAI: Proceedings of the Conference, volume 2020, page 4951. NIH Public Access, 2020.
13


Reviews:
[{"review_id": "ylIXo4mh4gf", "reviewer": "Reviewer_R7aA", "summary": "The paper looks at the setting where the goal is to learn a policy without access to a reward from the environment, with access only to feedback. Previous work (Xie et al (2021)) assumes that the feedback does not have the action. This paper removes that assumption and proposes a new contrastive approach that can integrate feedback with actions to learn a policy. Through different empirical evaluations (toy MNIST task, and large-scale synthetic eval on OpenML), the authors show how this approach can learn from feedback that has the actions taken by the agent.", "questions": "How does this work relate to learning from interaction works in robotics / HRI and human-in-the-loop learning?", "limitations": "The authors’ discuss the potential negative impact of the work well. Limitations of the approach and testing might be lacking; see weaknesses.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n- The paper removes some assumptions introduced in previous works to bring algorithms closer to being realised in realistic environments.\n- Learning from non-scalar rewards and feedback is an important direction and this work makes progress towards making this possible.\n- The paper has the appropriate ablations and model comparisons.\n\nWeaknesses\n\nI am not very familiar with the relevant literature in the subfield. As someone from a related but outside the subfield, here are some potential weaknesses. My main concerns lie with the writing of the paper.\n\n- Paper is not clearly written. The introduction does not motivate the central problem well. I would have liked to see a few examples where previous methods fail, but the paper is able to bridge the gap. The authors add these to the conclusion, but it might be useful to add examples to the introduction to motivate the problem. Section 3 is especially difficult to understand as there is a lot of notation with little intuition.\n- The empirical evaluations aren’t described well. I had to refer to Xie et al. (2021) to get a better idea of what the evaluations look like. Similarly, the previous paper also does a much better job of describing and motivating different applications.\n- The empirical evaluations are limited; assuming very simple policies and representations. These would be difficult to realize in realistic settings that the authors use as examples.\n- The intuition behind the feedback signal for the MNIST classification signal is not clear.\n- The empirical evaluations seem detached from potential practical applications.\n- Feedback is often noisy; the authors do not consider this scenario.\n- There is a lack of discussion of why a CB setting is better than an MDP formulation for the tasks proposed.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "UnTPuUN6v5n", "reviewer": "Reviewer_ejKG", "summary": "This paper studies the problem of interaction-grounded learning, where an agent takes an action a given context x, and receives a feedback vector y in return. A reward (to be optimized) is also generated from the environment, but not revealed to the learning process. The goal is to learn a policy that optimizes reward without explicitly observing reward from the environment, by learning a mapping from Y to the space of rewards, and learning a value function mapping X and A to [0, 1]. An existing challenge of this setting is that IGL fails when y contains information about the executed action a. This paper proposes a contrastive learning approach that allows IGL to succeed even when y contains information about a. Experiments on several RL benchmarks show improvements over IGL assuming conditional independence of y given a, nearing performance of contextual bandit (which assumes full access to reward).", "questions": "Questions about some other assumptions that seem to be made in the paper:\n* Does this algorithm / setting assume a discrete action space? What happens if the action space is continuous?\n* Does the setting assume that y is deterministic given (x, a)? It seems like no, given the experiments.\n* Is there an assumption that there exists an optimal \\Psi, that can map any (y, a) to the correct reward? I suppose if y is deterministic given r, then yes. But it seems like that's not the case given the experiments.\n* Does the algorithm require executing all actions in all states in the batch data D? If so, seems like another assumption that might break in real-world settings.\n* In Table 1, action-exclusive feedback is just the image of the reward, as in Figure 2?\n* Images y in 5.1 are randomly chosen from the dataset?", "limitations": "See above about limitations / assumptions for applications to real-world settings.", "rating": 7, "confidence": 1, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n* Paper is relatively clear and easy to follow. As someone who doesn't have much practice with reading theory-heavy papers (unfortunately this paper is a few steps out of my area), I found it easy to follow. I found the information-theoretic arguments intuitive.\n* Main contributions have potential broad significance (e.g., for applications as discussed in the introduction)\n\n--- \n\nWeaknesses:\n\n* See assumptions below -- discussion on unstated assumptions (if any) and their limits on real-world applications would be nice to have.\n* I found the format of Figure 2 somewhat confusing; to me the y-axis of the graphs seems to imply some kind of sequential process", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "3YN6cEGBd6M", "reviewer": "Reviewer_v1ct", "summary": "This paper explores a straightforward reformulation of a newly introduced formalism — Interaction-Grounded Learning (IGL) — in which an agent must learn to act optimally in an environment only given access to some context, an action space, and per-step feedback that is some function of a latent reward function. \n\nWhereas prior work formalizes the feedback component of IGL as conditionally independent of the context and language given the hidden rewards, this work (1) shows that this prior conditional independence assumption is too strong, leading to pathological failure modes when “action” information is present in the feedback, (2) presents a new formulation with less restrictive conditional independence assumptions that condition feedback on both latent reward AND executed action, and (3) presents a learning algorithm for deriving good policies given the new formulation and feedback. \n\nExperiments on a modified MNIST task where the (context, action) are (image, label) respectively, and high-dimensional feedback is given as some image y where the digit shown is taken as (action + 6 * binary-reward - 3 modulo 10), and a similar construction for the various OpenML CC-18 datasets shows that learning with the proposed algorithm under the new conditional independence assumptions outperforms learning with the prior assumptions in cases where action-information is present in the feedback. ", "questions": "- Notation in (1) is a bit unclear; L(\\pi, \\psi) seems to indicate a loss, but you’re actually trying to maximize the difference in Values right? Might it be better to call it J(\\pi, \\psi)?\n- On line 115, it’s not clear where the augmented data is actually coming from; how are you generating the marginal distributions \\mu(x) and\\mu(y)? Especially if the data coming in is online interaction data... this feels rather strong? Or am I misunderstanding?\n- In general - concrete examples to ground out the front half of the paper would be greatly appreciated!\n", "limitations": "In general this paper is severely limited by its evaluation. The paper seeks to show that the initial assumptions in the interaction-grounded learning framework are prohibitive and lead to failure modes, designing a new formulation of the framework’s assumptions, as well as a new learning algorithm to fix this. However, by only evaluating synthetic, toy constructed tasks with arbitrary feedback that is not realistic, it’s not clear that this is an actual problem.\n\nThis paper would be considerably stronger if evaluated on the motivating tasks in brain-computer interfaces or human-computer interaction used in the introduction and conclusion, and evaluating real-world instances of where action-intermingled-feedback is an actual problem.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The key strength of this paper is in its clarity; at its core it takes an existing formalism (“full conditional independence Interaction-Grounded Learning”), finds a key failure mode, and adjusts the formalism with new, less restrictive assumptions. Each of the steps taken along the way are well-motivated, and the proofs tied to the new assumptions (separability, access to a baseline policy for symmetry breaking) are clean and relevant.\n\nUnfortunately, the weaknesses of this paper are in the evaluation, and the underlying motivation behind this work; the introduction and discussion/conclusion motivate that this type of “Interaction Grounded Learning” is **critical** for scenarios in multimodal interactive feedback in human-computer interaction and in designing brain-computer interaction interfaces. Especially for the latter, the problem of having feedback data that is tied to action information is already a key problem in fMRI information orthogonalization (line 340), and for eye tracker recalibration for ALS patients (line 341). Unfortunately, none of the existing evaluations reflect these real-world use cases, instead constructing synthetic tasks based on MNIST or open classification benchmarks — tasks that are hard to understand.\n\nFor example, it’s hard to make the leap to these well-motivated, clear tasks from the current MNIST classification experiments where feedback is an arbitrary formula of the “right label” and the “predicted action” — in the paper experiments, the feedback provided an agent is a digit with label = (action + 6 * binary-reward - 3 modulo 10). In general, it’s not clear why the action-intermingling with feedback is an action problem in real-world settings, or why other, simpler approaches for learning couldn’t learn to decouple this information.\n\nEDIT: The new revision does address this a little bit, so I am updating my score.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept