Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors study the problem of incentivizing heterogeneous agents to explore in multi-armed bandit setting. The OTI algorithm with two phases is proposed. Theoretical guarantees are established for general agents, and for agents with UCB exploration. Moreover, a byproduct of their analysis suggests UCB's conservativeness.
main_review: The writing quality and clarity are good. The idea and the approach are novel to the best of my knowledge. And the problem setting is relevant and relatively new. Below are some questions/suggestions:

(1) Why does the principal want to identify the best arm that has the highest average reward among the agents who are in the test? I hope that the authors add some discussions to this issue. For instance, if a company is only testing a product on a group of people, there may be a gap between the best arm for this group and the whole population (all the customers). Can the algorithm and the analysis be carried out under the setting where there is a ground truth for the best arm? If not, what's the technical barrier?

(2) Effect of number of agents: 

(i) In the theory section, the authors highlight "proportional to $1/M$". However, there is a $\sum_{m \in [M]}$ before that term. Wouldn't this cancel the term $1/M$?

(ii) In the experiment section, $M = 60$ agents seems to be a quite large number compared to $K = 3$ arms. I think extra experiment results with larger number of arms $K$, each case run with different $M$, are desirable. With only one $K$ tested, the real dependency among $C(T)$, $K$, and $M$, seems very unclear.

Overall, the paper makes solid contribution and extension on a relevant and relatively new (but important) problem.


limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposes incentivized exploration in multi-armed bandits (MAB) that includes multiple and long-term strategic agents that have individual learning abilities and objectives. They find that the intrinsic needs of the agents to learn benefits the principal and that increasing the number of these agents can result in (almost) free exploration process for the principal. The contribution of the paper is that it goes beyond previous researches in the field that dealt with temporary myopic agents and studied long-term strategic agents for MAB, proposing Observe-then-Incentivize (OTI) algorithm in the process. 
main_review: The paper proposes incentivized exploration in multi-armed bandits (MAB) that includes multiple and long-term strategic agents that have individual learning abilities and objectives. They find that the intrinsic needs of the agents to learn benefits the principal and that increasing the number of these agents can result in (almost) free exploration process for the principal. The contribution of the paper is that it goes beyond previous researches in the field that dealt with temporary myopic agents and studied long-term strategic agents for MAB, proposing Observe-then-Incentivize (OTI) algorithm in the process. 

Strengths:

The paper is clear about the contributions and the extension of previous MAB settings to incorporating long-term strategic agents has strong real-world connections and significance. It is relevant to the NeurIPS community because the study deals with decision-making process of the principal and the agents and their interactions. The Observe-then-Incentivize (OTI) algorithm is a framework that can be extended further and be applied to diverse settings where the principal interacts with agents with varying self-interests. The mathematical formulation of the setting is clear and sound.

The paper is clearly written and communicates thoroughly the connection to real-world applications. The paper is well organized in sections and is easy to follow. Also, the mathematical formulation and the setting are clearly discussed in detail and understandable. 
The claims and methods are correct given their assumptions and principles. The empirical methodology is correct but would be much better if carried out with more agents and arms.  


Weaknesses:

The experimental section is a little weak and could be expanded with more large-scale experiments and discussion. The “Ban” protocol is used to ensure that the agents pull the incentivized arm. According to this protocol, when an agent refuses to pull the arm that is incentivized by the principal, the principal bans that agent and does not offer any incentive in the future. This strategy seems to be limiting for the agents and breaks the high connection to real-world scenarios. More discussions need to be in place to justify this strategy.  


limitations_and_societal_impact: Yes.  The authors have adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper studies incentivized exploration in multi-armed bandits with long-term strategic agents. The question of how to incentivize agents to explore underrated actions/arms has become an increasingly popular topic in the recent few years. Yet, it is still a relatively underexplored area for studying how to incentivize UCB agents. Therefore, I think the proposed method contributes to a significant problem. 
main_review: Below I want to summarize some of my concerns.

1. Best Arm Identification
 
In this paper, it is assumed that the principal aims to find the (global) best arm. I think the authors should explain more about why such a goal is significant in this setting, although three relevant papers with the same goal (best arm identification) are cited. Personally, I think "which can be used in the future" is not enough. 

In the cited paper (Audibert et al., 2010), there is only one agent. In this paper, there are multiple agents with different utility functions. The heterogeneity of the agents makes the problem a little bit different. For example, if there are only two arms A and B. Half of the agents hate A and like B, while the other half like A and hate B. In such a setting, finding the "averagely" best arm seems to be useless. In recent years, "personalized incentives" and "personalized recommendations" have received more and more attention. In this sense, finding a global best arm with respect to all of the agents seems to be less attractive. 

This problem is not relevant to the algorithms. What I want to stress is that I hope the authors can identify some concrete scenarios in which identifying the best global arm plays an important role. 

2. Take-or-Ban

In the take-or-ban protocol, an agent will be marked as "banned" forever as long as it refuses the bonus once. I hope the authors can specify more clearly the reasons for devising such a protocol. I'm not sure what "to avoid intractable agents behaviors" means: psychologically or mathematically intractable? Also, without such a protocol, is your algorithm still able to find the best arm? 

Here I understand such a protocol can simply the scenario, especially the agents' behavior. While I also want to see if it is a prerequisite for the proposed algorithm. 

3. Related Works.

Some of the papers cited in Section 6 helped me a lot in understanding the motivations. Yet, the related works are summarized on the last page of the paper. I think it may be a better idea to present the related works just after the introduction part as normal because without presenting enough relevant papers, it is difficult to illustrate that the assumptions on the agents are different.


limitations_and_societal_impact: The suggestions are included together with the main review.
review_title: null

Meta-review: All reviewers are positive with this submission. The reviewers agree that this paper is studying a relevant and new problem, and it is well-written. The reviewers also mention some suggestion for motivating the problem and the presentation order. I think the paper can benefit from these suggestions.
Decision: Accept (Poster)