Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper considers a combinatorial multi-armed bandits (CMAB) setting where multiple users interact with a learning algorithm. The algorithm recommends arms and the users decide whether they want to follow algorithm's recommendation (explore) or choose different arms (exploit). The paper considers the problem of incentivizing identical Bayesian users to follow the algorithm's recommendation. Precisely, an algorithm is considered Bayesian Incentive Compatible (BIC) if for every trial the algorithm plays an arm that has the best expected reward with respect to prior belief of a given user. The paper shows that the Thompson sampling algorithm for CMAB satisfies BIC given a sufficient number of bootstrap samples for each arm. The paper then considers the goal of designing BIC algorithms for providing bootstrap samples and proposes two algorithms: (1) based on a reduction to incentivized K-armed bandits problem, (2) based on a reduction to incentivized reinforcement learning.  
strengths_and_weaknesses: Strengths: The paper is very well-written with proper intuition for various assumptions/ideas. Prior work has also been addressed adequately. 

Weaknesses: I think that the contribution of the paper is incremental in nature given the amount of existing work on incentivizing exploration. For example, it was already shown by Selke & Slivkins (2021) that Thompson sampling is BIC for classical bandits and it does not seem surprising that the same result also holds for the combinatorial bandits setting (albeit for a different number of bootstrap samples). The bootstrap algorithms proposed by the paper are also based on existing ideas in Mansour et al. (2020) and Simchowitz & Slivkins (2021). 
questions: I would request the authors to specify the challenges in achieving incentivized exploration for combinatorial bandits, and the technical ideas used to overcome these challenges. 
limitations: I do not foresee any negative societal impact.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper investigates the incentivized exploration problem in combinatorial semi-bandits setting. The authors prove that Thompson Sampling is incentive-compatible when initialized with a sufficient number of samples of each arm in combinatorial semi-bandits. In addition, the authors design 
strengths_and_weaknesses: The paper is well-written and compares with previous literature thoroughly. The main results of this paper are solid, however, I have no enough expertise to judge whether the proof technique is trivial or not compared with the existing literature. The initial exploration algorithms seems rely on the reduction to previous literature and the BIC-bandit expert can help to judge the technical contribution of this paper. 

Weaknesses: This paper is a bit incremental from my perspective, which generalizes previous incentivized exploration to combinatorial semi-bandit problems. The significance of this paper needs to be justified further by BIC-bandit expert.

Overall, this is a good theoretical paper. Whether to accept the paper depends on the technical contribution of this paper.
questions: N/A
limitations: N/A
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: This paper studied incentivized exploration problem, and provided some theoretic study of using combinatorial semi-bandits to solve the problem. The paper does not seem to be complete, since there is no experimental sections to show the results to support the proposed algorithm.
strengths_and_weaknesses: The mathematical framework seems ok, including Thompson Sampling, BIC algorithm and so on. However, there is no experimental justification at all.
questions: Incentivizing Combinatorial Bandit is an interesting research topic. However, without experiment justification, there is no point to show the paper in the high standard NeurIPS conference. 
limitations: Since there is no experiment result presented in the paper, it is hard to reproduce the work.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This work investigates incentivized exploration in a combinatorial semi-bandits setting. There are two main theoretical results. First, it was proven that Thompson sampling (TS), when concatenated with a first algorithm that generates at least given number of samples for each atom, is Bayesian incentive-compatible (BIC). Second, the authors propose two approaches, both adapted from the existing literature to handle the additional challenges of combinatorial semi-bandits, that satisfy the requirements of the first algorithm before TS.
strengths_and_weaknesses: Strengths:

1) The studied problem of incentivized exploration in combinatorial semi-bandits is novel. In particular, TS is considered as the bandit learning algorithm, which is both interesting and challenging.

2) Theoretically establishing that TS is BIC is a strong contribution.

3) The two approaches to satisfy the requirements of $\texttt{ALG}$ are interesting, particularly in the combinatorial semi-bandits setting.

Weaknesses:

- There lacks a discussion of motivating/matching applications of incentivized exploration (which is well motivated in prior papers) specifically for combinational semi-bandits. It was briefly mentioned in Page 2 but I feel that a detailed example would be beneficial. 
- A discussion of how to handle $K>2$ in Section 3.1 would be valuable.
- It is worth discussing the constant (number of samples) in the theorems. Will they blow up $T$?

questions: - The second approach (Sec. 4.2) seems an overkill given that Simchowitz and Slivkins (2021) handles the more difficult episodic RL setting. Is there any way to simplify?
- Line 291, should it be (11) instead of (7)?

limitations: The authors did a great job explaining limitations and social impact in the Introduction section.
review_title: null

Meta-review: metareview: This paper investigates the problem of incentivized exploration in the combinatorial semi-bandits setting.  The reviewers are overall positive about the paper. The main concern of the paper is that the contribution is incremental given the number of prior works on incentivizing exploration.  The authors' responses have more explicitly addressed this concern, and we encourage the authors to incorporate their responses into the paper.  There have also been issues brought up about the lack of experiments, but I agree with the authors that given the main contribution of this work is theoretical, having empirical evaluations is a plus but not required.  Overall, I believe the contribution of the paper outweighs the concerns and would therefore recommend acceptance.
confidence: Certain
Decision: Accept