Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studied adaptive content-dependent online influence maximization where the seed nodes are sequentially activated based on real-time feedback. The authors formulated the problem as an infinite-horizon discount MDP and proposed a model-based RL algorithm that only requires node-level feedback. They proved the first sublinear regret bound for the adaptive content-dependent online influence maximization problem and validated the effectiveness of the algorithm on both synthetic and real-world data.
strength_and_weaknesses: Strength
1. This paper is the first to study the adaptive online influence maximization (OIM) problem, which is an interesting direction motivated by the recent advances in offline adaptive influence maximization.
2. The infinite-horizon discount MDP formulation of the adaptive OIM problem is reasonable.
3. The authors designed a model-based RL algorithm that only requires node-level feedback, while most previous works rely on edge-level feedback. They also proved the theoretical regret bound of the algorithm, which is the first sublinear regret bound for online adaptive influence maximization.

Weaknesses
1. The Bernoulli Independent Cascade Model relies on Assumption 3. However, IMHO, this assumption is relatively strong: it suggests that with larger $N$ and $K$, the A values should be smaller. This may not be reasonable in practice since the “influence” between two users should not be affected by the total number of users or contents.
2. In the infinite-horizon discounted MDP formulation, the action at each timestep is to activate just one user-content pair; it would be more interesting to consider the action of multiple user-content pairs per timestep.
3. I’m a bit concerned about how to obtain the user feature $x$ and content feature $\theta$ for real-world OIM problems.
4. In the problem formulation, the authors argue they focus on the asymptotic regime of large networks where $T<<N$, while in the explements, $T$ and $N$ are close (e.g., N=300, T=1000 in Fig. 1).

clarity,_quality,_novelty_and_reproducibility: This paper is generally easy to follow. It might be better to have more discussion on the regret results in Theorem 2 and 3. 
summary_of_the_review: This paper studied a new adaptive online influence maximization problem and provided an RL algorithm with theoretical regret bound. My main concerns are Assumption 3 for the Bernoulli Independent Cascade Model and the availability of user and content features in real problems. Also, the social networks considered in the experiments are relatively small (~2000 nodes).
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The adaptive version of the online adaptive topic-aware influence maximization problem is considered in this work, where its primary objective is the spreading of specific content in social networks. The problem is formulated as an infinite-horizon discounted MDP, and a model-based reinforcement algorithmic scheme, called MORIMA, has been proposed that learns a policy adaptively. The authors provide a regret analysis for the proposed algorithm. Experiments have been conducted on small synthetic and real-world networks, while the proposed algorithm has been compared with the IMLinUCB algorithm.

strength_and_weaknesses: **Strengths**
- The examination of the online influence maximization problem under the adaptive setting is quite interesting, as resembles what happens in the real-world. 
- The adoption of a model-based reinforcement learning algorithm for selecting seed users adaptively.
- A regret bound is provided for the proposed algorithm.

**Weaknesses**
- In general, the paper is well written, nevertheless, some of its parts need to be clarified.
- The exploration efficiency of the proposed reinforcement learning algorithm is not clear.
- The applicability of the MORIMA algorithm in real-world networks with millions of users seems to be not possible.
- Experiments have been conducted only on small synthetic and real-world networks that are  
clarity,_quality,_novelty_and_reproducibility: The main question about the proposed algorithm is its scalability on huge graphs with millions of users. It is not obvious how the proposed algorithm can be applied in practice. For this purpose, experiments should be conducted on larger graphs in order to validate the algorithm's ability to maximize the influence of specific contents. Even the `real-world` networks are pretty small and are technically created by sampling multiple dense sub-graphs from the Twitter network.

Also, some points should be discussed further by the authors. 
- For instance, is not clear how the states are represented in the Q-learning algorithm. 
- How the agent is able to efficiently explore the environment? Is it achieved by using the bonus term? 
- What happens in the case where N is extremely large? Is it still possible to compute the covariance matrix $\Sigma_{t-1}$?
- Matrix $A^k$ is not well defined. The sum of the elements of each row should be equal to one.
- The idea that a user has more than one chance to influence their neighbors and that of cumulative reward is not new.


summary_of_the_review: The authors should address the limitations of the MORIMA algorithm in a clear way. As aforementioned, the main question of this work is its applicability in real-world networks with a large number of users. The potential negative social impact of this work should be discussed further. How can this work have a positive impact on our society and what is its main objective?
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a network diffusion process to model (user, content) feature dependent network propagation and formulate the problem as an infinite-horizon discounted MDP, with the goal to select seed users and customize contents to maximize network influence. Then it proves a sub-linear regret bound for the proposed algorithms, and validate their results empirically on both synthetic and twitter social network datasets. 
strength_and_weaknesses: Strengths: 
1. The paper proposes a network diffusion process to model (user, content) feature dependent network propagation and formulate the problem as an infinite-horizon discounted MDP, with the goal to select seed users and customize contents to maximize network influence. 
2. it proves a sub-linear regret bound for the proposed algorithms, and validate their results empirically on both synthetic and twitter social network datasets. 


Weakness: 
The paper only compares to random policy and IMLinUCB, lack of strong comprehensive baseline comparison and empirical evaluation. 


clarity,_quality,_novelty_and_reproducibility: Questions: 
1. Comparing to IMLinUCB, MORIMA variants seems to have low variance, so curious to understand why that's the case? 
2. Among MORIMA variants, MORIMA without slow switching (in green) seems to be more stable over timesteps comparing to MORIMA (blue). Should we factor in stability over timesteps in choosing these variants? If so, how? 
3. In Figure (b), "MORIMA with known A" seems to underperform the other two MORIMA variants when timesteps reaches 200+. Is that expected or why is that? 
summary_of_the_review: The paper proposes a network diffusion process to model (user, content) feature dependent network propagation and formulate the problem as an infinite-horizon discounted MDP, with the goal to select seed users and customize contents to maximize network influence. Then it proves a sub-linear regret bound for the proposed algorithms, and validate their results empirically on both synthetic and twitter social network datasets. 

In the experimentation section, the paper only compares to random policy and IMLinUCB, lack of strong comprehensive baseline comparison and empirical evaluation. In addition to that, there are some unclarity in the experimentation results, shared in below: 
1. Comparing to IMLinUCB, MORIMA variants seems to have low variance, so curious to understand why that's the case? 
2. Among MORIMA variants, MORIMA without slow switching (in green) seems to be more stable over timesteps comparing to MORIMA (blue). Should we factor in stability over timesteps in choosing these variants? If so, how? 
3. In Figure (b), "MORIMA with known A" seems to underperform the other two MORIMA variants when timesteps reaches 200+. Is that expected or why is that?
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper studies a new adaptive online influence maximization problem and proposes a RL algorithm with sublinear regret. The reviewers all agree that the contribution is sound, however, they raise concerns regarding some of the assumptions regarding the model & the experimental evaluation (e.g., small graphs, baselines). 

In my own reading of the paper, I find the model used by the authors very simplistic. As a consequence, it is unclear what would be the impact of the proposed algorithm in practice. In this context, it is rather surprising that the authors seem completely unaware of an extensive line of work in the machine learning literature (e.g., Du et al., NeurIPS 2013, JMLR 2017) which has shown that temporal point processes are strictly superior at capturing the spreading dynamics of influence and information in real networks than discrete-time models as those used by the authors.
justification_for_why_not_higher_score: Simplistic model. Insufficient experimental evaluation.
Decision: Reject