Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper investigates injection attacks on recommender systems, where an adversary injects a number of fake user profiles into the training data of the system, with behaviors chosen such that the system is more likely to recommend specific items to real users. To this end, the authors investigate existing attack schemes and find that they spend too much effort -- and thus fake-user budget -- on targeting users that are hard to recommend the chosen item to (i.e. the attacks are difficulty-agnostic) and that they predominantly target the chosen item to users belonging to a specific cluster of the population (i.e. they suffer from a diversity deficit).
To mitigate these issues, the authors propose two new targeting objectives: a difficulty-aware objective that focuses the optimization effort on users closer to the top-k threshold of having the target item recommended and a diversity-aware objective that steers the optimization towards users where it can cause the largest increase in recommendation score. These objectives are then combined to create a new type of attacker called DADA (difficulty-aware and diversity-aware), together with a greedy algorithm that optimizes the joint objective. In an empirical evaluation on three datasets with three types of recommender systems, the authors demonstrate that DADA outperforms existing attacks, leading to a higher number of users that receive the target item(s) in their top-k recommendations.
strengths_and_weaknesses: ### Strengths
- The paper proposes two interesting hypothesis for why existing injection attacks on recommender systems can fail to be effective in promoting items to more users: a) being difficulty-agnostic and wasting attack-budget on users where it likely won't make a difference instead of succeeding for more easier users and b) suffering from a lack of diversity in in the set of targeted users and only reaching sub-communities.
- The paper introduces two new targeting objectives as well as a way to combine them to mitigate these problems. It empirically demonstrates that these objectives lead to more effective attacks where the target item is recommended to more users, outperforming prior approaches.
- The paper is overall easy to understand and follow, though there are a number of presentation issues (see weaknesses).

### Weaknesses
- It is not clear how the diversity-aware objective increases diversity, and more justification is needed. See the questions part for more details.
- More clarity is needed regarding the threat-model. The paper assumes a gray-box threat-model where attackers don't have access to the targeted system, but have knowledge of the user data used to train the system such as movie viewing history. This seems like a strong assumption to me, though I'm willing to accept it, given that prior work also makes it. However, there needs to be more discussion of how this training data can be obtained, i.e. which pieces of information about user behavior are typically accessible on platforms and whether they are sufficient to train a surrogate model.
- As far as I can see and inferring from the review-checklist, the results (e.g. in table 3 and 4, figure 1) are only computed for one run. Given that there are a number of stochastic components involved here (the selection of the target items, k-means initialization, etc.), results should really be provided as aggregates over multiple runs, else the confidence in the conclusions is low. If the based on multiple runs this should be clearly stated and e.g. error bars should be provided.
- There are some presentation issues:
    - I have an issue with some of the formulations repeatedly used throughout the paper, for instance:
        - The phrase "without loss of generality" is used a few times (e.g. in section 3.1.1 and 3.2.1) to refer to some of the choices made in the experiments. However, in its precise use, this phrase is reserved for assumptions in mathematical statements and not to justify design choices in empirical experiments. If you would want to use it there, you should show that other choices lead to similar results.
        - The phrase some "users contribute more gradients" is used a number of times. It's possible to understand it's meaning, but it feels rather imprecise. It could be understood as some users appearing multiple times in the objective and therefore being counted more than once in the gradient computation. An alternative such as some "users dominate the gradient computation" or something similar seems preferable.
    - There are a number of language and grammatical issues that make the paper feel not very polished.
questions: - Why does the diversity-aware objective increase diversity, i.e. why did you chose it to be what it is? As far as I understand, the objective seems to primarily focus optimization effort on users where this effort would maximize the increase in targeting score for the target item. This in itself seems reasonable, but I am not convinced that it will necessarily increase the diversity in the targeted set of users. There also seems to be no metric that quantitatively measures diversity in the paper, so the provided visual results (scatter plots) feel a inconclusive regarding a lack of diversity for existing methods as well as an increase in diversity as the result of applying the diversity-aware objective. To be clear: I do not necessarily think that the objective is a bad one, but I don't think it can be called diversity-aware with the current amount of justification and doing so can be misleading.
- Were the experiments conducted over multiple runs? It looks like there is only one run (see weaknesses).
- As an ablation, it might be interesting to study some of the existing objectives (e.g. RevAdv) with the change of zeroing gradients corresponding to users where the target item is in the top-k recommendations. This might reveal how much of the gains come from the changed objectives, vs how much from concentrating optimization effort on users where the goal is not reached yet.
- The description of the attack objectives describes the case where there is one target item, however, in the empirical analysis target sets of multiple items are used. In this case, do you zero-out the contribution of a user to the gradient as soon as one of the target items appears in the top-k recommendations or do you continue until all items are recommended?
- Why is the difficulty-aware objective called $Ten(.)$?
limitations: - The proposed attack might run into some scalability issues, since for any change in the training dataset or for generating more or less fake users a full re-run is required. Since this is an attack and not a production system it is probably ok, but it might be helpful to briefly mention this and maybe discuss whether there is the possibility of a warm-start using a previously computed solution.
- Again, the threat-model should be discussed more extensively in this context, in particular which types of platforms offer enough public data to enable the attacks.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper pointed out two issues of existing injective attacks against recommender systems, difficulty-agnostic and diversity-deficit issues.
To address these two issues, the paper propose a difficulty and diversity aware attacker, name DADA, to force the attacker to attack easy users distributed in various communities.
The experimental results on three real-world datasets demonstrate that the proposed method achieves superior performance compared with state-of-the-art attackers.
Besides, the analytical experiments also show the proposed objectives could generated fake users indeed focus on easy users in a diverse way.

strengths_and_weaknesses: ## Strengths
* This is a well-structured paper with clear presentation. 
* Motivational observation subsections are very insightful.
* The paper conducted very solid experiments with state-of-the-art baselines on three benchmark datasets.


## weakness
*  The paper lacks explanations for some phenomena in the experiments. For example, the results in Table 2 and Table 3 imply that stronger models are more robust to attacks.
*  Hit Ration ignore the rank of the target items in top-N recommended list. Thus, metrics like NDCG should be also reported in the experiments.

questions: see above
limitations: The method proposed in this paper could be used to attack the real-world recommendation application for profits.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: In this paper, the authors pointed out two issues with existing recommender system attackers, difficulty-agnostic and diversity-deficit. They then analyzed these two issues and proposed two ways to alleviate them. Both methods modify the loss function and introduced new notions in the gradient. Comparison with multiple baselines over the ML-100K and ML-1M datasets showed that the proposed approach can outperform all of them.
strengths_and_weaknesses: Strength:
1. The diversity-agnostic issue is well explained and justified. The proposed solution is intuitive and reasonable.
2. The authors conducted comparisons with more than 10 baselines and did very detailed analysis.
3. Different recommendation algorithms (WRMF, NCF, LightGCN) were experimented with, which demonstrates the generalisability and robustness of the proposed approach.

Weakness:
1. The diversity-deficit issue is not very convincing to me. Do the fake injected users always form a cluster that is always a subgroup of a larger cluster of users? More explanations and proof (if there is any) of this phenomenon can be very helpful to the readers.
2. I wonder if the choice of \beta_1 and \beta_2 can reflect the real-world scenario. In other words, is it possible to inject 1% of users with 1% of items into a recommender system in real world? This seems to be quite unlikely. It would be very interesting to see the results when \beta_1 and \beta_2 are much smaller (0.01%ish), which can mimic the real-world scenario better.
3. It would be helpful to annotate if the results in Table 1 and 2 shows significant differences.
4. It would be great and super helpful to have more discussions (with more details) on the defense of the proposed attack.


questions: Please see my questions in the last section.
limitations: It would be great and super helpful to have more discussions (with more details) on the defense of the proposed attack.
review_title: null

Meta-review: metareview: This paper explores an interesting new attack vector against recommender systems -- one that targets recommending an item to users whose top-K lists are easier to manipulate for that item and to a diverse set of users from different interest communities.

Overall, the paper does a reasonable job with showcasing the effectiveness of the attacks. However, there are concerns both about the actual effectiveness of the attacks in practice, where defenses can use external information to detect the fake user profiles (based on their behaviors) used to conduct the attack as well as about the ethical concerns raised by the lack of an appropriate defense strategy for the attack, if it were really effective in practice. The authors are strongly encouraged to discuss the limitations and ethical concerns raised by the attack. 

In summary, this is a paper that deserves to be accepted out of procedural fairness -- it has been rated well by the three reviewers. It does describe a cute intellectually interesting attack, even if it is unlikely to be practical or useful.  
confidence: Less certain
Decision: Accept