On the Detection of Reviewer-Author Collusion Rings From Paper Bidding
Authors: Steven Jecmen
Nihar B Shah
Fei Fang
Leman Akoglu
Venue: Accepted by TMLR ()

Abstract: Collusion rings pose a significant threat to peer review. In these rings, reviewers who are also authors coordinate to manipulate paper assignments, often by strategically bidding on each other's papers. A promising solution is to detect collusion through these manipulated bids, enabling conferences to take appropriate action. However, while methods exist for detecting other types of fraud, no research has yet shown that identifying collusion rings is feasible. 

In this work, we consider the question of whether it is feasible to detect collusion rings from the paper bidding. We conduct an empirical analysis of two realistic conference bidding datasets and evaluate existing algorithms for fraud detection in other applications. We find that collusion rings can achieve considerable success at manipulating the paper assignment while remaining hidden from detection: for example, in one dataset, undetected colluders are able to achieve assignment to up to 30% of the papers authored by other colluders. In addition, when 10 colluders bid on all of each other's papers, no detection algorithm outputs a group of reviewers with more than 31% overlap with the true colluders. These results suggest that collusion cannot be effectively detected from the bidding using popular existing tools, demonstrating the need to develop more complex detection algorithms as well as those that leverage additional metadata (e.g., reviewer-paper text-similarity scores).

Published in Transactions on Machine Learning Research (12/2024)
On the Detection of Reviewer-Author Collusion Rings From Paper Bidding
Steven Jecmen
sjecmen@gmail.com com com
Carnegie Mellon University
Nihar B. Shah
nihorstics.. cmaa. edu
Carnegic Mellon University
Fei Fang Carnegie Mellon University
feifang@emmu.cm edu
Leman Akoglu
lakogluSandrew..cm cmnu, edu
Carnegie Mellon University
Reviewed on OpenReview: https: //
// operreveew..ee....m net/ forum? id- o58uy91 V2V
Abstract
Collusion rings pose a significant threat to peer review. In these rings, reviewers who also authors coordinate to manipulate paper assignments, often by strategically bidding each other's papers. A promising solution is to detect collusion through these manipulated bids, ena enaiing conferences to take appropriate action. However, while methods exist for detecting other types of fraud, no rescarch has yet shown that identifying collusion rings is feasible.
In this work, we consider the question of whether it is feasible to detect collusion rings from the paper bidding. We conduct an empirical analysis of two realistic conference bidding datascts and evaluate existing algorithms for fraud detection in other applications. We find that collusion rings can achieve considerable success al manipulating the paper assignment while remaining hidden from detection: for example, in dataset, undetected colluders are able to achieve assignment to up to 30% of the papers authored by other colluders. In addition, when 10 colluders bid on all of each other's papers, no detection algorithm outputs a group of reviewers with more than 31% overlap with the true colluders. These results suggest that collusion cannot be effectively detected from the bidding using popular existing tools, retonstrating the need to develop more complex detection algorithms as well as those that leverage additional metadata (e.g., reviewer- papee sixi-similarity scores).
1 Introduction
In scientific peer review, researchers are called upon evaluate the work of other researchers the same scientific community and provide recommendation as whether each work should be publishod. Within the field of computer science, academic conferences are some of the primary venues of publication. These conferences receive large pool of paper submissions from authors, and they rely to accurately assess the quality of each submission and determine which to accept or reject. The resulting acceptance decisions can have substantial impacts on the careers of the authors, as publication of a paper in a competitive computer science conference is often used as an indication of research quality. Thus, is vital that the peer review processes of these conferences are fair to the authors.
peer review process
However, the perceived prestige of conference publication can incentivize authors and reviewers act in unethical ways. One such form of unethical behavior that has occurred in computer science conferences is

---

Published in Transactions on Machine Learning Research (12/2024)
Conferences recruit many (if not all) qualified authors of submitted papers to serve reviewers, and consequently many reviewers at a conference are also authors of one or more submissions to that same conference. In a collusion ring, a group of reviewers and authors at a particular conference coordinate to manipulate the paper assignment such that they are assigned to review each other's papers. The most straightforward way that colluding reviewers can manipulate the assignment through "paper bidding," " a common phase of the paper assignment process. During the paper bidding phase, each reviewer is asked express their level of interest in reviewing each paper. These bids are then taken into account when determining the paper assignment and often have large influence (Jecmen et al. 2020; Wu et al. 2021 Leyton- Broonn et al. 2022). Thus, colluders can and do use strategic bidding to achieve a desired paper assignment (Littmaan 2021]:
that of collusion rings (Vijykkuarr 2020 Littman 2021].
"The colluders hide confticts of interest, then bid to review these papers, sometimes from duplicate accounts, in an attempl to be assigned to these papers as reviewers."
Once assigned, the colluders can give positive reviews to and push for the acceptance of other colluders' papers.
Approaches to addressing academic fraud can be divided into mitigation-based and detection--asee meth- ods (Wilson] 2020). Several past works have proposed modifications to paper assignment algorithms that aim at mitigating the impact of malicious bidding (Guo et. al., 2018: Jecmen al., 2020: Wu al.) 2021: Boehmer et al.) 2022 Jecmen et al. 2022). However, such approaches necessarily come with a cost in terms of assignment quality, as they ignore some aspects of reviewer preferences expressed through bids. Morcover, they do not identify which (if any) individuals were engaged in collusion. In contrast, detection methods have received little attention in prior work. Ideally, an accurate detection method could identify any colluders so that the conference can intervene to stop them. For example, a conference could decide to block the assign- ment of detected colluders to each other's papers, potentially leading to higher-quality assignments among the honest reviewers as compared to mitigation methods. Reliable detection methods could also allow the conference organizers to take more scrious action against the colluding individuals, perhaps after manual investigation provides further evidence. In the words of Littman 2021):
"Better paper- ansirnssiigmmnt technology would help close one loophole that is being exploited. Bul, without beller invesligalive lools, we may never be able to hold the colluders to accounl."
However, careless deployment of detection algorithms may result in false positives: honest reviewers and authors being falsely identified as colluders. This is a serious danger in scientific peer review, since falsely- accused reviewers and authors may have their reputations tarnished or their papers unfairly rejected. Es- tablishing effective methods to detect collusion rings is thus a critical path for research on this problem.
A long line of research has applied anomaly-detection techniques to successfully detect various forms of frand in other settings. Many of these settings involve a network of people in which the fraud appears as set of anomalous interactions: for example, auction fraud on online platforms, fraudulent financial transactions, fake reviews on sites such as Amazon, and fake accounts on social media (Akoglu et al. 2015). This raises an important question of whether these techniques can be similarly applied to effectively detect fraudulent interactions in the context of paper bidding for peer review. However, there is a vast range of approaches to detection in the research literature that could potentially be applied to this problem. To facilitate a thorough analysis, our work focuses on methods that attempt to detect collusion using only the paper bidding and authorship data (i.e., the data that directly reflects the strategic bidding of colluders). Our focus on bidding is additionally motivated by the fact that real-world investigations of collusion often begin with analysis of the bidding data. Furthermore, many conferences use bidding- based heuristics to mitigate collusion (e.g., ignoring the bids of reviewers with too few positive bids; Jeemen et al. 2022). Concretely, we consider the question: is it possible to effectively detect collusion rings from only the bids and hutshih?p?? Answering this question alone requires an extensive empirical analysis. However, resolving this question provides important direction for future research on detecting collusion rings. Our work does not attempt to investigate various other questions of interest to the problem of collusion detection: for example, comparing the effectiveness of detection- and mitigation-based approaches, conducting game-theoretical analysis of
2

---

Published in Transactions on Machine Learning Research (12/2024)
the detection problem, gathering new data on the behavior of real colluders. As additional motivation for our focus on using only bidding data, many conferences in practice use only bids to determine the paper assignment (with the potential addition of some coarse subject-area matching); for instance, this is the case for most conferences hosted on the popular HotCRP conference management system (where the first instances of collusion were caught.).
Within the realm of detection algorithms, we consider algorithms that attempt accurately identify single group of colluders rather than those that identify a list of several Aikentiall--colluddig groups with lower confidence. While algorithm outputting multiple groups would be more likely to output the colluding group, the aforementioned cost of false positives in our setting means that conferences should rightly be hesitant to accuse a reviewer of collusion without a high degree of certainty. Detection algorithms with a high false-positivy rate could instead be applied as part of a collusion collusion-mitigattii mitigation approach in which a large number of reviewer-paper pairs are flagged as potentially collusive and blocked from being assigned. However, we view this class of algorithms as falling within the "mitigation" " category rather than the " "detection" " category we focus on, and such should be compared to to the other collusion-miilgaation approaches that have been proposed in the literature.
One major challenge in the peer review setting is that there is no ground-truth data about how colluding reviewers behave in practice. In this work, instead of assuming a specific form of collusive bidding behavior, we address this challenge by considering a wide range of parameterized behavior for the collusion rings. Specifically, since the purpose of a collusion ring is to achieve colluder-to-collude paper assignments, we vary the density of bidding between colluding reviewers and polluder-authored papers. Denser bidding corresponds to stronger attack, but also may allow the colluding group to be more easily detected. Thus, our analysis aims to establish the regions of this parameterized space of colluder behavior at which collusion can be effectively detected.
Our contribution is a set of empirical results on two different realistic bidding datasets concerning the feasibility of detecting collusion rings. Our analysis consists of three parts. First, we characterize the typical density of bidding found within groups of honest (non-colluding) reviewers, as high-density groups of honest reviewers are potential false positives for detection algorithms. Second, we evaluate the performance of several established algorithms at detecting injected collusion rings based on the anomalous density of bidding within the ring. We consider a combination of fundamenial approaches to finding dense subgraphs and density-basee based fraud detection methods, including two algorithms that have been shown to effectively detect fraud in other settings. Third, we evaluate the success of the injected collusion rings at achieving desired paper assignments, contextualizing the potential impact of fraud.
their
Overall, our results suggest that collusion rings cannot be effectively detected from only the bidding and authorship data using popular existing algorithms. Our findings include the following:
Using only the bidding and authorship data, all detection algorithms we analyze fail to detect some injected collusion rings that are larger than any honest-reviewer groups with a similar density of bidding. For example, when the collusion ring consists of 10 reviewers who bid on all of each other's authored papers, the output of the best-performing detection algorithm across both datasets has only 31% overlap (Jaccard similarity) with the true colluders average.
Colluders are able to achieve assignment to a substantial fraction of the papers authored by other colluders while avoiding detection by all analyzed algorithms (30% and 24% in each of the two datasets).
A sizeable fraction of colluders can get at least one their papers reviewed by another colluder while avoiding detection (54% and 35% in each of the two datasets).
These results suggest that future research on detecting collusion rings must focus on more complex detection methods, especially those that leverage other metadata (such as reviewer-paper text-similarity scores or reviewer publication history).
The code and data we use in our analysis are available online at Terieew-collusionddeteeee
cow comm//jccnnne/pperr
3

---

Published in Transactions on Machine Learning Research (12/2024)
2
Related Work
Recent rescarch has looked at improving various aspects of the peer review process (Shah 2022 In this line of work, various solutions have been proposed to the problem of reviewer-author collusion that we consider (Jecmen et al. 2022). Jeemen et al. (2020) propose an algorithm for randomizing the paper assignment such that the probability of a colluder achieving assignment to another colluder's ' paper is limited. Wu et al. (2021) propose learning a model of reviewer proferences from the bidding and using this model to determine the paper assignment, thereby reducing the influence of collusive bidding. This work also provides a semi-synthetic paper bidding dataset, which we use in our analysis. The works (Guo et al., 2018; Bochmer et al., 2022) consider the problem of finding a cycle-free paper assignment. In the 2021 AAAI Conference on Artificial Intelligence, a large AI conference, several precautions were taken to address the problem of collusion: two-cycles in the paper assignment were disallowed and a constraint on the countries of the reviewers assigned to the same paper was added (Leyton- Broonn et al. 2022). Notably, these solutions all aim to mitigate the impact of collusion rings on the assignment, rather than detecting them outright. While the present work focuses on detecting attacks in the paper bidding phase, other works show that attacks on the text-similarity algorithms used by conferences can be effective (Markwood et al. 2017: Tran & Jaiswal. 2019 Eisenhofer et al., 2023
Most relevant to our work is the prior work (Jecmen et al. 2023 which observes paper bidding in mock conference setting and collects data on the strategics employed by participants acting as colluding reviewers. In contrast, rather than attempting to precisely simulate the behavior of colluding reviewers, we consider an intentionally simplified setting where malicious reviewer behavior can be easily parameterized, allowing us to sweep out a wide range of potential colluder strategics. We note that the most common colluder strategies observed by Jeemenn et al. (2023) consist of injecting dense bidding between colluders, as is done by the strategies considered in our analysis. [Jemenn et al. (2023) also evaluate the performance of very simple detection methods on the collected dataset, while we consider a set of more complex detection methods based on denso-subgraph discovery.
Outside of the setting of scientific peer review, similar problems of fraud have been studied in the anomaly detection literature. In online platforms such as Amazon or Yelp, several methods have been proposed to detect products or sellers who purchase frandulent reviews from users dAkoglu et al.) 2013 Eswaran et all, 2017 Kumar et al., 2018). Other works propose density-based methods for detecting groups of fraudsters in these and similar online network settings (e.g., fraudulent transactions
cBay or fake followers on Twitter) (Pandit et al.) 2007 Prakash et al. 2010: Hooi et al., 2016); we evaluate the performance of (Hooi 2016) in our analysis. However, our setting (scientific peer review) is distinct from these online platform
et al. settings in a few ways. Most significantly, our work focuses on collusion in the paper bidding phase, where the objective of colluders is to achieve a dcsired outcome in the subsequent paper assignment; in contrast, the fraudulent reviews are themselves the objective of fraudsters on (e.g.) Amazon. As a result, the incentives for peer-review colluders are not the same as those of frausters in the other settings: for example, since each reviewer is assigned to review limited number of papers, "camouflaging" by bidding positively on non-colluder papers may result in those papers being assigned instead of the targeted papers. In addition, fraudulent interactions on online forms are often provided by large numbers of fake accounts specifically used for fraud; since making fake accounts on common peer-review platforms is non-trivial, interactions between colluders in peer review may be more often done under their real identities, leading to different patterns of behavior. Numerous works have proposed other methods for graph-based anomaly detection, including those that address online spam, cyber-attacks, and other forms of frand (Akoglu et al. 2015). Generic approaches for dense-subgraph detection (Goldberg) 1984; [Tsurakakiss et al. 2013) Hooi et al. 2020) can also be applied to detect anomalies in graphs; we evaluate these methods in our analysis.
Aside from reviewer-author collusion rings, reviewers can attempt to dishonestly improve the chances of acceptance of their own papers in other ways. One such form of malicious behavior occurs when reviewers provide negative reviews to the papers they are assigned, thereby improving the relative standing of their own authored work. In contrast to the issuc of collusion rings, which require the cooperation of multiple reviewers, this sort of "strategic reviewing" can be done by each reviewer alone. In a controlled experiment, Balietti et al. (016)) found that people do exhibit strategic reviewing in a competitive peer evaluation setting.
4

---

Published in Transactions on Machine Learning Research (12/2024)
Several works have proposed methods to mitigate strategic reviewing with theoretical guarantees (Alon et al,
2011 Xu et al. 2019 Dhull et al. 2022). On the detect ion side, Stelmakh et al. (2021) design statistical test
for the presence of strategic reviewing and evaluate it on data collected from a peer evaluation experiment.
3 Preliminaries
In this section, we detail the setting of our analysis, the datasets we analyze, and the research quest ions we answer.
3.1 Setting
We consider a conference peer review setting with a set submitted papers P and a set of reviewers R. Conferences generally recruit the authors of the submitted papers to serve as reviewers, along with external, non-author reviewers. The authorship set A C R P contains all pairs (r,p) such that reviewer authored paper p. Additionally, the conflict-ofiittresst set C C R x P contains all pains (r,p) such that reviewer has conffict-of-intersst with paper p and should not. be assigned to review it (A S C). Once submissions are received, the conference asks each reviewer to indicate their level of interest on each submitted paper via paper bidding. While conferences often allow each reviewer to choose from a number of discrete levels (e.g., "Fager"; "Willing"; "Not willing"), we consider a simplified setting with binary bids (positive or neutral); note that allowing additional levels of bids can only give colluders more flexibility to manipulate the paper assignment. The bid set B C R P contains all pairs (r,p) such that reviewer bid positively on paper p (BNC= = 0). The conference also computes text-similarity scores between each reviewer and paper using function T R P + [0, 1], where T(r, p) indicates the level of similarity between the text of paper p and the text of the past work of reviewer r.
The conference then computes the paper assignment in the following manner. First, the conference computes similarity scores between each reviewer and paper using a function S x P [0, 1]. In our experiments, we use S(r. p) = ("(rr,))"((rpp))) ((-P(E)))) based on the function used in the 2016 Conference on Neural Information Processing Systems (NeurIPS), large machine-learning conference (Shah al.] 2018). The conference then computes assignment of papers to reviewers such that the total similarity of the assigned pairs is maximized, subject to constraints that (i) each paper is assigned to exactly 3 reviewers, (ii) each reviewer is assigned at most 6 papers, and (iii) no reviewer-paper pairs in C are assigned. While we use the staled values in our experiments, the exact reviewer and paper loads vary between conferences. The maximum-similarity assignment can be computed efficiently as a min-cost flow or as a linear program. This framework for paper assignment has been used in numerous conferences and venues (Shah 2022).
3.2 Datasets
We next provide details regarding the two datasets that we analyze in this work.
The first dataset, which we refer to as "AAMAS", contains subset of the real bidding from the 2021 International Conference on Autonomous Agents and Multiagent Systems, an AT conference. This dataset is publicly available from PrefLib (Mattei & Walsh 2013) and contains de-identified bids from reviewers that did not opt-out from data collection. In this conference, bids were selected from ("Yes", "Maybe", "Conflict", "No response") ); we consider "Yes" and "Maybe" responses to be positive bids and include those reviewer- paper pairs in B. We set C as the set of all reviewer-paper pairs with "Conflict" bids. Since the datasct does not include authorship information, we reconstruct the authorships by subsampling confficts-of-inteees uniformly at random for each paper. The resulting dataset has 526 papers and 596 reviewers, 398 of whom authored at least one paper. The datasct also does not contain text-similarity scores T(r.p). We generate synthetic text-similarity scores using the procedure described in Appendix A based on the text-similarities from the 2018 International Conference on Learning Representations reconstructed by Xu et al. (2019)
Our second dataset, which we refer to as "$2ORC", is the semi-synthetic datasct constructed and made publicly available by Wu et al.] (2021). This dataset contains synthetic bids between a large subset of published computer science papers and authors from the Semantic Scholar Open Research Corpus (Ammar
5

---

Published in Transactions on Machine Learning Research (12/2024)
designed to match statistics from the NeurIPS 2016 conference (Shah et al 2018 Bids were chosen from values {0, 1, 2, 3), where non-zero values indicate positive bid (included in B). For the authorship set A, we use the real authorships between the reviewers and papers in the dataset, discarding the 90 bids placed on self-authored papers. We assume that the conflicts-of-intrree are only the authorships A). The resulting data has 2446 papers and 2483 reviewers, 984 of whom authored at least one paper. This dataset also contains real text-similarity scores between each reviewer and paper T(r,p): computed using the popular TPMS algorithm (Charlin & Zemel] 2013). We compare the level of agreement between these text-similarity scores and the bids to those found by Stelmakh et al. (2023) and find that they have realistic level of error; see Appendix A for details.
et al. 2018:
In our analysis, we assume that both of these datasets contain only bids from "honest" (non-colluding) reviewers. The AAMAS dataset contains information from reviewers that did not opt-out from the data collection, and we expect that any colluding reviewers the conference would have done so. The S2ORC bids are synthetic and do not model malicious reviewer behavior.
3.3 Problem Statement
Our goal is to detect collusion rings. We suppose that there exists group of colluding reviewers M R who try to manipulate the paper assignment by altering their bids, with the aim of being assigned to review the papers authored by other members of the colluding group. The objective of collusion-detection algorithm is to output M given the set of bids B, along with the authorships A and the other conflicts C. Note that the text-similarities pr,,) cannot be used for detection in our analysis we consider the problem of detection using only the bidding itself and the authorships.
As our original datasets do not contain colluding reviewers, we inject collusion into the datascts by choosing group of reviewers to be the colluders M and modifying the bids of these reviewers (i.e., adding or removing elements of M P to/from B). The strongest form of collusion would be to add bids between all reviewers in M and all papers authored by other reviewers in M, while additionally removing all other bids by reviewers in M. However, malicious reviewers may not want to perform such an obvious manipulation out of fear of being caught. Due to the lack of concrete evidence on colluding groups and the exact bidding strategy that they might employ, we consider a wide range of possible colluding groups parameterized by both a size parameter (i.c., the number of colluding reviewers) and "density" parameter (roughly corresponding to the attack strength).
As there are some modeling choices involved in concretely defining the bidding density parameter, we con- sider two different notions of densily, corresponding to two different graph representations of the bidding relationships between the reviewers of the conference. The first representation (Section 4 is a unipartite graph in which vertices represent reviewers. The second representation (Section 5 is a bipartite graph in which vertices represent both reviewers and papers. We motivate and define each of these formulations in their respective sections. For each graph formulation, we investigate the following three high-level research questions:
Q1: For what values of size and density do groups of honest reviewers already exist in the datasct (without injected collusion)? (Sections 4.1 and 5.1
Q2: For what values of size and density groups of injected colluders be accurately detected by existing algorithms? (Sections and
5.2
Q3: At each size and density, how successful are groups of injected colluders at achieving their desired paper assignments? (Sections 4.3 and 5.3
All of our experiments in these sections were conducted on server with 515 GB RAM and 112 CPUs.
4 Unipartite Bidding Graph
In this section, we represent the reviewer bidding data in the form of unipartite, directed graph where each vertex corresponds to reviewer. The graph contains directed edge (r1, if reviewer rı bid
6

---

Published in Transactions on Machine Learning Research (12/2024)
1.0
1111
885
455
121
1.0
1.0-1652
1652
15555
123
882
520
218
54
edge density
0.9
1113
885
1668
1278
1412
3213
0R1
416
12
5
0
edge density
0.9
0.9-1652
1652
1555
4225
2976
4679
1,204
9596
1:154
4597
2474
1512
331
0.8
1113
6289
1,644
2.544
7.144
2.755 5.085 S.2ES
4016
0
1652
6015
2.244
1:144
5.354
L.7ES 3 4E5
185
0.7
1113
6289
6.144
4.5 966 7.1
2.157
5705
0.7
1652
6035
4.344
2.555
8:355
0.6
3.444
2:155
6.45
3.7
1.255
2
9.351
0.6
1652
9:454
1.255
2:556
1.777
2
0.5
1.354
2.8
2:19
1964
6.254
0.5
185
3
4
10 11 number of reviewers (k)
5
6
12
13
2
5
6
7
8
9
10
11
12
13
number of reviewers (k)
(a) Exact counts, AAMAS
(b) Exact counts, S2ORC
edge (n (v)
&
1.0
D 1.0
0.9
i 0.9
0.8
0.8
0.7
0 0.7
0.6
0.6
0 0.5
0.5
edg
2
6
10 12 14 18 24 26 30 32 number of reviewers (k)
4
8 10 14 20 22 24 26 28 30 32 34
number of reviewers (k)
(c) Groups found by greedy peeling, AAMAS
(d) Groups found by greedy peeling, S2ORC
Figure 1: Exact counts of honest-reviewer groups with varying size and edge density (Figures la 1b and the size and edge density of
groups found by a heuristic method (Figures Ic Id]. In Figures la 1b values in cells marked with "z" represent lower bounds since exact counts were infeasible to compute; the vertical axis represents lower bounds the edge density. In Figures [ each point corresponds to existing honest-reviewer group (found by a greedy peeling method), and the shaded area indicates the region in which there exists at least one honest-reviewer group. The vertical axis represents exact values for the edge density. Note that the vertical axis docs not start at for casier comparison with other figures.
least one paper authored by reviewer 72. Informally, the density a group of reviewers in this graph is the fraction of possible edges present between the reviewers. We formalize these definitions shortly. In Section we consider an alternative, bipartite graph representation of the bidding.
Our motivation for considering this graph representation is that it most naturally represents the reciprocal behavior expected of a collusion ring. Several existing works (Guo et al. 2018 Boehmer et al. 2022 Leyton- graph, aiming to
Brown et al. 2022) on reviewer-author collusion consider a similar reviewer-to-reviewe prevent cycles (i.c., "rings") between reviewers in the paper assignment. For our purposes, the notion of density implied by this graph has several useful properties. First, malicious reviewers may not attempl to manipulate the assignment of every paper they author. For example, each colluder might have one bad paper that they want their fellow colluders to review (e.g., because it is lower quality) and many good papers not involved in the manipulation. Even if each colluder authors many good papers, collusion would still appear as a dense subgraph. Additionally, collusion rings are based on quid-pro-quo arrangement between the colluders, where each colluder needs help getting some papers accepted to the conference. Each colluder therefore should receive some benefit from the other colluders, reflected as density.
Formally, we denote the (directed) graph by 91 = E1,, where V = and ε = (p1,((()))))) R (r1,p) B A (r2,p) A). Given this graph representation, we characterize potential group of colluding reviewers S C R in terms of two parameters. The first parameter ks = |S] is the size of the group. The second parameter is is the "edge density" of the group, defined as follows. For arbitrary graph G = (V,E) and any subset of vertices V' C V, we use E|A to denote the set of edges in the subgraph induced by V. We then define the edge density of S be 7s
[[[S] the fraction of possible edges present. We henceforth
((()))
omit the subscript S from kg and is since the subset in question will be clear from context.
We now analyze the problem of detecting collusion from G1. In the following three subsections, we provide empirical analysis to answer each of the three research questions ified in Section 3.3.
7

---

Published in Transactions on Machine Learning Research (12/2024)
4.1 Conest-Reviewer Groups (Q1)
In this subsection, we characterize the density of bidding among groups of honest (i.e., non-colluding) reviewers in our datasets. For each value of the parameters (k, 7), we count the number of groups of size k with edge density at lcast 7. Since we aim to detect collusion among authors, we consider only reviewers that have authored at least one paper. This shows the frontier of identifiable detection: if honest-reviewer groups exist at some (k,7): then a colluding group with that same size and density cannot be identified as suspicious with high confidence based on the bidding within the colluding group. Stated differently, this analysis gives the number of potential false positives for detection algorithm at each (k, Recall that false positives are a serious concern in collusion detection due to the danger to honest reviewers' reputations.
In Figures La lb we show the results of this analysis. These counts were obtained via a standard backtracking search. Cells marked with "2" were unable to be completed in 24 hours, as obtaining exact counts takes worst-case time exponential in k. Instead, the values in these cells represent lower-bounds. We note that these figures may be independently useful to conferences, since many conferences use houristic chocks for collusion based on reviewer bidding patterns as part of the paper assignment (e.g., not assigning certain reviewer-paper pairs or ignoring certain reviewers' bids). The results in Figures la 1b may help these conferences understand the false positive rates of such heuristics.
Since exact counts are infeasible for large values of k, we additionally use a heuristic method to find dense subgraphs of larger sizes. We start with the entire graph and iteratively remove the vertex of smallest degree to produce a sequence of subsets of decreasing size (commonly called "greedy peeling"). In Figures Ic ld we plot the edge density y of these subsets against the size k. Thus, for all points (ks,)) 7) in the shaded region, at least one group of honest. reviewers exists of size k and with edge density at least 7.
Overall, these results show that there is a large region of the space of (k; where colluding group could not be detected based on its size and edge density due to the existence of many similar groups of honest reviewers.
4.2 Detection Algorithm Evaluation (Q2)
For values of (k, 7) larger than those that exist in the honest bidding, we may hope that colluding groups can be identified as suspicious by an appropriate algorithm. However, this may not always be possible: since the size of the colluding group is unknown, a detection algorithm must identify that the colluding group is more suspicious than the honest-reviewer groups of different sizes. In this section, we evaluate whether several existing techniques for dense-subgraph discovery suffice to detect colluding groups with larger values of (k,7), 7), including one algorithm demonstrated to effectively detect fraud on Twitter.
We consider two desiderata in selecting algorithms to evaluate. First, the algorithms should not require explicitly specifying the size of the output subset. This is a requirement in our setting, since conference program chairs have no evidence regarding the size of the colluding groups that they can use to direct the detection algorithms. Second, our algorithms should be based on different notions of subgraph density that implicitly balance the size and edge density of the groups in a different way. Since 91 is unattributed, this choice is the primary design decision made by density-based detection algorithm. By considering a variety of such choices, we hope to find one that detects the true colluders across the largest possible range of injected size and density. As a result, we consider the following algorithms:
Traditional deusest-subgraph discovery Goldberg 1984 Charikar 2000): Output the subset of vertices S that maximizes f(S) = | This corresponds to the subgraph with highest average dogree. In our case, since we count all edges in both directions, this is a generalization of the standard definition for undirected graphs. To solve this problem, we implement the LP-based exact algorithm from Charikar 22000). We refer to this as "DSD".
Optimal quasi-clique discovery Tsourakakis et al. 2013): For a given parameter a E [0, 1], output the subset of vertices S that maximizes the "odge surplus" f(S) [,] - 2a (1 The second term in f(S) corresponds to the expected number of edges in the subgraph if each edge occurs independently with probability a. Since we count all edges in both directions, we include the factor
8

---

Published in Transactions on Machine Learning Research (12/2024)
of 2 in the second term as compared to the standard definition for undirected graphs To solve this problem, we implement the two approximation algorithms proposed by (Tsourakakis et al. (2013)
one based on greedy peeling and one based on local search. We refer to these algorithms as "OQC- Greedy" and "OQC-Local" respectively. As recommended by Tsourakakis et al. (2013) a = 1/33
we set
Tell'ail (Hooi et al. p020 This method first defines the "adjusted mass" of a subset as the difference between the number of edges in the subset and the expected number of edges in the subset if edges are rewired randomly (preserving vertex degroes). Subsets are then scored based on the probability of the adjusted mass under fitted Generalized Pareto distribution. Most parameters of this distribution are fixed as constants based on empirical observations across several datasets. Since this method operates on undirected graphs and is non-trivial to adapt to a directed setting, we first map our bidding graph G1 to an undirected graph g = (V1,)) before inputting it to this algorithm. The input graph 9 has the same vertex set as 91 and has an edge (r1, r2) iff both edges (r1,72) E & and (r2, r1) E E1: that is, = {(r1, r2) E R²; (r1,72) E & A r1) E & Denote the degree of verlex v in by deg(v) and the CDF of the Generalized Pareto distribution as FGP. Concretely, the algorithm finds a subset of vertices S S V that approximately maximizes the objective function
f(S) = FGP [[[S]] - Eves deg(v)
Note that this algorithm implicitly takes into account the connectivity between the chosen subset and the rest of the graph.
4|g]
DSD, OQC-Greody, and OQC-Local operate on pro-specified notions of density, without considering the sparsily of subgraphs in the input graph. In contrast, Tell defines density in a data-driven fashion by identifying the distribution of masses that subgraphs of certain sizes follow, arguing that other notions of density tend to be biased towards larger subgraphs. TellTail was also shown to detect fraudulent followers on Twitter. These algorithms cover a representative set of the landscape of dense-subgraph mining solutions.
We evaluate the detection algorithms against the following collusion model. For each setting of (k;,), we choose a subset of reviewers uniformly at random from among those reviewers that authored at least one paper. We then add edges uniformly random between colluding reviewers until the subgraph has edge density at least 7. This modified graph is then passed as input to each detection algorithm. We repeat this procedure for 50 trials for cach sctting. We report the mean Jaccard similarity between the set of injected colluders and the set of reviewers oulput by the detection a algorithms.
For the detection algorithms that use local search to optimize their objective (OQC-Local and TellTail), we return the result with highest objective value over 11 initializations. The first run is initialized according to the triangle-counting heuristic suggested by [Tsurakakiis el al. (2013), and the remaining 10 runs are initialized uniformly at random. We find that the detection performance of OQC-Local is significantly better when only the heuristic initialization is used, since the random initializations often resulted in output with higher objective value but lower overlap with the true colluders; thus, show the results with heuristic initialization only in this section and defer the results with all initializations to Appendix B However, the poor performance of OQC-Local when all initializations are used indicates that the objective value of OQC-Local is misaligned with the detection objective.
The results for DSD, OQC-Local, and Tell are shown in Figure 2 Results for OQC-Greedy are shown Appendix B as it generally performs worse than OQC-Local. In both datasets, DSD performs very poorly; this is because it consistently returns overly large subset of reviewers. On the AAMAS dataset, OQC-Local does somewhat better, achieving Jaccard similarity above 0.5 for values of k 2 26 and y 0.9. TellTail performs by far the best, detecting colluders consistently for moderate-to-large values of (k, 7). However, there exists a wide range of settings in which all algorithms fail to detect injected colluders: for example, reviewer groups with 8 < k < 12 and 7 = 1.0 do not exist in the honest bidding and yet are still not detected by any algorithm. On the S20RC datasct, OQC-Local and TellTail appear to perform equally well. There is a similar region in which honest-reviewer groups do not exist and yet detection fails to identify the correct group with high probability (e.g. 10 S k S 12 with 7 1.0), albeit smaller than in the AAMAS dataset.
9

---

Published in Transactions on Machine Learning Research (12/2024)
1.0
1:00-:01
2:00-:00 03
20- 00
10
08
1.0
00
00
00
00
10
edge de density
0.9
2.00
0:01
00
301
0.01
0.01 0:25
0 0.01
23 0.07
0:38
08
0.04
0.9
200
0:
28
0.00
0.31
010
0.01
33
01
0.01
084
0.04
01
06
02 100
0.8 0.5
0.0 0:53 2.2 0.05
0:23
0.09 0.09
0.07
0.09 0:
0.8
0.00
0.01
0.23
01
0.04
0.3 6.00
6.:00 03
-:000 00
20-
03
20- PO
--30
+ 20-30---
00
5-30-
00
000
00
0.7
00
00:00000000
09
00
00
09
00
- 02
00
A5
00
60 00
0.6
0.02 2:53 0.00 0.22 3 0.00
2:04
0.04
0/25
0.09
2-27
0.07
0.28
0.00
0.6
2.0
0.00
9.20
0.0L
0:23 0ILL
D.OL
23
0.03
24
9.
5/02
0.03 5.00
0.5 10.20-
56
1.50
28-
x -350- DO
9.58
26-2- ON
000
0.5
[-0
00
00
-O DE DO -2E ON
-22 DM
2
6 8 10 12 14 16 18 20 22 24 26 28 30 32 34
2
4
6
8
10
12 14 16 18 20 22 24 26 28 30 32 34
number of reviewers (k)
number of reviewers (k)
DSD, AAMAS
(b) DSD, S2ORC
edge density (y)
1.0 2:00=
03
0:02
00
302
0.03
3:55
0.02
DJ
DRS
05
0.52 22
1.0
00
0.L
00
2:00
6:33
922
0.0 08
1
2:00
00
200:
1.0 2.00
0.9 0.0 0o: 0.2
00000000000000000000000000000000
302
0.1
0.0
0.50 2:22 0.24
0.9
100 0.0
00
0:21 0.0
0:1
000
8 83
R5
1.0
1
0. 1.00
0.8 2.00
0:000 00
2:20
2.20
03
2:20
D1
2.31
02
5.24
2505
2.05
0.8
00000:00000000000
0.00
2:30
1:00
2:33 2.05
5.00
05
6.0 a
2.00
C2
0:
0.7 R.2 5:53 0.0 052 202 0.03
2:54
0.04
0:23
300
0.07
0:28
0.12
0.1
022 0.23 9:33
0.7
200
0.00 0:22 0.01
8.22 080 0.01 0.28
5.37
SUET
73 052
1:32 05? 5.57
0.6
8:00
5.61 nn 0.2 3/0 0.03
09
8:50=
2.
DO
D. 0.02
23
0.08
2.33
0.14011
edg
0.6
2:30
2.00
0.31
3. 00 2.00
0.01
6:33
0.5
0.09
10:0=:00.:0000000
e.0 0:2 302 0.03
2:02
0:04
0:33
3.09 0.05
0:07
0.28
0.08
0.30 0:15
0.5
2.00
0:00
0.50
0.09
0:33
08L
0.02
23
2.25
0:22 0.02 0.02 0.02 0:33 0:18
032 6:30
2 4
6 8 10 12 14 16 18 20 22 24 26 28 30 32 34
2
4
6
8
10 12 14 16 18 20 22 24 26 28 30 32 34
number of reviewers (k)
number of reviewers (k)
(c) OQC-Local, AAMAS
(d) OQC-Local, S2ORC
1.0 Moy 3:11 oot 0.2 302 0.0
UN 1:0 L0
1.0
Fan u. 28 0. 0:2 9:3
L0S
1.0
L10
1/0
00 1.00
edge density
0.9
2:00 00:0 2.000 10
= 200000000 2.:00:000 M1
20
02
5
100
5.5.0
0.9
2.00
2 2.01 2
02
01 =
2.
5.00
2:00
0.8
M1L 9:21 un 0:1
392
9:02
0.2
2:30
00
1.01
1.0
110
1:0
S
0.8
100 om
28 0.0
0:1
012
11
0:44
054
0-31
1.0
1/0
L02 1.00
0.7
8.00 0.:00
0.50 0.0...10000 0.1 301 0.02
a:50
0.03
x 00
0.53
3.03
0.03
25
0:53
6.33
x:34
6.75
100
0.67 0:95 02
..0
d
0.7
2.00 0.00
000
6.01 0.52
x 01
0.61
0.22
3.03
0: 6.24
6-72
24
32
0.
0:55
0:22
2.00
0.6
1 22:00::00 00 00
0.0
2:00
0.01
0.03
0.03
00
0.03
20
24
0.04
00
2.00
2.10
0.6
200
2.000000000
09
2.
0:1
5.00
0.01
00
0Z
0:23
5.02
5:05
10.4.05m.00
0.5
3.51 o. 0:2
3.02
0.02
53 0.03
0:53
3.00
0.04
33
0.05
0.24
0.05
0.04
9:
0.5
3.0
0.0
28
0.00
0.31 0.0
6.01
2-33
0L 0.21
0.0
6.02
0:05
0.05
5.14
0.11
5:0
2
6
10 12 14 16 20 22 24 26 30 32
4
10 14 16 18 20 24 26 28 30 34
number of reviewers (k)
number of reviewers (k)
(e) Telfaii, AAMAS
(f) TelfTail, S2ORC
Figure 2: Performance of detection algorithms on Gt. Values indicate the mean Jaccard similarity between the true set of colluders and the algorithm output, along with standard errors. Higher values correspond to better detection performance.
4.3 Manipulation Success Evaluation (Q3)
In this subsection, we evaluate the impact of collusion terms of the colluders' success at achieving their desired paper assignments, contextualizing the results of the previous two subsections. We conduct experi- ments in which we inject colluding groups into G, for each setting of the parameters (k,, 7) in the exact same manner as in Section
4.2 We then fix this modified version of G1 (i.e., the input to the detection algorithms in Section 4.2 as the "target graph" and modify B in order to realize this graph.
However, there are many possible strategies that these colluders could employ to modify their bids that correspond to the addition of these edges, since each edge (r1, r2) denotes the presence of at least one bid from reviewer r1 on a paper authored by reviewer r2. Additionally, the relationship between bids and edges is not one-to-one due to co-authorship between reviewers, which means that exactly realizing the target graph via bidding may not be possible. Due to these challenges, we instead modify the bids of colluders to achieve modified version of Gi that is "no more suspicious" than the target graph. Specifically, edge (21. in the target graph such that ri is in the injected colluding group. If r2 is also a colluder, we add bids from ri on each paper hothe by r2; otherwise, we choose one existing bid from ri on a paper authored by r2 uniformly at random and remove all other bids from 71 on papers authored by 72. In this way, we ensure that the edges within the injected colluder group are a subset of the edges in the target graph and the edges outside the injected colluder group are a superset of the edges in the target graph. Subject to these constraints, this procedure allows colluders to bid according to worst-case strategy.
consider each
10

---

Published in Transactions on Machine Learning Research (12/2024)
1.0
120
640
1.0
R12
9:23 037
edge density
+m) 2:
8:51
0.9
0:3 2:40m000 5.22
02
5:33
0.8 0.0033
0:2 318 0:2
3-26
327
0.
10
0:11
0:40 16
S
0.8
053 0.330.2--111 01
0:1
0.16
6:72
024
6:21
6.76 0-3
6
0.7
0..0 am 231
8:81
m1
01
0
0.7
8:31
0.01
--31
0.6
2.00000
012 318
3:21 3-3
0.27
24
0.6
855
0.333.24111 015 6:14
0. 0:17
0.14 1.
24
25 6.25
22
0.
0.30
0.5
2.52
E.. x0l
0.0 0c5 142
211 0.
0.17 2:20
0.24 5.27 9:28 0.29
2.23
0.23
0.55
0:37
0.49
0.5
0.23
2:22
9:28 1.0
8.22
2.01
9:29 0.37
2.
2-22
0.28
2.22
0.29
2.0. 2. B.D.
0.24
25 022
2.0.
6.27
8.03
2 6 10
16 18 20 22 24 26 28 30 32
4
10 12 14 16
20 22 24 26 28 30 34
number of reviewers (k)
number of reviewers (k)
(a) Fraction of target papers with a colluder assigned, AAMAS
(b) Fraction of target papers with a colluder assigned, S2ORC
edge density (x)
E 1.0
1.0
055
3:30
352
0.57
351
14 0. 0:55
3.17 2
025
0.30
0:39
0 0.22
5.32
0.61
0 ILEN F- M 1.5
0.9
022 231
151
0.53 3.57
0.50
2.01
5
0.64
6.56
2:2
0.9
123
632
0.34
0.0
42
544
1.50
0.8
025
2.22
3:47
92
02
0.38
90
2 0.8
0.09
5.02
32
0.35
5:22
IZ
5.01
5.01
(:45
5.01
0.7
5:5022
024 3.32
0.6. 5:43
0.49
54 54
0-18 3-38
6.53
6.62 54
0.7
6:e-9:1
316 022
0
6.29
0.35 0.36
6.
0:41
042
6:8:
0.46 0L65
5.45 0.55 5.47
am2 =.2
8: 02 F2 8 2222
01
2
2.71
edg
0.6
2:33
2
5.02 92
72
R-P:
=
2.00
2.01
0.5
m:04
022 3.2
0:34
0.56
0.66
0.5
R:5 9.1 a12 0.2
0.22 0.26
0.24
6.37
0:51 (L40 5:42
2
6 8 1012 14 16 18 20 22 24 26 28 30 32 34
2
4 6
8 10
14 16 18 20 22 24 26 28 30 32 34
number of reviewers (k)
number of reviewers (k)
(c) Fraction of colluders with colluder assigned,
(d) Fraction of colluders with a colluder assigned, S2ORC
AAMAS
Figure 3: Success of colluders in terms of k and 7. Values indicate the mean for each metric along with standard errors.
Given the modified bids, we compute a paper assignment as detailed in Section 3.1 Since the exact objective of colluders not obvious, we evaluate the success of the malicious reviewers in two ways. First, we count the fraction of "target" papers (i.c., those authored by at least one colluder) with at least one colluder assigned; this indicates the extent to which colluders succeeded at influencing the acceptance decision for all of their papers. This may be too harsh of a metric, since colluders may not aim to influence the decision for all of their papers simultaneously. Second, we count the fraction of colluders who authored at least one paper that has at least one other colluder assigned; this indicates the fraction of colluders who recieved some benefit from participating in the collusion ring. We conduct 50 trials for each setting of the parameters and report the mean for each of the two metrics.
The results are shown in Figure 3 For the results the AAMAS dataset, of particular interest are the results with & < 20 and 7 S 0.8, since these were not detected by any of the algorithms in Section 4.2 We see that at the extremes of this region, approximately one-third of colluder-authored papers have at least one colluder assigned, and approximately one-half of colluders have at least one colluder assigned to one of their papers. At (k- = 10,7 = 0.8), where a larger number of honest-reviewer groups exist, both success metrics are also reasonably high. The results on the S2ORC dataset are similar: for example, at (k 14, = 0.8), 22% of target papers and 35% of colluders are successfully targeted while all detection algorithms achieve low success rates. Similarly, 21% of papers and 34% of colluders are successfully targeted when colluders are camouflaged among numerous honest-reviewer groups at (k 12,y 0.9). Thus, colluders can influence the acceptance decisions for moderate fraction of their submitted papers without being detected by the evaluated a algorithms. The counts of honest-reviewer groups in Section 4.1 suggest that a portion of the success of colluders may be unavoidable in this setting, regardless of the strength of the detection algorithms. In practice, the implication of these results that any group of reviewers could docide to collude and unethcaally improve their chances of getting a paper accepted at the conference, tainting the fairness of the peer review process, while remaining completely safe from bidding- bassed detection.
11

---

Published in Transactions on Machine Learning Research (12/2024)
1.0
218
bid density (n)
bid density (n)
0.8
42
0.8
449
B07
1164
0.6
286
669
1301
1580
1382
5390
1.844
0.4
0.4
6.654
0.2
6670
2:4E5
9.4
1,088
0.2
3.484
1.866
2,487
3
5
3
number of reviewers (k)
number of reviewers (k)
(a) AAMAS
(b) S2ORC
Figure 4: Exact counts of honest-reviewer groups with varying size and bid density.
5 Bipartite Bidding Graph
In this section, we represent the reviewer bidding data as a bipartite graph, in which one set of vertices corresponds to reviewers and the other set of vertices corresponds to papers. This graph contains three types of undirocted edges between reviewer r and paper p: bid edges, indicating that reviewer bid positively on paper p; authorship edges, indicaling that reviewer authored paper μ, and conflict-of-nnteess edges, indicating that reviewer r has a conflict-on of interest with paper but did not author it. Our motivation for considering this graph is that it directly represents all data that the detection algorithms have access to.
.22 EE())), where V/2 = R RUP is the vertex set and e(()) B, E² = A, and e/5) = C \ A are the sets of bid, authorship, and conflict edges respectively. As in Section 4 we characterize (potentially colluding) group of reviewers S C in terms of size parameter ks |S| and density parameter yss- In G2, we consider new notion of density, which we call "bid density". For any subset of reviewers R' C R, define P[R']- UrER {p E P (r,p) E A} to be the subset of papers authored by at least one reviewer in R'. The bid density of S is then defined the total number of bids made by reviewers in S on papers in P[S], divided by the maximum possible number of bids by reviewers in S on papers in P[S]:
Formally, we denote this graph as G2 = (V2, (E) (B) c(A)
E IS
BS SUPP
1
S||P|S|| - E [S P[S]I - 1 [SUP
We again will omit the subscript S from ks and ns since it will be clear from context.
We now analyze the problem of detecting collusion from G2. As in Section the following three subsections present empirical analysis aimed at answering the three research questions identified in Section 3.3
5.1 Conest-Reviewer Groups (Q1)
First, we count the number of honest-reviewer groups in G2 for each value of size h and bid density n/- We consider only reviewers that have authored at least one paper. Since the bid density for a group is naturally lower than the edge density in general, we consider values of bid density ranging between 0.2 and 1.0.
The counts are shown in Figure 4 Due to the more complicated definition of bid density, we cannot efficiently prune branches of the scarch tree and must cnumerate all (IRI) revicwer subsets. As a result, the range of feasible k are significantly more limited than in Section 4.1 counts for k > 6 for AAMAS and 5 for see that numerous honest-reviewer groups do
S20RC could not be completed in 24 hours. In S2ORC, exist at every feasible setting, up to 1.0 and a = 4. Honest-reviewer groups are less prevalent in AAMAS, but still exist at A: = 5 and < 0.8. Like the figures in Section 4.1 these figures may be independently useful to conferences as indicators of the number of false positives for any heuristic defenses against. collusion they use.
As in Section we also run greedy peeling method to heuristically find high-density reviewer groups of larger sizes. However, this method performs very poorly and relegate the results to Appendix B
12

---

Published in Transactions on Machine Learning Research (12/2024)
bid density (n)
G
00.
03 -20
-.00-0 00
50
01
30
00
00
00
1
1.0
00
mm
00
0-00
06
00
0:
08 0:
100
60
00
0.8 0:3
0:1
1 0.00
3:23
02
0:2
350
1:00
100
1-88
100
1:00
35
0.8
0.0
3.30
0.00
0:28
3.00
0.00
0.36
00
&
0:11
010
6:1
01 0.24
08
3:00
0.6
58
28
00
0.6
80
08
0.00
08
0000
0.4
0:0 0:1 3.1
3.53
3.02
0.53 3-23
3.00 0:33
6,0 0:3
303
0.03 033
0.4
0.50
9.29
0.00
0.56
0101
0.00
0.35 0.00
6.00
35
010
6.1
0.01
02
0.0
0.01 5.01
0.2
2:0
0000000
8,00
8:00
3.22
2:50
0.22
8:00
8,00
0.23
0:02
0:33
2.00
0.00
2.00
0.00
bid
0.2
2.00
3.35 0.00
5.50
0:
2.
0:35
2.
B.DE
6.21
2.00
0.00
C/2
2.00
5.63
2
6
10 12 16
18
20 22 24
26 28 30
32 34
2
4
10 12 14 16 18 20 22 24 26 28 30 32 34
number of reviewers (k)
number of reviewers (k)
(a) OQC-Greedy, AAMAS
(b) OQC-Greedy, S2ORC
bid density (n)
20 a 2: 2-04-4
2:77 0:4
00
0.55
L.0 100
101
2.00 20
1.0
10
10
022
5:00
9-23 9 6A5
10
(0
2.00
0.8
E.2 0.0 8:1
3.2 n.
3:54
0:1
5:4
17
0.8
6.c0
3:53
0.0
0:00
ont 0.01
6:51
011
6.0)
0:51
000
6:3
0.03 0.0 6:15
0.6 200..00:000 2.00
2:30 00
2:00
2000
8.23
2.0
2.00
2.00
2.05
2:01
04
0.6
2.00
2.002:00.:00
2.00200000
2:30 1.00 2.00 2-32
2:05
3.
0.00
2 00
2.00000
0.4 2.2 03: 053 20: 0.00 2:53
202
0.44
2:56
3.00
0.2
0:57
0/0
0.20
032
0.14
8:7
0.4
0.00
2:22
0.00 0:28
DUL DUL
0.21
O 0.00
9-52
6:2
0
0.0
1/2
0 5.50
0.2 2:0 0.500.00 =.00
Eat n. 1/2 6. 5.53
0.000.00
2.50
0.53 9.53
=.00
5.50
0.24
0.00
2.00
0:05
2.00
0.55
0
2.00
0.07 6:38
0.2
2.00
0.000000000
3.5,
5.50
0.00
2.00
0.01
2.
0:31 9
2.25
2.05
2.05
6:01
2.00
2.00
2.00
6.2
2.00
5.00
2
6 10 122 14 16 18 20 22 24
26 28 30
32
6
10 12 14 16 20 22 24 26 28 30 32 34
number of reviewers (k)
number of reviewers (k)
(c) Fraudar, AAMAS
(d) Fraudar, S20RC
bid density (n)
1.0
EAt na ne 3nc 8:31
357 0:6
12 51 0
0.42 10L
1.24
1.0
Ince 2: n.m 6:50 om 8:01 0:34 0:0 8:2 0 92
6:24
1128
0:24
0.8
051
102 0.10 212
0:53
0.00
0.8
0.50
0.01
0.50
0:0
0.0L
0:21
0.00
0.34
0.22
0:23
5.22
31
5.30
000..00::00
2 0.
2:25
2.07
07
00
05
2:35 60 0.03
0.6
2.00
2.00
2.30
1.00
0.00
2-00
05
0-03
0.05
2.05
2.05
0.4
6:t 8m 181 nm 5:21
8:6
3.24
3
6 9:21
0:10
08
0.4
6.c
3:33
8.0
0:38
9 6.0
6-36
01
6.0)
0-31
6:51
0:03 0.0
C/1 0-01 5.67
0.2
500
2.00 00
5:00 5000 00 2.00 00 00 30 100 0.06
bid
0.2
30
am
20
0.00
05 0M 0.
000
0:
0:000
2 4
6 8 10 14 16 18 20 22 24 26 28 30 32 34
2
4
6 8 10 12 14 16 18 20 22 24 26 28 30 32 34
number of reviewers (k)
number of reviewers (k)
(e) OQC-Specialized, AAMAS
(f) OQC-Specialized, S2ORC
Figure 5: Performance of detection algorithms on G2. Values indicate the mean Jaccard similarity between the true set of colluders and the algorithm output, along with standard errors. Higher values correspond to better detection performance.
5.2 Detection Algorithm Evaluation (Q2)
Next, we evaluate the performance of a variety of denso-subgraph discovery algorithms at detecting injected collusion in G2, including an influential fraud-detection algorithm for the online review setting. Specifically, we consider the following algorithms:
DSD, OQC-Greedy, OQC-Local, and TellTail: These algorithms are introduced in Section 4.2 As input to each algorithm, we provide an unattributed bipartite graph g = (V2, E) with the same vertex set G2. We consider two variants for the edge set of the input graph: the edge set consists of only the bid edges E = BB, and the edge set consists of the union of the bid and authorship edges &- = eB gg(4), In the OQC algorithms, we use the original objective function for undirected graphs = [8[8] - (1/3) (!.))
Fraudar (Hooi et al. 2016): This algorithm is designed to detect fake product reviews (e.g., on Amazon Yelp) or fake followers (c.g., on Twitter) from a bipartite graph of users and products g (V, (V, where & contains edges from users to products. Denote by deg(v) the degree of a vertex in G. The returned subset of vertices S C V chosen to approximately maximize the objective function f(S) [s] Σ (u.))se((]] (log(5 + deg(v)))- -1 This objective is reassttt meaning that fraudulent users cannot affect their own objective value by adding extra edges to honest products. In our setting, we consider the reviewers to be users and papers to be products. We again input an unattributed bipartite graph with the same vertex set as G2 and two variants for the edge set:
E = (B) and & E1 U u((
13

---

Published in Transactions on Machine Learning Research (12/2024)
SpC--apcia We additionally adapt the objective function of the optimal quasi-clique discovery algorithm to our setting in a natural way. For a subset of reviewers S C R. we define the objective function
IS [SPP[S]] - - ( S(S) {
-12 1 (SPPISIi).
Analogous to the concept of "edge surplus" that motivates the optimal quasi-clique objective fune- tion, this corresponds to the number of excess bids above the expectation if each bid is made independently with probability a. We use local search to optimize this objective. As in the other OQC algorithms, we set a = 1/3.
Since we aim to detect group of colluding reviewers, we discard any paper vertices in the subset output by each algorithm, considering only the output reviewer vertices as the detected reviewers.
In addition to the algorithms from Section 4.2 Fraudar is an influential fraud-detoction algorithm that leverages a density-based signal for detection. It was designed to detect fraudulent reviews on platforms like Amazon, a similar problem to our own. OQC-Specialized was not proposed in prior work, but naturally generalizes the objective function of OQC-Local to operate directly on G2.
We now describe the collusion model against which we evaluate the detection algorithms. For each value of n), we inject a group of colluding reviewers into G2 as follows. We first choose a set of h reviewers, denoted by M, uniformly at random from among all reviewers who have authored at least one paper. We then choose reviewer- paper pair uniformly at random from M P[M] and add this pair to E if there is no existing edge of any type between this pair. We repeat this until the bid density is at least n. This modified graph is then used to construct the input to each detection algorithm. We repeat this procedure for 50 trials for each setting of parameters. We report the mean Jaccard similarity between the set of injected colluders and the output set of reviewers from the detection algorithms.
As in Section 4.2. for each local search algorithm (OQC-Local, TelITail, OQC-Specializod), we return the result with highest objective value over 11 initializations. The first initial subset is chosen according to generalization of the heuristic suggested by Tsourakaiis et al. (2013). For each vertex, count the number of hid-author-bid-autho cycles that this vertex participates in and divide by the total bid- and authorship- degree of the vertex; we then choose the initial subset to be the maximum vertex according to this metric along with all of its neighbors. The remaining 10 runs are initialized uniformly at random.
We show results for the OQC-Greedy, Fraudar, and OQC-Specialized algorithms in Figure 5 Results for the remaining algorithms, which generally performed worse, are in Appendix B For all algorithms (other than SQC-Specialized), we find that the performance is very similar when the input edge set is E and when it is (5 u =) thus, all results shown are for the case with edge sct E cBB) On AAMAS, we see that both OQC-Greed, and Fraudar are perform very well for high values of (k, (e.g., k > 20, 2 0.8). However, both of these algorithms fail to detect colluders entirely at more moderate values of l (e.g., k= 14, 1.0). OQC-Specialized achieves some success for much wider range of parameters, but fails to achieve Jaccard similarities above 0.9 even for extreme values of (k; this may indicate that the true colluding group is not a local optimum for this algorithm's objective function. On S2ORC, performance of all algorithms is significantly worse. OQC-Greedy and Fraudar still identify the colluding groups for extreme values of (k, n)) but OQC-Specialized fails to consistently identify the colluders for any parameter values. Overall, there exists significant region of moderate parameter values al which no algorithm achieves good detection performance.
5.3 Manipulation Success Evaluation (Q3)
Finally, we evaluate the success of the colluders at manipulating the assignment as a function of the parame- ters (k, 7). For each setting of these parameters, we inject a group of colluders as described in the preceding
section and compute paper assignment using the procedure detailed in Section 3.1 As in Section 4.3. we
14

---

Published in Transactions on Machine Learning Research (12/2024)
bid density (n)
S
-21
1.5
01
0I
01
01
m
1.5)
1:51
0
1.0
m1
31
3.3
1
1.0
0.8
0.000.000
0:4
115
8:1
3:23
124
000
3:
0.37
0.2
6:10
9:2
6:4 8-16
0.8
029
3:27
612
0:4
316
0.18
320
621
22
1.20 627
838
L:3
0:20
0.6
8:31
01
bid den
0.6
8:31
2-
2:31
0.4
0.0000
0.6
311
3.6
0.1 3424
6.36
0.24
0:32
0.25
6:36
0.4
0.54
0.06
0.58
0
6.11
0.33
9.34
6.17
13
19
9.22
0:23 5.22
0.2
5.22
102
0.
8:10
3:54
0:44 9:27
1. 6:6
0.22
0.26
6:38
0.2
201
6.04
6.56
2.00
0.67
0:38
9.39
0:23
6.3
6.15
8.01
2 6 8 10 12 16
18
20 22
24
26
28
30
32 34
2
4
6
10
12 14 16
18
20 22
24 26 28 30
34
number of reviewers (k)
number of reviewers (k)
(a) Fraction of Largel papers with a colluder assigned, AAMAS
(b) Fraction of largel papers with a colluder assigned, S2ORC
S 1.0
1.0
Eas 0:23 A11
0:0
SA1
8.22
3.50
8
55
2.22
0.C0
0.G.
0.65
067 0
22
3
6:2
624
0.
028
0:31
2
2.22
A
2-02
5:4T
CAT
bid density
1...0000 8:39
0:1
2-46
00
00
0:5
01
033
81
D 0.8
6:53
100
32
0.23
9:31
03
01
542
01
C42
0.6
1.1 027
333
0.W. 3-45
3/4
0.18
3:25
0.5A
0.53
0.6
3:34
011
822
321
0.24
0.37
0:30
0
3.36
6.3
0.38 0
1.75 0-0 (.4)
0.4
2.5
0.522 2:12
2
02
02
0L
2.72
01
0.4
2022022
2.00
2-
02
2.03
0.
8-0.
2
2.00
8.012.0.
0.2
200011 0.5 218
0.2 2:21
2.51
0.5
2.23 3.40
B.DD
4L
9.4
044
0.40
0:2
bid
0.2
053
2:24
wer
0.8
030
0.14
031
0.38
0-
21
9-21
0.22 022
1-27
0.28 5:27
2 6 10
16 18 20 22 26 28 30 32 34
4 6
10 12 14 16 18 20 22 24 26 28 32
number of reviewers (k)
number of reviewers (k)
(c) Fraction of colluders with colluder assigned, AAMAS
(d) Fraction of colluders with a colluder assigned, S2ORC
Figure 6: Success of colluders in terms of & and n- Values indicate the mean for each metric along with standard errors.
evaluate the success of the colluders in two ways: the fraction of colluder-authored papers with at least one colluder assigned, and the fraction of colluders with at least one colluder assigned to one of their authored papers. We conduct 50 trials for cach sctting of the parameters and report the mean for each of the two metrics.
The results are shown in Figure 6 On AAMAS, we see that at (k- = 16, 17= 0.8) where no detection algorithm was able to detect colluders with high probability, the colluders can successfully achieve assignments to 30% of their target papers and 54% of the colluders. Similar results are seen elsewhere on the frontier of the undetected region. On S2ORC, although the detection algorithms performed poorly, the success values are also lower than in the corresponding case on AAMAS. Still, in the cell (k: 26, 17 = 0.8) where the detection algorithms performed poorly, colluders can achieve assignment to 26% of target papers and 42% of colluders, sizeable influence on the paper assignment. We note that these success rates are quite similar to those found in Section 4.3 for the unipartite graph setting, which may provide some indication that the ability of undetected colluders to manipulate the assignment is robust to the exact graph representation. Overall, colluders are still able exert influence over the acceptance decisions for a non-trivial fraction of their own papers while avoiding detection.
6 Discussion
We provide an empirical exploration of the problem of detecting reviewer-author collusion rings from manip- ulated bidding, framing the problem as a subgeapu discovery problem in two different graph represen- tations. Overall, we find that within our parametrized model of colluder behavior, malicious reviewers can manipulate the paper assignment to moderate success while remaining within typical or difficult-toddetett levels of density in the bidding graph (using popular existing anomaly-detection algorithms). This provides evidence to support the conclusion that malicious reviewer behavior cannot be effectively detected using just the bidding data. While our analysis cannot conclusively prove that bidding data is insufficient to detect collusion, our methodology thoroughly explores this problem from several different analyses of realistic con- ference data. Our work considers only a specific parametrized model of collusive behavior in which colluders receive reciprocal benefits within the conference in question. However, our results indicate that even this behavior cannot be detected. In reality, colluders may be more sophisticated in their attempts to avoid detection, making detection more challenging.
15

---

Published in Transactions on Machine Learning Research (12/2024)
One limitation of our work is that our analysis is performed on semi-synthetic datasets, since real datasets containing bidding and authorship data are not publicly available. Thus, it is possible that the bidding in our datasets is not fully representative of real conference bidding: for example, in the AAMAS dataset, honest reviewers with abnormal bidding patterns may have opted-out of data collection. To further verify our results, program chairs could replicate our methodology on their own conference's real bidding data. Another limitation of our work that we consider an intentionally narrow rescarch question: the feasibility of detection from just the bidding, without the use of other metadata. As one of the first works on detecting collusion, our analysis of this question provides a basis for future work to build on.
While our results do not immediately direct any specific detection algorithm to be deployed in practice, caution should be taken to avoid negative societal impacts from the misuse of any detection algorithms deployed in the future. The cost of misidentifying an honest reviewer as part of a collusion ring can have enormous consequences for their reputation and career. To prevent such an instance, conference program chairs should at least conduct an associated thorough manual investigation before pursuing any form of action against suspected collusion ring.
An important contribution of our work is to provide direction for future work. Most clearly, future work can consider the problem of detection in richer-featured settings than just the binary bidding setting we consider here and demonstrate if detection is feasible in such settings. Some examples of additional features that may be helpful are the strengths of bids, text-similarity scores, and past co-authorships. In these settings, future work can evaluate other types of anomaly detection methods beyond the dense-subgraph discovery algorithms we consider. This analysis would be very helpful for further directing the scope of future research malicious bidding-if such research finds negative results even with additional metadata, research ought to continue to focus mitigation-basse line of research is the lack of availability of high- fully-featured conference data. Future work can help address this need for datasets by collecting and releasing paper bidding data; one approach is to conduct an experiment from which (labelod) data can be collected, as donc by Jeemen et al. (2023) for collusion and by Stelmakh et al. (2021) for a different problem of malicious behavior peer review. Our work also provides some direction for research on collusion mitigation by establishing the region of colluding behavior that can be casily detected. Mitigation efforts need not be concerned with collusion in this region and can instcad focus mitigation within the undetectable region, particularly in the area where honest-reviewer groups do not exist.
techniques to address collusion rings. One major challenge for this
Acknowledgments
This work was supported in part by ONR 4000142212181, NSF CAREER Award 1942124, NSF IIS 2200410, and NSF IIS 2310482.
References
Leman Akoglu, Rishi Chandy, and Christos Faloutsos. Opinion fraud detection online reviews by network effects. Proceedings of the International AAAI Conference on Web and Social Media, 2013.
Leman Akoglu, Hanghang Tong, and Danai Koutra. Graph based anomaly detection and description: A survey. Data Mining and Knowledge Discovery, 2963)6626-688, 2015.
Noga Alon, Felix Fischer, Ariel Procaccia, and Moshe Tennenholtz. Sum of us: Strategyproof selection from the selectors. In Conference on Theoretical Aspects of Rationality and Knowledge, pp. 101-110. ACM, 2011.
Waleed Ammar, Dirk Groeneveld, Chandra ula,avatula, In Beltagy, Miles Crawford, Doug Downey, Jason Dunkelberger, Ahmed Elgohary, Sergey Feldman, Vu Ha, al. Construction of the literature graph in Semantic Scholar. arXiu preprint arXiv:1805 180.0022,,, 2018.
Stefano Balietti, Robert L. Goldstone, and Dirk Helbing. Peer review and competition in the art exhibition game. Proceedings of the National Academy of Sciences, 113(30):8414-88) 8419, 2016.
16

---

Published in Transactions on Machine Learning Research (12/2024)
Niclas Bochmer, Robert Bredereck, and André Nichterlein. Combating collusion rings is hard but possible. In Proceedings of the AAAT Conference on Artificial Intelligence, volume 36, pp. 4843 0843-4850, 2022.
Moses Charikar. Greedy approximation algorithms for finding dense components in a graph. In International Workshop on Approzimation Algorithms for Combinatorial Optimization, pp. 84-95. Springer, 2000.
Laurent Charlin and Richard S. Zemel. The Toronto Paper Matching System: An automated paper-reviewer assignment system. In 1CML Workshop on Peer Reviewing and Publishing Models, 2013.
Komal Dhull, Steven Jecmen, Pravesh Kothari, and Nihar B. Shah. The price of strategyproofing peer assessment. In The 9th AAAI Conference on Human Computation and Crowdsourcing, volume 2, 2022.
Thorsten Eisenhofer, Erwin Quiring, Jonas Möller, Doreen Riepel, Thorsten Holz, and Konrad Rieck. No more reviewer #2: Subverting automatic paper-reviewer assignment using adversarial learning. arXiv preprinl ar Xiv:23303 14443, 2023.
Dhivya Eswaran, Stephan Günnemann, Christos Faloutsos, Disha Makhija, and M. Kumar. ZooBP: Belief propagation for heterogeneous networks. Proc. VLDB Endow., 10:625 636, 2017.
Andrew V. Goldberg. Finding maximum density subgraph. University of California, Berkeley EECS Department Technical Report, 1984.
Longhua Guo, Jie Wu, Wei Chang, Jun Wu, and Jianhua Li. K-loop free assignment in conference review systems. In 2018 International Conference on Computing, Networking and Commumications (ICNC), pp. 547. IEEE, 2018.
Bryan Hooi, Hyun Ah Song, Alex Beutel, Neil Shah, Kijung Shin, and Christos Faloutsos. Fraudar: Bounding graph fraud in the face of camouftage. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 995-904, 2016.
Bryan Hooi, Kijung Shin, Hemank Lamba, and Christos Faloutsos. TellTail: Fast scoring and detection of dense subgraphs. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 4150-4157, 2020.
Steven Jecmen, Hanrui Zhang, Ryan Liu, Nihar Shah, Vincent Conitzer, and Fei Fang. Mitigating manip- ulation in peer review via randomized reviewer assignments. Advances Neural Information Processing Systems, 33:12533 12545, 2020.
Steven Jecmen, Nihar B Shah, Fei Fang, and Vincent Conitzer. Tradeoffs in preventing manipulation in paper bidding for reviewer assignment. arXiv preprint orXiw:2207.11315, 2022.
Steven Jecmen, Minji Yoon, Vincent Conitzer, Nihar B Shah, and Fei Fang. A dataset on malicious paper bidding in peer review. In Proceedings of the ACM Web Conference 2028, pp. 3816-3826, 2023.
Srijan Kumar, Bryan Hooi, Disha Makhija, Mohit Kumar, Christos Faloutsos, and V. S. Subrahmanian. Rev2: Fraudulent user prediction in rating platforms. Proceedings of the Eleventh ACM International Conference Web Search and Data Mining, 2018.
Kevin Leyton-Brown, Mausam, Yatin Nandwani, Hedayat Zarkoob, Chris Cameron, Neil Newman, and Dinesh Raghu. Matching papers and reviewers large conferences. arXio preprint orXiv:2202. 12279,, 2022.
Michael L. Littman. Collusion rings threaten the integrity of computer science research. Communications of the ACM, 64(6):43-44) 44, 2021.
Ian Markwood, Dakun Shen, Yao Liu, and Zhuo Lu. Mirage: Content masking attack against information- based online services. In 26th USENIX Security Symposium, pp. 833-847, 2017.
Nicholas Mattei and Toby Walsh. PrefLib: A library for preferences http://www, preflib. org. In Algorithmic Decision Theory: Third International Conference, pp. 259 270. Springer, 2013.
17

---

Published in Transactions on Machine Learning Research (12/2024)
Shashank Pandit, Duen Horng Chau, Samuel Wang, and Christos Faloutsos. NetProbe: A fast and scalable system for frand detection in online auction networks. In Proceedings of the 16th International Conference on World Wide Web, pp. 201 210, 2007.
B Aditya Prakash, Ashwin Sridharan, Mukund Seshadri, Sridhar Machiraju, and Christos Faloutsos. Eigen- Spokes: Surprising patterns and scalable community chipping in large graphs. In Pacific-Asia Conference on Knowledge Discovery and Dala Mining, pp. 435 448. Springer, 2010.
Nihar Shah, Behzad Tabibian, Krikamol Muandet, Isabelle Guyon, and Ulrike Von Luxburg, Desigu and analysis of the NIPS 2016 review process. Journal of Machine Learning Research, 19:1-34 34, 2018.
Nihar B Shah. Challenges, experiments, and computational solutions in peer review. Communications of the ACM, 65(6):76-8) 87, 2022.
Ivan Stelmakh, Nihar B. Shah, and Aarti Singh. Catch me if I can: Detecting strategic behaviour in peer assessment. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pp. 4794 4802, 2021.
Ivan Stelmakh, John Wicting, Graham Ncubig, and Nihar B Shah. A gold standard dataset for the reviewer assignment problem. Xivi preprinl arXiv:2303.1677 2023.
Dat Tran and Chetan Jaiswal. PDFPhantom: Exploiting PDF attacks against academic conferences' paper submission process with counterattack. In 2019 IEEE 10th Annual Ubiquitous Computing, Electronics & Mobile Communication Conference (UEMCON), pp. 0736-0743. IEEE, 2019.
Charalampos Tsourakakis, Francesco Bonchi, Aristides Gionis, Francesco Gullo, and Maria Tsiarli. Denser than the densest subgraph: Extracting optimal quasi-cliques with quality guarantees. In Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 104-112, 2013.
T. N. Vijaykumar. Potential organized fraud in ACM/IEEE computer architecture con- ferences.
http:///t/tiiiiaa////taaaaaaaaaa - fraa--f--aad-------- ie--iin-----eee -omputer architecture www.prrreonoooeoooooooooooeee ccd611693704
cooferroncee--cdd11117770 2020. Accessed Jamuary 15, 2024.
Paul Wilson. Academic fraud: Solving the crisis in modern academia. Exchanges: The Interdisciplinary Research Journal, 7(3):14 14114-44422 2020.
Ruihan Wu, Chuan Guo, Felix Wu, Rahul Kidambi, Laurens Van Der Maaten, and Kilian Weinberger. Making paper reviewing robust to bid manipulation attacks. In International Conference on Machine Learning, pp. 11240-11250. PMLR, 2021.
Yichong Xu, Han Zhao, Xiaofei Shi, and Nihar B. Shah. On strategyproof conference peer review. Proceedings of the Twenty- Figgtt-Eighth International Joint Conference on Artificial Intelligence, pp. 616 622, 2019. doi: 10.24963/IJCA1.2000 2019//87. URL https:://of: 1tt///..0/////////eee...
A Text Similarity Details
In this section, we provide more details about the text-similarity scores our datasets, introduced Scction 3.2 We synthotically generated the AAMAS text-similarity scores to supplement the original bidding dataset. The S2ORC text-similarity scores were included in the original dataset by Wu al. (2021), ; and were computed by running the widely-used TPMS algorithm the abstracts of the reviewer's 's past papers and the paper in question. Figure 7 shows the CDFs of the text-similarity scores among the reviewer-paper pairs that did not have conflicts-of-intrees..
Since our work analyzes the effect of bids on the paper assignment, it important to validate that our text similarity scores have realistic level of agreement with the bids. We do this by comparing to the results of Stelmakh et al. (2023), who evaluate the accuracy of commonly-used text- similiimilarity algorithms predicting reviewer expertise using a ground-truth dataset. Specifically, (Stelmakh et al. (0233) evaluate the accuracy of
18

---

Published in Transactions on Machine Learning Research (12/2024)
the text-similarity scores by considering peviewer-paper-Peee triples (r, (r.p1,P2) E X P P where reviewer reported greater expertise paper than paper P2. They find that the TPMS algorithm produces a weakly greater text-similarity score for (r, than for (r, on 80% of "easy" triples (where the reviewer reported high-cxpertise vs low-cxpertise) and on 62% of "hard" triples (where the reviewer reported high- expertise vs bigher-expertise). We use these results generale and/or validate the text-similarity scores our datasets, considering the bids be a proxy for reviewer expertise.
For the AAMAS dataset, text-similarity scores sampled i..d. from of three Gaussian distributions, depending on the bid value for that reviewer-paper pair (from {"Yes", "Maybe", "No response")) For the "No response" pairs, the Gaussian distribution was fit to the dataset of text-similarities reconstructed from the 2018 International Conference on Learning Representations by Xu et al. (201)). The variance of the Gaussian distributions for the "Maybe" and "Yes" pairs were the same, but the means were chosen based on the statistics from Stelmakh et al. (2023). Specifically, we set the means such that expectation: (a) 80% of the triples (r, (r.P1.P2) E R x P where bid "Yes" or "Maybe" on pt and r bid "No response" on p2 had T(r,p1) ) 2 Pr); and (b) 62% of the triples (r,p1,p2) E R x P where bid "Yes" on p and bid "Maybe" P2 had [(r,p) P2),
For the S2ORC dataset, we compare the text-similarity scores in the dataset to the statistics from Stelmakh ett al. (2023). Recall that the S2ORC dataset contains bid values for each reviewer-paper pair in (0, 2, We find that 83% of the triples (r,PI,P2) P2) E R x P x P where r bid from {1, 2, 3) on p1 and r bid on p2 had Ti) 2 T(r,22) Additionally, we find that 65% of the triples pp) E R x P x where r bid 3 on P1 and r bid {1, 2} on P2 had T(r 2 T(r,p2) We see that the accuracy of the similariilassiis in the S20RC dataset is similar to that of the ground-truth dataset.
Omitted Experimental Results
In Figure we provide evaluations of detection algorithms on G1 omitted from Section 4.2 This includes the algorithm, as well as the OQC-Local algorithm with all initializations (detailed in Section 4.2
These algorithms generally perform worse than those with results shown in Section 4.2
In Figure we show the size and bid density of heveserreviewer groups detected by a greedy peeling method on G2 omitted from Section 5.1 In this method, we begin with the entire set of reviewers. At each iteration, groedily remove the reviewer whose removal results in the highest bid density within the set of remaining reviewers. We then plot the bid density for each group size across the ilerations. Clearly, this method performs very poorly at finding groups of high density.
In Figure 10 we present the results of the detection algorithms on G2 omitted from Section 5.2
DSD, OQC-
Local, and Tall These algorithms generally perform worse than those with results shown Section 5.2
1.0
1.0
0.8
0.6
0.6
0.4
0.4
0.0
0.0
0.00
0.02
0.04
0.06
0.08 0.10
0.12
0.0
0.2
0.4
0.6
text-similarity score
text-similarity score
(a) AAMAS
(b) S20RC
Figure 7: CDF of the text-similarity scores for reviewer-pape pairs without conflict-of-nntrest.. The mean text similarity score among these pairs is 0.030 for AAMAS and 0.52 for S2ORC.
19

---

Published in Transactions on Machine Learning Research (12/2024)
edge density (y)
1.0
0:00-:01 10:0-:00 00
2.00-
30- 10
30
31
1.0
00
00
0.9
2.00
0:01 0:1
0.01
0/03
0:54
0.0
0:11 0:1
0.9
200 0:
9 28 0.01
0.31
010
31
08
02
00
102
0.8 0.5
0.51 202 0.03
0.
0.54
0.09
0.8
0:23
01
022
1.:00 03
x:50- 09 m-20
20-
00
2-20
20-
00
00
0.07 0:27 00
0.7
00
00:00000000
09
00 00
02
09
00
01
-3
-0-00000
0.6
2:53 0.00
0.22 3 0.02
2:25
0.05
0/23
2.24
0.I0
0.28
0
0.6
2.0
0.00
9.20
0.00
9:23 0IL
D.OL
23
9
D.OL
1:43
2.00
edo
0.5 10.20
56
D0
-000
0.5
1-0
00
DO -22 ON
-22
2
6 8 10 12 14 16 20 22 26 28 30 32 34
4
6 8 10 12 14 16 18 20 22 24 26 28 30
32 34
number of reviewers (k)
number of reviewers (k)
(a) OQC-Greedy, AAMAS
(b) OQC-Greedy, S2ORC
edge density (y)
1.0 1:00= 03
0:02 302 0.03 0.04
00
01
00
0.05 0:25
0.00
0:2
00.
0.53 0:27
1.0 00
0.00 6:33 O8L 0.27
00
043 0:3
00
00
200 1.0 2.00
(0
0.9 0.5 no
:0:000000
0.2
302 0.03
0.1
0.54
305
0.0
0.0
0.33
0.9
10 0.0
20
0.0
0:11
00L
0.03
33
0.15 64
033
00
02
1
16 1.00
0.8
2.00 03 8:00 00
2:20
2.20
03
2:20 00
2.30
00
8:30
1:000:00:..0000
0.8
0:
2:30
1.00
2:00
2.03
8.06
2 07
2.00
62 0:
0.7
R.1 2:53 0.02 0:2
392
0.02
3:53
0.04
0.54 3.0
0.08
0:25
0.05
0.26
0.00 0.00 0:28
0.7
200
9:21 0.00
0.21 08L 0.01 0.22 OL
9:23
011
0.25 0.23
0
052 5,54
0.6
hea 5.61 nn 0:2
3/ 0.03
8:50
0.63
8.50
DO
20
0,05
0.06
edg d
0.6
00
0.31
2.00
0:01
0.31
0.02-2
3.05
6:79
0.5
0.1
00:00:00
e:2 0:2 302 0.02
0:04 0.54
0.04
25
0.09 0.28
0.08 0:27
0.5
2.0
0:00
2.30
0:33
08L
0.01 23
0:22 0.0: 0.01
02 0.04 0:23
0.
2
4
6
8 10 12 14 16 18 20 22 24 26 28 30 32 34
4
6
8 10 12 14 16 18 20 22 24 26 28 30 32 34
number of reviewers (k)
number of reviewers (k)
(c) OQC-Local (including random initializations),
(d) OQC-Local (including random initializations). S2ORC
AAMAS
Figure 8: Performance of additional detection algorithms on Gi. Values indicate the mean Jaccard similarity between the true set of colluders and the algorithm output, along with standard errors. Higher values correspond to better detection performance.
1.0
S 1.0
bid density
0.8
0.6
bid densi
0.6
0.4
0.4
0.2
0.0
0.0
2
6
10
12
14
16
22 32
4
8
12 18 20
22 24 26 28 30 34
(a) AAMAS
(b) S2ORC
Figure Size and bid density of honest-reviewer groups found by a heuristic method on G2. Each point corresponds to an existing honest-reviewer group (found by greedy peeling method), and the shaded area indicates the region in which there exists at least one honest-reviewer group.
C Density of Detected Groups
One question relevant to the analyses in Sections 4.2 and 5.2 is why the detection algorithms fail to detect the true colluders. Specifically, one may be interested in distinguishing the scenarios in which the detection algorithms output a group of reviewers less-dense than the true colluders (indicating that the colluders could potentially be detected if a better algorithm was developed) from the scenarios in which the detection algo- rithms output a group of reviewers with higher-density (indicating that will be difficult for any algorithm to detect the colluders). In the following plots, we show the mcan (edge or bid) density of the groups output by the detection algorithms when colluding groups of varying size and density are injected. Note that the detection algorithms are not restricted to outputting groups of the same size as the injected colluders.
The results for all algorithms are shown in Figure 11 (unipartite setting) and Figure 12 (bipartite sctting). Overall, we generally see that in the (lower-left) ) region of the plots in which the algorithms fail to detect the true colluders (according to Figures 2 and 5 the algorithms are instead detecting reviewer groups with low density. This indicates that their objective may not be aligned well with the profile of the true colluders in this region. However, in the unipartite setting for the S20RC dataset, Tell detects highly-denss group of honest reviewers when the colluding group is small and less-dense. This provides some additional
20

---

Published in Transactions on Machine Learning Research (12/2024)
bid density (n)
G
00.
01
20
00
20
AM
0
00
0
10
3 1
1.0
00
mm
00
30-
0:35
00
00
10
60
00
0.8 501 0.000 0:3
1
0.00
3:23
04
0:0
9-15
17
6.70
C48
0.69
0.8
000
3.30
0.00
0:28
3.00
0.00
0.31
011
&.0)
0:11
010
6:3
0.31 0.1
1.00
0.6
26
10
0.6
80
0000
08
0.00
0.4
0:00000 0:1 3.1
5:53
3.93 0:53 3:54
9.04 0.56
35
0
0:4
9.2
0.23 6:24
0.4
0.50
9.29
0.00
0.58
0101
0.00
0.35 0.
0.00
0-
0101
6.1
0.03
02
0.2
0.01 5.01
0.2
E:0
00
102
8.00
6:00
3.22
2:50
0.53
8:00
8.00
0.54
25
0:05
0.25
0.05
0.06
bid
0.2
2.00
3.
5.50
0.00
0:
0:35
2.05
B.DE
91
6.01
2.00
2.000
2.00
C/2
2.00
5.63
2
6
10 12 14 16 18 20 22
24
26
28 30
32 34
2
4
6
10 12 14 16 18 20 22 24 26 28 30 32 34
number of reviewers (k)
number of reviewers (k)
(a) DSD, AAMAS
(b) DSD, S2ORC
bid density (n)
10
0.0 0:1
16
035
22
0.0
001
DEL
11
1.0
10
10
0:2
5:33
03
5.34
5.05
0.32
0.55
06
0.8
E.2 0.00000000 85
3.2
pm
3:57
315
036
3:00
9
6.
0.8
0.00
3:3
0.03
0:00
00
0.01
6:36 011
0.0)
0:41
0:01
6:33
0.01
0.66
0:13 1.
0.6
2000:002:00
2.00
2.000
00
2:30
2.00
02
8.23
200
2:00
2:24
2:24
2.04
2.24
2:34 05
0.6
2.00
2.000000
2:30
1.00
2.00
2:32
2:05
3.05
0.00
2
0.00
2.00
2.00
2.00
0.4 5.5 0.22 053 20: 0.00 2:52
282
052
2:23
0:23
0:52
0.6
0.21
9
0.05 8:3
0.4
0.50
2:32
0.00
0:28
000 0.00
0.20
O 0.00
9-55
0-01
0:0: 0.01
140
0.0 5.51
0.2 2:0 2.500.00 =050
5.s n. s.t
0.00=.00
5.22
2:50
0.22
2.00 =000
9.51
5.50 0.00
0.22
0:00
0:33
2.00
5.05
am 6.0
2.00
0.2
2.00
a
8.30
0.00
2.00
2.05
0:31
2.25
9
2.05
2.05
6:01
2.00
2.00
5.00
2
6 10
14 16 18 20 22 24 26 28 30 32
4
6
10 12 14 16 20 22
24 26 28 30 32 34
number of reviewers (k)
number of reviewers (k)
(c) OQC-Local, AAMAS
(d) OQC-Local, S2ORC
bid density (n)
1.0
E:s n:t 3/2 0:0
0:00000::00
1:22 382 6:3
BE 1:
21
1:0
1:5 110 10
10
1.0
Iece 29 n.w 6:50 o 6.01 0:28 911 6.0) 01 02 6:31
0.05 1L10
5.15
31 5.17
0.8
5:2
0.1 101
1::0=
0.00
2:52
0.53
102
0.52
0:38
1.00
2:5
0.8
0.50
0.09
0.50
0:
0.00
0:20 01
0:
0:22
0.02
1:24
9:50 5.37
110.:00::00
00:00:0000000 20
2.50
01
2.00
1.00
00
1:30
1000
00
2:30
2.00
2.33
2:07
60
0.6
2.00
2.00
2.30
1.00
0.00
2-30
D5 S.
5-0
2
1.00
2.00
1.00
0.4
Ent 8m 1n1 0.0
5:22
0:5
53
am
0:33
0:33
0.00
0.54
0.04 6:38
0.4
3:33
8.0
0:38
90 6.01
0:51
OL
0-31
6:01
0:51 0.
5.1 0.0
0.2
20
2.00 00
6000 000
00
20
600
bid
0.2
30
am
20
5.00
05 0M 01
(0)
0:000
2
4
6 8 12 14 16 18 20 22 24 26 28 30 32 34
2
4
6 8 10 12 14 16 18 20 22 24 26 28 30 32 34
number of reviewers (k)
number of reviewers (k)
(e) Tellaail, AAMAS
(1) TeliTail, S2ORC
Figure 10: Performance of additional detection algorithms on G2. Values indicate the mean Jaccard similarity between the true set of colluders and the algorithm output, along with standard errors. Higher values correspond to better detection performance.
evidence that colluder detection in the region of (k < 14, S 0.8) is not feasible, due to the presence of dighly-dense group of honest reviewers that can mislead the detection algorithms.
D Experimental Results with Variant Datasets
As we detail in Section 3.2 the datasets we analyze in this work are semi-synthetic. The AAMAS dataset does not contain authorships, and so we construct authorships by subsampling three confficts-of-interse for each paper. The S20RC dataset contains bids that were synthetically constructed by Wu et al. (2021) based on bidding statistics from NeurlPS 2016. In this section, we vary these synthetic aspects and conduct additional experiments on these variant datasets.
In the variant AAMAS datasel, we construct authorships as follows. For each paper, instead of subsampling three confficts-of-interestt uniformly at random, we choose an integer from {.. uniformly at random and subsample that many confficts-of-intrrsst to use the authorships. In the variant S2ORC datasct, we modify the original bids by removing 15,000 positive bids (over 10% of the positive bids) uniformly random and adding positive bids among 15,000 non-bidding pairs uniformly at random.
The results of the experiments on these variant datasets are shown in Figures 13 14 (unipartite setting) and Figures 15 16 (bipartite setting). In general, we see that the results are very similar to those obtained with the original datasets; for comparison, see Figures 23 Figures - and Figures 10 Notably, in the unipartite setting, the Tell DQC-Greedy and OQC-Local detection algorithms perform slightly better both variant datasets than the original datasets. In the middle region where k 15 and y 2 0.8, these algorithms are able to achieve moderate-to-high success rates, unlike on the original datasets.
21

---

Published in Transactions on Machine Learning Research (12/2024)
1.0
R3
021
0.11
111
71
11
011
310
0:34
1.0
P10
3m0 0:3
0.00
010
0.53
0.00
01
0. 01
000 0
AP 102
L. 110
edge density
0.9
3.30
80
8.0
0.9
6.30
8.36
0.8
11
0:1
51:
00
0:31
311
0:11
0:3
311
20
0:3
0.12
0:33
032
6.12
6:3
5
0.8
08
0.2
0:13
9:25
0.05
0.25
0.05
0M
9
0M
0.04
5.2
2:05
2.2
3.00 8:00
0.7
111
0:1 911
1.11
0:31
311
0.11
0-3
7.11
0.
911
92
6:12
911
0.7
3.9
0.55
0.25
0.5 0:55 9
0.65
0:25
0
0.06 6:08
0.05
0.00
0/11 0:05 0.55
6.00
20
2.50
0.00
8.00
3.:00
8.30
60
3-30
6-30
80
8.05
3.00
0.6
00
0.30
00
6.30
80
A-00
DO
OO
3.0
0.
1.03
8.00
1.03
0.5
031
0:11
0:31
0.11
0:31
2.32
0.81
0.23 9.11
0.11
0.33
9.11
6:12
8:13
0.5
2.00
3.00
0:15
9:25
0.55
0.05
25
0.09
8:03
0-15
0.05
6.05
0.04
1.04
0.00
0.p0
2
6 8 10
12 14 16 18 20 22 26 28 30 32
6
10 12 14 16 18 20 22 24 26 28 30 32 34
number of reviewers (k)
number of reviewers (k)
DSD, AAMAS
(b) DSD, S2ORC
1.0
2:00
20
323 0.22 037 331
055 01
010 01
Σ 1.0
00
2:20
0.00
0:40
80
21
1.00
1/21
01
1:00
10:
M 1:00
edge density
0.9
522
0.27
122
027
0.9
042
522
0.0
2.
222
0.27
1:00
10
50
90
36
31
025
0.24
0.40
0.8
86
042
18
5:31
10
1:31
60
60
62
00 1:0
0.7
2.2 027 323 0.77 0:3
328
0.22
0.7
327
0.28
0.33
326
0.25
0.24
28
0.24 6:32
0.7
0/3
322
12
3/2
0.13
9.23
1/03
0:67 ACT
0.6
20
58
00
56
26
0.6
00
8.00
0.5
Rer 02r 22z
0.27
0.2
2-27
0.27
9.2 3.27 0.27 9.27 27
0.27
0.27
927
0.29
8:th
0.5
2A2
3.2
41
0-2
0.43
9
0.52 D-42 AL 0.L 641 0.29 0.L
0.57 9.50 30
1:33
2
6 8 10 12 14 16 18 20 22 24 26
28
30
32
34
2
4
6
10 12 14 16 18 20 22 24 26 28
32 34
number of reviewers (k)
number of reviewers (k)
(c) OQC-Local, AAMAS
(d) OQC-Local, S2ORC
1.0
1:00 0: To
20
0:00
56
6:53
01
00
00
3:35
10
1.0
06
3: 00
00 6:33
00
00
1:33
01
0.
1:02
00
00
0.0:
1000 1.00
edge density
0.9
usr
352
0.9
082
039
0:51
1:35
050
0.8
2:00
2.0
2.10
5.30
8:00
20
8.23
5-28
2.00
2:00
0.8
2.00
200
8.20
2 2.40
2.24
2.
22
2
8.07
2.07
0.7
031
354 035
031
7.7
0.7
0-7
75/
758
1
1. 8:22
0.7
211
384
0.4
0.44
R32
TAT
U:3: 0000
0.6
0:00
2.00
0.6
3:30
26
25
01
1.02
0.5
0.57
057
3.5
00
0:57
259
0:57 0.57
0.57
6.57 0.57
0.59
6:57
0.5
3.04
0.44
31
0.04
0.24
0.84
0.34 0:
5.82 0:73
8:04
1:55
5.00
2
4
6 8 10 12 14 16 18 20 22 24 26 28
30
32 34
2
4
6
10 12 14 16 18 20 22
24 26 28
30
32 34
number of reviewers (k)
number of reviewers (k)
TellTail, AAMAS
TellTail, S2ORC
1.0
1.00
20
2.00
1.00
200
3:20
2000
3.20
00
04
0.03
03
80
1.0
06
4.30
1.00
00
8.20
2.00
2.00
2.30
00
2.01..200
0:
00
2.0:
10
2.00
8.0:
edge density
0.9
0.00
2A3
0.48
0.48
2/5
0-4
0.45
0.09
648
020
0.50
0.79
0.27
M 0.9
252
352
0.12
0:22
252
0.12
0.52
052
0:34
0:
0.9
0:20
0.50
1:0:
0:30
0.0
0.8
10
0.8
08
32
11
00
10
0.7
5.45 0-18 0:18
3/2 1.13
3/8
2.20
0.18
0
0.18
0.48
0.7
350 0:2
52
3 6:50
8
6.32
0.00
25 0:00
070
0.6
3.2
0.48
2
0.4
2/43
0.48
0.45
0.49
@ 0.6
352
3.52
852
9.52
952
0.2
0.00
20
m
20
20
20
30
0:48
06
35
00
02
0.5
06
30
0.2
06
mm
60
cn
5.32
0n A/1
2 4 6 8 10 14 16 18 20 22 24 26 28 30 32 34
2
4
6
10 12 14 16 18 20 22
24 26 28 30 32 34
number of reviewers (k)
number of reviewers (k)
(s) OQC-Creedy, AAMAS
(h) OQC-Greedy, S20RC
1.0
2.00 200
2.00 2 2000
5:20
200
2:20
2.20
200
2:10
2.30
2.00
21
1.0
00
2:
5.00
00
2.10
2.22
05
2.00
2
00
2.02
200
2:00
2:0:
edge density
0.9
Ror
5A7 0/28
0.43
7.47
0.05
0.4
322
0.27
048
020
0.45
041
027
0.41
0.9
150
031
00
051
0.30
32
050
0.7
07
033
0.35
0:21
030
1:37 0:30 0.0
0.8
2.77
0.47
200
5/2
0:4
3/2
6.43
6.17
2:
0.47
0.47
2
0.49
S
0.8
350
0.0
a51
2:20
6:11
0
a51
2.
6.12
2.0
0.00
6:55
02
39
0.9
211
0.25
A.o
0.7
2.00
20
2.00
2.03
50
0.41
7A1
0.07
048
0.08
2:35
0:48
0.7
10
1:0
0:1
3:30
851
80
A5L
00
5.01
5:39
1:87
2.0.
0.6
081
20
0.57
0.11
0.57
D.T 7/28
0.17
0.47
0.09
0.6
350
0.1
0.1
9
9N0
051
6:3
933 939
0.5
to
8.00
3,00
0.00
8:20
00
20
20
00
25
0.00
2.05
0.01
1:00
0.5
0.00
100
200
8.70
00
01
10
2.
10
00
8.00
2.
8.00
2
6 8 10 12 14 16 18 20 22
24 26 28 30 32 34
2 4
6
10 12 14 16 18 20 22
24 26 28 30 32 34
number of reviewers (k)
number of reviewers (k)
(i) OQC-Local (including random initializations), AAMAS
(i) OQC-Local (including random initializations)
S2ORC
Figure 11: Density of detected groups on G1. Values indicate the mean edge density of the group of reviewers output by the algorithm, along with standard errors.
22

---

Published in Transactions on Machine Learning Research (12/2024)
bid density (n)
1.0
50
1:5
18
10
186 18g
1.0
611
200
0:1
000
611
14:
1
0.8
0.8
2.2
125
0.26
327
2.1
037
33
035
043
044
643
041
1:03
60
00
0
10
18
00
00
30
5:02
60
60
0.
5.00
0.4
5.38
3.
0:8
5.88
8.38
6-38
0-36
3
0:33
9 36
9
631
0.4
A11
3/1
6.11
6:1
941
0-43
6.11
31
4
6-11
6
941
A4)
6:1
5-0
0:1
0.2
1000.000
1.03
00
0.00
8.00
0:00
20
29
90 0. 8:28
00 00
2.05
0:01
2.
0.2
1.00
0.00
2.20
8.00
2.00
2:00
29
00
5.08 8.05
00
00 9.00
0.00
0.00
8.00
2
4
8 10 12 14 16
18 20 22 24 26 28 30 32
34
2
4
6 8 10
12 14 16
18 20 22 24 26 28 30
32 34
number of reviewers (k)
number of reviewers (k)
(a) OQC-Greedy, AAMAS
(b) OQC-Greedy, S2ORC
bid density (n)
1.0
0
20
1.00
1.0
6-26
x.38
k.0
0.8
511
235
w32
115
022
0-22
31
08L
02
0.8
0:52
382 3M2 0:2
052
9 0.0
0:22
21
012
9:
02 0.00
0.6
2.02
113
0:3
8000
6.33
312
038
0,6
2:00
2:00
302
200
0.33
0:3 902
2:36
2:
6.33
5:06
13
042
2:01
0.23
2.05
0:00
6/2
052
8:00
0.4
13
032
113
343
0:3
0:33
31 33
0:3
3:20
000
0.23
0:33
0.00
0.08 6.38
0,4
0.1
202
302 0.2
0.23 0.02
0.00
0:33
12
0202
0.33
0:03
0:03
0
0.23
8:00
0.2
511
11
311
318
933
bid
0.2
202
6:1
0:2
982
0:22
32
6:31
6
02
5/21
0:2
2
8 12 14 16 18 20
22 26
28
30
32 34
2
6
10 12 14 16
20 22 24 26
28
30
32 34
number of reviewers (k)
number of reviewers (k)
(c) Fraudar, AAMAS
(d) Fraudar, S2ORC
1.0
BAY 2:2 eer 5:
1 9 0-10 3A 1
1.0
R3T 352
353 1. 0-18 352 01 51
01 6:53
45
0/1
6: 0:0
bid density
0.8
10.00
1.0
22
ML
0.8
0
2.31
0:0
011
0.6
2.2
0:7
5/4
0.
6:33
0.6
AV
354
0:57
6:53
30
17
052
9:58
0.0
6:07
0.4
0.4
10
00
2:
2-05
2.01
057
2.00
20
c0
mm AM 26 25
4:47
60
00
0:00
0:
100
bid
0.2
R37
16
257
mm
80
10
0.57
05
5:
087
00
1.57
2
4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34
2 4
6
8
10 12 14 16 18 20
22 24 26 28 30 32 34
number of reviewers (k)
number of reviewers (k)
(e) Speeiazeial, AAMAS
(f) OQC-Specialized, S2ORC
40
1.5
00
10
8.0
20
31
00
60
66
1.0
0000
00
00
100
1-30
60
1.31
60
0
3/05
1.41
1.0
bid density
0.8
033
113
313
033
6:33
311 11
6:4
381
08
63
0.8
6.36
332
3.5
0.36
6.6
3
6.36
35
036
636
35
25
0.23
6:33
79
0
0.6
113
20
0.6
16
00
1.00
0.4
5:15
033
933
0:3
315 315
9:3
011
0:1
0:
93
03 8:12
0.4
7:25
0.25
0.59 2
9. 0.29
26
26
0.15
1 1.
0.25
1:25
0.26
0.2
2.0
613
2.9
513
=50
0:3
8.00
6.33
50
9.50
0:3
013
5:38
0:3
0.33
0.2
3.07
0.36
500
335
6.36
2000
0.3
2000
35
6.
35
2.05
36
6.36
32
2.07
06
0.03
6:3
8.03
0.0
2
4
8
12 14 16 18 20
22 24 26
28
30 32
2
6
10
12
16
18 20 22 24 26
30
32 34
number of reviewers (k)
number of reviewers (k)
(g) DSD, AAMAS
(h) DSD, S20RC
bid density (n)
1.0 R:2
10
1.0
0:2
120
0:29 022
0.1
6.32
6:32
0
1.0
000
00
2.03
23
60
1.0
08
2.
02
8.73
02
3.03
01
02
8.63
1.03
0.6
2.2
027
427
626 026
0.1
31
332 6.35
0.6
6:1
128
6:36
63
125
0.2
6.23
12
07
6.36
6
0.5
026
6.29
076
0.4
km)
2m
31
0L
1
0.4
100
02
1.0
0.2
2.35
:2
031
22)
222
0.2
0.2
3.25
3.2
0.28 2.29
22
0.7
0.22
930
022
6:25
0.2
0.25
123
235
0.2
0.28
925
0.27
5.25
51 28 0.28
25 922 0.22 5:25
330
919
2
4
6 8
12 14 16 18 20 22 24 26 28
30
32
34
2
6
10 12
14 16
18 20 22 24
26
28
30
32 34
number of reviewers (k)
number of reviewers (k)
OQC-Local, AAMAS
(i) OQC-Local, S2ORC
bid density (n)
1.0
2.00.00
10
0:1
3-12 626 6:16
3.5 323
100
10 0.0
3.2
1.0
R10 3.7 3,00 6:33 6:8 3.07 0.08 6/1
0:17 0.00 0:38
6.09
0.22
0.35
0.17
30 0:
0.8
515
020
6.6
3
0.8
3.2
0:8
6.28
022
00
0.3
00
20..0
1.03
00
20
8.00
0.00
35
00
00
60
00
1-01
01
00
0.00 00
80
800 6-36
3-30
00
06
00
60
1/05
05
60
6.60
1.00
0.4
6.1
036
515
311
636
6:6
3
36
0.1
016
6.6
0:1
03
01
6.18
0.4
6.0
3.00
6:3
6.8
0.09
0.06
37
0.
6.37
0707
0.68
5.05
0.07
0.67
0.2
2000000
8.03
8
8.00
20
9.00
00
80
0.00
2.05
2.00
0,2
1:00
2:20
8.00
100
2:10
20
A 00
2.00
30 2.00
0.
8.00
2 4
6 8 10
12 14 16 18 20 22
24 26 28 30
32 34
2
6 8 10
12
14 16
18 20 22
24 26 28 30 32 34
number of reviewers (k)
number of reviewers (k)
(k) TelThil AAMAS
(1) Telfaal, S2ORC
Figure 12: Density of detected groups on G2. Values indicate the mean bid density of the group of reviewers output by the algorithm, along with standard errors.
23

---

Published in Transactions on Machine Learning Research (12/2024)
edge density (y)
edge density (y)
edge density (y)
edge density (v)
1.0
0.03
0.04
1.0
1.0
1.00
1.00
0.8
0.03
0.00
0.04
0.00
0.06
4.00
0.07
0.8
+.00
0.02
+.00
0.02
+.00
0.8
0.01
001
+.01
0.04
0.20
0.8
0.54
0.97
m.03
0.6
2.00
0.01
0.03
2.00
0.04
2.00
1.00
0.06
2.00
0.07
0.6
2.00
0.00
1:00
0.01
2.00
0.01
0.02
1.00
0.02
1.00
0.6
0.02
1.00
0.0
1.00
2.
0.04
0.06
0.02
0.12
1.05
0.6
1.00
0.01
0.02
2.01
1.02
0.05
0.41
1:14
0.77
1.09
5
10
15
20
25
10
15
20
25
5
10
15
20
25
10
15
20
25
number of reviewers (k)
number of reviewers (k)
number of reviewers (k)
number of reviewers (k)
(a) DSD, AAMAS
(b) DSD, S2ORC
(c) OQC-Local, AAMAS
(d) OQC-Local, S2ORC
edge density (y)
edge density (y)
edge density (y)
edge density (v)
1.0
+.00
0.01
0.11
10
100
00
+.00
1.00
1.00
00
1.0
+.00
0.01
+.03
0.05
100
.0
1.00
000
1.00
0.0
1.0
0.01
+.00
0.04
+.01
+.00
0.04
0.39
14
0.99
+.01
1.0
+.00
0.00
00
0.01
+00
1.00
1.00
000
1:00
.000
0.8
1.00
0.01
2.00
0.01
0.34
1:15
1.00
1,00
1.00
00
1.00
0.00
1.00
0.00
0.77
11
1.00
100
1.00
1.00
0.8
1.00
0.01
1.01
0.04
1:01
0.03
0.05
2:01
1.12
0.23
0.8
1.00
0.01
2.00
0.01
1.00
0.01
1.00
0000
1.00
1.00
0.6
0.02
0.04
0.03
0.32
0.07
0.30
0.93
0.6
0.01
1.01
003
0.04
0.04
0.05
1.00
0.01
2:01
0.01
0.67
1.14
10
15
20
10
20
15
20
25
number reviewers (k)
number of reviewers (k)
number reviewers (k)
number of reviewers (k)
(e) TellTail, AAMAS
(1) TellTail, S2ORC
(g)
OQC-Greedy, AA-
(h) OQC-Greedy, S2ORC
MAS
edge density (y)
edge density (y]
1.0
4.00
0.01
0001
0.04
0.02
000
0.04
001
1.00
0.00
1.0
0.01
0.01
0.12
+.10
1:
0.21
1:00
0.0
+.00
1.00
0.8
5.00
0.01
0:01
0.03
0.01
0.04
*.01
0.04
0.06
+:01
0.8
0.01
0.01
+.00
0.01
0:00
0.01
1:13
0.80
5.10
0.90
1.01
1.01
0.06
0.6
0.00
1:00
1.01
001
1.13
10
20
number of reviewers (k)
number reviewers (k)
(i) OQC-Local (including random initializations),
(i) OQC-Local (including random initializations).
AAMAS
S2ORC
Figure 13: Performance of detection algorithms on Gi with variant datasets. Values indicate the mean
Jaccard similarity between the true set of colluders and the algorithm output, along with standard errors.
edge density (y)
edge density [y)
edge density (y)
edge density (y)
1.0
1:03
0.12
0.23
03
0.30
0.38
0.45
1.0
0.12
0.23
0.24
02
0.31
000
0.33
0:03
1.0
0.18
042
5.05
0.49
0.58
0.63
1.0
0.37
0:03
0.35
0.47
00
0.49
0.8
2.03
0.07
0.22
0.02
0.32
0.0
0.34
2.03
2.01
0.39
0.8
2.04
0.16
2.04
0.19
02
0.26
2.02
0.28
0.36
1.02
0.0
1.06
0.16
1.05
0.45
2 02
0.53
0.53
2 2.03
0.60
2.03
0.0
2.06
0.26
0.35
0.06
0.04
0.44
0.45
0:03
0.46
2.03
0.6
0.02
0.07
0.19
0.02
0.24
0.02
+.03
0.31
02
0.35
0.6
0.03
0.15
4.04
0.25
0.21
0.02
0.22
0.02
0.27
0.02
0.6
0.18
4:06
0.35
0.03
+
0.42
04
0.49
03
0.56
+.04
0.6
4.05
0.26
0.37
2.04
0.03
0.33
0.35
02
0.39
t.0
3
10
15
20
25
5
10
15
20
25
5
10
15
20
25
10
15
20
25
number reviewers (k)
number of reviewers (k)
number reviewers
number of reviewers (k)
(a) Fraction of target pa-
(b) Fraction of target pa- pers with colluder as-
Fraction of colluders with colluder assigned,
(c)
(d) Fraction of colluders with a colluder assigned, S20RC
pers with colluder as- signed, AAMAS
signed, S20RC
AAMAS
Figure 14: Success of colluders in terms of & and y with variant datasets. Values indicate the mean for each metric along with standard errors.
24

---

Published in Transactions on Machine Learning Research (12/2024)
bid density (n)
bid density (n)
bid density
1.0
0.00 0.01 0.02
0.000:00=..11
0.000
1.00 1.00
1.0
0.00 0.00
0.000
0.00
0:000 0.00
1 00 1.00
bid density
0.01 0.12 0.42 1.00
1.000.00.001133:
1.0
0.00 0.
=000
0000
00
0.01
0.01
0.6
+.00
0.00
01 01
0.01 0.01
+.01 *.01#.11
0.04 0.23
0.6
4.00
0.00
0.00
0.00
00
0.00 0.01 0.01
+:
0.6
0.01
00
0.00 00 0
0.03
0.04
0.12 0.19
0.6
0.00
=.00
0.
00
+.00
0.1
= 000
0.01 0.01
0.2
2.00 2. 2:
0.00
0.2
0.00 =.00
0.00
0.2
01
2:00
01
0.04
0.2
0.01
5
20
25
10
25
15
number of reviewers (k)
number of reviewers (k)
number of reviewers (k)
number of reviewers (k)
(a)
Ore-Greedy, AA-
(b) OQC-Greedy, S2ORC
(c) Fraudar, AAMAS
(d) Frandar, S2ORC
MAS
bid density (n)
bid density (n)
bid density (n)
bid density (n)
0.09
2.00
0.65
1.0
08
0.17
=0000.00.02
0.010.04
1.0
=000
0.00 0.000000
1.00 16
0.01
0.6 +.00 0:0 00 + 14 +10 12
0.01 0.01 0.29 0.77
0.6 1.000
0.00 0.01 0.01 0.39
+ 00 0.
0.6 00.00 +.00 =.00 +:0
0.01 0.02 0.03 0.06 0.20
0.6
0.00
=
0. 0.11 0.01 0.01
00
+.00
=0:00:::00 00 =0
0.2 4.01 00 1.01 2.01
0.01 0.01 0.01 0.02 0.03
0.2
4.00 00 1.00
0.00
0.00 0.00 0.00 0.01
0.2
000000000000000000 01 #1 = 2.00 2.000
0.01
0.02 0.2 0.03 0.05
0.2
= 0:00 2.00 00 00
0.00
0.00 0.1 0.01 0.01
5 10 15 20 25
10 15
25
10
20
25
15 20 25
number revlewers (k)
number of revlewers (k)
number of revlewers
number of reviewers (k)
(e)
OQC-Specialized,
(f)
OQC-Specialized,
(8) DSD, AAMAS
(h) DSD, S2ORC
AAMAS
S2ORC
bid density (n)
bid density (n)
bid density (n)
bid density (n)
1.0 2.01 133.
0.01 0.20 0.39
1:16 1 0.0
0.50 0.88
1.0
1.00
0.00
2.00
0.00
10 10 1.10
0.00
0.10 0.11
1.0
2.00 1:00 00
0.00 0.00 0.02
13 00
0.81 1.00
1.0
=:00
0.00 0.01 0.01 0.06 0.30
0: 1.00 01 1
0.00
0.17
=1
0.6
00
0.2 +.00 0.00 #.01
0.00 0.00 0.01 0.02 0.07
0.2
4.00 =.00 0.00
0.00
0.00
0.00 0.00 0.00
0.2
0.01
0.00 0.00 0
0.02
0.03 0.03 0.04
0.2
0.00
1.00 +.00
0.00
0.1 0.01 0.01
00 =:00
5 10
15 20 25
10
15
20 25
5 10
15 20 25
10
is 20 25
number reviewers (k)
number of reviewers (k)
number reviewers
number of reviewers (k)
(i) OQC-Local, AAMAS
()) OQC-Local, S2ORC
(k) TellTail, AAMAS
(1) TellTail, S2ORC
Figure 15: Performance of detection algorithms on G2 with variant datasets. Values indicate the mean Jaccard similarity between the true set of colluders and the algorithm output, along with standard errors.
bid density (n)
bid density (n)
bid density (n)
bid density (n)
1.0
+.04
0.15 0.24
02
+.
0:33
0.37
0.02
1.02 02
0.46
1.0
0.12
+.04
0.18 0.26
03
0.28 0.30
0.02
1.b
0:30
= 06
0.44
0.
0.51
04
0.
0.62
04
0.66
1.0
0.22
=.06
+
0.32
04
0.04
0.39
03
0.45
0:48
0.6 +.07
0.09
0.33
0.6
0.02
03
0.16
0.26
0.6
03
544
0.6
=.03
+.04
0.26
0.2
0.06
2.03
0.18
2.02
0.0
0.08
1.02
0.16
0.2
0.10 0.28
04
0.4...4..033 0.0
03
0.39 0.36
0.2
2.03
2.04
0.15
2.03
5 10 15 20 25
10 15
20
25
5 10
20
25
15 20 25
number of reviewers (k)
number of reviewers (k)
number of reviewers (k)
number of reviewers (k)
(a) Fraction of target pa-
(b) Fraction of target pa-
(c) Fraction of colluders
(d) Fraction of colluders with a colluder assigned,
pers with a colluder as-
pers with a colluder as-
with colluder assigned, AAMAS
signed, AAMAS
signed, S2ORC
S20RC
Figure 16: Success of colluders in terms of k and n7 with variant datasets. Values indicate the mean for each metric along with standard errors.
25


Reviews:
[{"review_id": "hBYE5mbLIE", "reviewer": "Reviewer_SFrt", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n1. The paper studies pure bid-based methods for detecting collusion groups and demonstrates that pure bid-based detection is ineffective. The aspect of the attack and corresponding defense is novel.\n2. The empirical evaluation is solid: it evaluates the attack and detection on two (semi-)real datasets and studies the different sizes and densities of the collusion group.\n\n**Weaknesses**\n\nThe detection only outputs one group and the evaluation is to calculate the similarity between the group and the ground-truth collusion group. This understates the power of detection. A stronger version of detection could output multiple groups and the detection can be successful if one of them guess the collusion group is mostly correct. This will introduce a trade-off between detection and assignment quality: more detection groups will increase the chance of successfully detecting the collusion group while influencing the assignment quality. The results of this trade-off should be inexplicitly related to the exact counts in Figure 1.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studies the reviewer-author collusion detection from paper bidding. Specifically, the paper focuses on detecting the collusion rings. The threat model is that there is a group of reviewers, which are authors and bid each other's paper. Intuitively, it turns out the edge density among this group should be large and therefore the paper studies several group detections by optimizing the edge density and its variants. The empirical study shows that none of detections are effective. The attack can escape from the detections while attain a high success rate of getting the target the papers.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Evaluation with any detection that can output multiple groups will be a solution for detecting the collusion group. However, the paper missed this kind of detection, which weakens the conclusion \"All detection algorithms fail to detect some injected collusion rings\".", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "SPHovHOYWs", "reviewer": "Reviewer_iUJd", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**:\n\n1.  Dealing with collusion is a relevant problem for the ML community.\n2.  The paper studies a new variant of the problem (detecting collusion) that has not received sufficient in the past.\n3.  The paper is sound and the claims are almost all very clearly supported by experimental evidence (see point 1 in changes as the main issue regarding this criterion).\n4.  The paper is very well-written, nicely structured, and a pleasant read.\n    1.  I especially enjoyed that the empirical research questions are clearly identified, which gives the experimental analysis a clear goal and structure. The structure is supported by a good split of content between the appendix and the main body.\n5.  The paper generates some insight that could be of general interest to the broader CS community (e.g., regarding the effectiveness of collusion)\n\n**Weaknesses:**\n\nThe paper does not make any significant methodological contribution. It is a purely experimental effort that applies existing methods. At the same time, the experiments are of very limited technical and implementation difficulty. As such, the main contributions of the paper relevant to others are the take-aways from the experiments. While there are some generally interesting observations, in my personal opinion, the generated insights are not sufficient to warrant an independent publication in a top journal. In particular, I have two main problems with the experimental analysis (details below):\n\n1.  The analysis is generally quite superficial. Trends/results are merely observed and not explained.\n2.  Due to the used datasets, the generalizability of the experimental takeaways (the main contribution of the work) remains largely unclear.\n\n*Style of Experimental Analysis*\n\nThe analysis of the experiments could be expanded in terms of the description of the results, the identification of trends, and the search for explanations of these trends (For instance, the results from the different sections could be linked better together. E.g., in the analysis from 4.2, how much possible room for improvement is there in light of the results from Section 4.1?). Generally, the paper only rarely asks \"Why?\" questions. There are numerous possibilities to deepen the analysis. Just to give one example, at the end of Section 4.3, when concluding that \"approximately one-third of colluder-authored papers have at least one colluder assigned, and approximately one-half of colluders have at least one colluder assigned to one of their papers.\" it would be interesting to ask what the properties of these papers are and why only \"so few\" colluders benefit. I am not saying that these are questions that need to be addressed in the paper but some more in-depth analysis would be appreciated and would increase the relevance of the work.\n\n*Generalizability*\n\nAnother issue is that the data used in the experiments is only semi-realistic. I am aware that there are no better datasets out there to be used; however, given that the experiments are the main focus of the paper it is worrying that it is unclear how much the results generalize beyond the specific settings examined in the paper. This concern gets strengthened because the data generation methods make numerous model assumptions (that might all be plausible but come without a justification and analysis of their impact). One way to address this issue could be to vary the generation methods slightly and see how this impacts the results. If results turn out to be robust, it would become more plausible that the results generalize.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studies the problem of detecting collusion rings between reviewers in peer-reviewing systems based on reviewers' bids. The main approach here is to search for dense subgraphs of the bidding graph (i.e., a set of reviewers who bid on many of each other's papers). In particular, it analyzes the following three questions in two different models of the problem:\n\n1.  How dense are the subgraphs in bidding datasets without collusion?\n2.  Can existing fraud detection algorithms effectively identify collusion rings?\n3.  How effectively can colluders manipulate the system?", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See Weaknesses. In addition:\n\n1. (the authors have satisfactorily clarified this issue in their response); In the experiments on detecting colluding reviewers, the best result over multiple initializations is returned. This is only possible if the ground truth is known, which will not be the case when \"applying\" the algorithm right? As a result, all the results discussed in these and other settings should be understood as best-case results? If this is correct, then this should be highlighted and discussed prominently and should be presented alongside average and worst-case findings. \n2.  Why is text similarity not used for detecting collusion rings? What is the reason for explicitly disregarding this information source? Are there scenarios out there where bids are available but text similarity is not (the authors point to assignment algorithms that combine information from both sources to come up with an assignment)\n3.  It would be nice to make the discussion of the differences between the two settings in terms of algorithms and results a bit more pronounced.\n4.  It should be clarified how much the contribution of the paper goes beyond that of Jecmen et al. 2023 and in what exact ways the two papers do (and do not) overlap.\n\nQuestions that came to mind while reading that paper (no need to address them but could be used to guide extensions): How do the detection algorithms perform if there are multiple colluding sets of authors? Another possible way to extend the paper would be the customization of fraud detection algorithms to the collusion setting.\n\n*Some minor comments:*\n\n- Page 2: \"Concretely, we consider the question: is it possible to effectively detect collusion rings from only the bids and authorships?\" This might be a good point to briefly point to alternative approaches that are not pursued in the paper to clarify the scope of the paper and its relation to other approaches.\n\n- Page 4: \"(e.g.) Amazon.\"\n\n- Page 6: \"Formally, we denote the graph...\": might be good to point out that the graph is directed to avoid confusion.\n\n- Page 7 (y-axes): Are these values exact, rounded, or lower bounds?\n\n- Page 9: \"We then add edges uniformly at random between reviewers until the subgraph has edge density at least γ\"- > This is referring to colluding reviewers, right?\n\n- Page 10: \"uniformly at random and remove all other such bids.\" What does \"such\" refer to?\n\n- Page 11: What precisely are \"target\" papers (cf. line 5)", "broader_impact_concerns": "The paper already discusses the risk of collusion algorithms that output false positives at various places; not in a broader impact statement though.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "eNfyq7AbuZ", "reviewer": "Reviewer_9GQ8", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "After receiving some pointers from the editor, I now recognize that I was not calibrated for TMLR when I wrote this review. In particular, I note in the review critieria: \"We explicitly avoid [the] terms (“significant”, “impactful”, “novel”), and focus instead on the notion of “interest”.\" My review appealed to these forbidden concepts. My apologies.\n\nIn the end, it's of course up to the editor to accept or reject the paper, and it's fine with me if they reach a different conclusion than I did. I probably would not have taken the time to provide a peer review if I had realized at the outset that my professional judgement about the importance of the work wasn't being sought. I have no concerns about the work's correctness. It appears that what remains is a question about the extent to which the paper's claims are supported by the evidence.\n\nThe paper makes two broad kinds of claims. One kind is unproblematic: the following algorithm running on the following synthetic dataset yielded the following pattern of results. I have no concerns about any claims of this kind made in the paper. Of course, it's not altogether clear why anyone would care about such claims, given the substantial gaps between the questions that were studied and the phenomena motivating the paper. The paper certainly gives the impression that its findings have broader scope; see e.g. in the abstract:\n\n\"In this work, we tackle the question of whether it is feasible to detect collusion rings from the paper bidding. To answer this question, we conduct empirical analysis of two realistic conference bidding datasets, including evaluations of existing algorithms for fraud detection in other applications. We find that collusion rings can achieve considerable success at manipulating the paper assignment while remaining hidden from detection\"\n\nOne might argue that this text doesn't constitute a \"claim\". The authors are fairly careful to offer caveats in their claims; nevertheless, one is left with the sense that the paper offers useful, actionable conclusions about the real world beyond having studied a limited, synthetic setting:\n\n\"These results suggest that collusion cannot be effectively detected from the bidding using popular existing tools\"\n\n\"Overall, we find that malicious reviewers can manipulate the paper assignment to moderate success while remaining within typical or difficult-to-detect levels of density in the bidding graph. This provides evidence to support the conclusion that malicious reviewer behavior cannot be effectively detected using just the bidding data.\"\n\nIt seems to me that the concerns raised in my original review (if rephrased to avoid verboten words like \"significance\") speak to the question of whether these claims are justified: whether the experiments described in this paper support meaningful conclusions about the extent to which collusive behavior can be detected in real conferences. I agreed to review this paper because I was optimistic that I would learn something about the topic, but in the end was quite unconvinced that the study says much about real collusion, given (in part) its restriction to only bidding data, to certain algorithms, to substantially synthetic data, to a particular model of collusion behavior, to a simplified model of bidding, to collusion in which rewards are exchanged only within the system, and to a single scoring function. \n\nRelatedly, the authors also make claims like:\n\n\"An important contribution of our work is to provide direction for future work.\"\n\nIt's not clear to me what this direction is supposed to be, beyond \"it would be a good thing to relax the many limitations of our work\". I don't see that as an important contribution so much as a drawback of the existing work.\n\nAnyway, again apologies for misunderstanding the TMLR review criteria, and I hope this clarification is useful. I understand that the authors will already have seen my original review, so rather than editing it I've prepended these comments; my original review (which contains more technical detail) follows.\n\n=========\n\nOverall, I am not convinced that these conclusions are significant, and so I do not recommend publication at TMLR. The paper studies a given set of heuristic algorithms in a synthetic setting, under restricted assumptions about the data available to the algorithms and about how colluders will behave, and concludes that these methods are not effective at identifying collusive behavior. This is not a proof that realistic collusion cannot be detected in real data! \n\nThe paper makes a large number of highly restrictive assumptions that limit the importance of its analysis:\n* Maybe the algorithms could be better\n* Maybe the simulated data models the wrong behavior. \n* Why restrict only to bids and authorships? \n* Why restrict to one model of collusive behavior: it's a game theoretic setting, and behavior will surely change in response to detection strategies.\n* Simulating only positive and negative bids is simplistic. I'm not convinced that multiple levels of review enthusiasm necessarily make the colluder's job easier; e.g., they may also make detection easier.\n* It is not at all clear that colluders should all receive reciprocal benefit. Some reported examples of collusion involve senior people pressuring junior people to accept their papers. In such cases, the reciprocal benefits come outside the reviewing process (support for tenure; absence of retaliation; etc.)\n* The paper studied one scoring function (many have been considered) and a simple matching algorithm based on min-cost flow that lacks any constraints that aim to mitigate collusion.\n\nMore fundamentally, the paper focuses on *detecting* collusion. This is a fundamentally hard objective; there is no way that (particularly absent other meta-data) collusive behavior should have a different signature from bidding from a clique of reviewers with similar interests. I am personally not convinced that detection is more important than mitigation. Surely the important goal here isn't moral policing, it's finding a way to run conferences that does not allow collusion rings to succeed.\n\nSome other comments:\n\nDo the authors investigate a trade-off between how easy the collusion is to detect and how effective it is in the first place? (Ineffectiveness comes both from probability someone else will be matched and probability your reviewing load will be assigned to something else.)\n\nIf the authors are willing to assume that colluding bidders would opt out of data collection, the opt-out field would be a great signal to use in detecting collusion.\n\nAn interesting question would be the extent to which various mitigation strategies proposed are effective at preventing the matching of colluders with papers, even in simulated data such as that studied here.\n\np7 claims the analysis would help conferences estimate the false-positive rates of their mitigation strategies. This is really implausible, and not only because in many cases the submission is not able even to compute these values, instead offering lower bounds. Nobody thinks that an enormous nummber of distinct reviewers are engaged in collusive behavior at conferences. So the actual number of potential matches to which a heuristic applies is a better estimate of the false positive rate than the numbers given here, not least because it would be based on a conference's own data rather than on simulated data or partial data from a long-ago AAMAS conference. Seems to me a conference should be comfortable that a mitigation strategy has a small risk of false positives if it applies only to a small number of potential matches overall, and otherwise should expect a large number of false positives. \n\nThe whole analysis of dense subgraphs seems to miss a fundamental issue. Some areas are popular, with many papers and reviewers, and others are not. Unpopular areas are likely to yield dense subgraphs organically. Calibrating the density of detected subgraphs by comparing to the popularity of the area might be useful.\n\nCitations seemed heavily skewed towards those involving a certain set of coauthors. Various other relevant research is cited but its ideas are not engaged with as deeply.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies the problem of detecting collusive behavior in paper bidding at computer science conferences. The work attempts to detect simulated collusion in two datasets, using various off the shelf algorithms, and shows that collusive behavior cannot reliably be detected.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see above.", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Overall, the reviewer appreciated your detailed response and paper updates. I am happy to recommend acceptance and ask for one minor update (see below). 

Two reviewers mention that the evaluation and design of collusion detection algorithms used in this paper might be restrictive in that detection is "binary." The authors provide a reasonable justification regarding the practical aspects of this (and the impracticability of false positives). This leads the authors to conclude that "All detection algorithms fail." Unless I missed it, I suggest the paper should at least briefly discuss this issue. I found your response to reviewer SFrt useful, and something at the level would be sufficient. 

I also suggest the authors consider the final comment from reviewer 9GQ8, but I will let them decide whether it needs to be addressed in the final version of their work.
Decision: Accept (poster)