# WIRE: A Weighted Item Removal Method for Unsupervised Rank Aggregation
## 5 Experiments
### Comparison Framework and Evaluation Measures  
The effectiveness of WIRE was compared against a wide variety of non-weighted and weighted rank aggregation methods. The first set included Borda Count and CombMNZ as they were formalized in [29], the first and the fourth Markov chain-based methods (MC1, MC4) of [19], the Markov chain framework (MCT) of [20], the Robust Rank Aggregation (RRA) method of [21], the Outranking Approach (OA) of [24], the traditional Condorcet [15], and Copeland Winners [30].  
The second set included the following four state-of-the-art weighted methods: DIBRA with and without list pruning (termed DIBRA-P and DIBRA, respectively) [13], the weighted approach of [26] based on Preference Relation Graphs (PRGs), and the Agglomerative Aggregation Method (AAM) of [27]. In all methods, we used the same hyper-parameter values as those suggested in the respective studies. Our item removal method was tested in combination with DIBRA, and we refer to it as DIBRA-WIRE in the discussion that follows. In all experiments, we kept the values of the two hyper-parameters of WIRE constant. Therefore, the number of buckets was fixed to \(B=5\), and we set \(\delta_{1}=0.5\).  
The quality of the output lists was measured by employing the following three widespread IR measures: Precision, Normalized Discounted Cumulative Gain (nDCG), and Mean Average Precision (MAP). The first two are computed at specific cut-off points of the aggregate list \(\mathcal{L}\), and we refer to them as P@\(k\) and N@\(k\), respectively. Precision@\(k\) is defined as follows:  
\[\text{P@}k=\frac{1}{k}\sum_{i=1}^{k}\text{rel}(\mathcal{L}_{i}), \tag{10}\]  
where \(\text{rel}(\mathcal{L}_{i})\) is a binary indicator of the relevance of the \(i\)-th element of \(\mathcal{L}\) (1/0 for relevant/non-relevant).  
On the other hand, nDCG is defined as the Discounted Cumulative Gain (DCG) of \(\mathcal{L}\) divided by the Discounted Cumulative Gain of an imaginary ideal list that has all relevant elements ranked at its highest positions as follows:  
\[\text{DCG@}k=\sum_{i=1}^{k}\frac{2^{\text{rel}(\mathcal{L}_{i})}-1}{\log_{2}( k+1)}, \tag{11}\]  
\[\text{N@}k=\frac{\text{DCG@}k}{\text{iDCG@}k}. \tag{12}\]  
Finally, Mean Average Precision (MAP) is defined as the mean of the average precision scores among a set of \(T\) topics as follows:  
\begin{table}
\begin{tabular}{l l r r} \hline \hline
**Dataset** & **Description** & **Topics \(|T|\)** & \(n\) & \(k\) \\ \hline MOLO & Synthetic, modest number of long lists & 20 & 50 & 100 \\ MASO & Synthetic, many short lists & 20 & 100 & 30 \\ FESO & Synthetic, few short lists & 20 & 10 & 10 \\ MOSO & Synthetic, modest number of short lists & 20 & 50 & 30 \\ MAVSO & Synthetic, many very short lists & 20 & 500 & 5 \\ FEVLO & Synthetic, few very long lists & 20 & 10 & 200 \\ CTT22 & Real, 2022 TREC Clinical Trials Track & 50 & 41 & 1000 \\ NTDT23 & Real, 2023 TREC NeuCLIR Technical Docs Track & 41 & 51 & 1000 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Attributes of the benchmark datasets.  
\[\text{MAP}=\frac{1}{|T|}\sum_{t=1}^{|T|}\overline{P}(t)=\frac{1}{|T|}\sum_{t=1}^{| T|}\frac{1}{R_{t}}\sum_{k=1}^{|\mathcal{L}|}\text{rel}(\mathcal{L}_{k})\text{P@}k, \tag{13}\]  
where \(\mathcal{L}_{i}\) is the \(i\)-th element of the aggregate list \(\mathcal{L}\), \(\text{P@}k\) denotes the Precision measured at \(\mathcal{L}_{k}\), and \(R_{t}\) is the total number of relevant entries for the topic \(t\in T\).