<table><caption>Table 4: Results of Task T2, single-label multiclass quantification under prior probability shift.</caption><thead><tr><th>Rank</th><th>Run</th><th>RAE</th><th>AE</th></tr></thead><tbody><tr><td>1</td><td>UniOviedo(Team1)</td><td><strong>0.92173</strong> ± <strong>0.70476</strong></td><td>0.02097 ± 0.00566</td></tr><tr><td>2</td><td>Lamarr</td><td>1.03016 ± 0.84658</td><td>0.01412 ± 0.00322</td></tr><tr><td>3</td><td>UNSW</td><td>1.07856 ± 0.96585</td><td><strong>0.01274</strong> ± <strong>0.00353</strong></td></tr><tr><td>4</td><td>SLD</td><td>1.16158 ± 0.99066</td><td>0.01343 ± 0.00346</td></tr><tr><td>5</td><td>PACC</td><td>1.19418 ± 1.13479</td><td>0.01552 ± 0.00424</td></tr><tr><td>6</td><td>KDEy</td><td>1.20166 ± 1.05091</td><td>0.01367 ± 0.00367</td></tr><tr><td>7</td><td>DM</td><td>1.27189 ± 1.09683</td><td>0.01578 ± 0.00405</td></tr><tr><td>8</td><td>ACC</td><td>1.34787 ± 1.16063</td><td>0.01640 ± 0.00427</td></tr><tr><td>9</td><td>CC</td><td>2.30963 ± 1.38323</td><td>0.01660 ± 0.00310</td></tr><tr><td>10</td><td>PCC</td><td>2.67505 ± 1.60472</td><td>0.01931 ± 0.00337</td></tr><tr><td>11</td><td>TeamCUFE</td><td>4.02872 ± 2.12809</td><td>0.02587 ± 0.00334</td></tr></tbody></table>

<figure><img src="image_20.png" /><figcaption>Fig. 3: Error-by-shift plot for T2.</figcaption></figure>

Figure 3 displays the “error-by-shift” plot of T2. In this case, we measure the amount of shift in terms of absolute error (since the signed difference we used in Figure 2 is only defined in the binary case). This plot reveals some interesting facts. First, that most methods degrade their performance at increasing levels of PPS. UniOviedo(Team1) seems to be the most robust in this respect, though. Second, the performance of all methods seems erratic at very high and (specially) very smaller amounts of PPS. However, the density of experiments (greened background) tells us that the number of experiments involved in such cases is very small, which explains the higher variability. This plot also reveals that the distribution of the “amount of shift” generated via APP is close to normal; something that was already echoed in previous research [38].
<table><caption>Table 5: Results of Task T3, ordinal quantification under prior probability shift.</caption><thead><tr><th>Rank</th><th>Run</th><th>NMD</th><th>Macro-NMD</th></tr></thead><tbody><tr><td>1</td><td>UniOviedo(Team1)</td><td><strong>0.06438</strong> ± <strong>0.04773</strong></td><td>0.08007‡ ± 0.02397</td></tr><tr><td>2</td><td>Lamarr</td><td>0.06585 ± 0.04706</td><td><strong>0.07878</strong> ± <strong>0.02038</strong></td></tr><tr><td>3</td><td>PCC</td><td>0.06680 ± 0.05069</td><td>0.09004‡ ± 0.03532</td></tr><tr><td>4</td><td>KDEy</td><td>0.06904 ± 0.04982</td><td>0.08297‡ ± 0.01993</td></tr><tr><td>5</td><td>UniOviedo(Team2)</td><td>0.07212 ± 0.05310</td><td>0.08771‡ ± 0.02564</td></tr><tr><td>6</td><td>CC</td><td>0.07974 ± 0.04760</td><td>0.08820‡ ± 0.01524</td></tr><tr><td>7</td><td>TeamCUFE</td><td>0.10726 ± 0.07755</td><td>0.18647‡ ± 0.11300</td></tr><tr><td>8</td><td>DM</td><td>0.10939 ± 0.06160</td><td>0.11445‡ ± 0.01200</td></tr><tr><td>9</td><td>SLD</td><td>0.11107 ± 0.06903</td><td>0.10947‡ ± 0.01220</td></tr><tr><td>10</td><td>ACC</td><td>0.11944 ± 0.06496</td><td>0.12473‡ ± 0.01498</td></tr><tr><td>11</td><td>PACC</td><td>0.12363 ± 0.06522</td><td>0.12830‡ ± 0.01189</td></tr></tbody></table>

## 4.3 Task T3: Ordinal Quantification under Prior Probability Shift

Table 5 shows the results of the participating teams in T3. Also in this case, UniOviedo(Team1) achieved the best result for the official evaluation metric, which in this case is NMD since this is an ordinal problem. The score obtained by UniOviedo(Team1) is statistically significantly better than the rest of the participating systems. However, in terms of Macro-NMD, the Lamarr team achieved the best result. Nevertheless, according to the Wilcoxon test, no method appears to be statistically significantly different in terms of average rank performance.

It was expected that UniOviedo(Team1) and Lamarr would perform well in this task, as both teams implemented solutions that take into account the ordinal nature of the data. For example, UniOviedo(Team1) directly optimised for the evaluation loss, while Lamarr explicitly regularised their solutions towards a uniform distribution, which ultimately favours smooth solutions. What was unexpected, however, was the seemingly good performance of PCC, a method that not only disregards the ordinal nature of the data but also does not attempt to counter any shift in the priors. This is a clear indication that the samples provided may be characterized by a lower degree of prior probability shift compared to tasks T1 and T2. The reason why, is that the samples are “natural” (generated via NPP), i.e., are not generated via artificially (via APP) imposing widely varying degrees of prior probability shift, as was instead the case for T1 and T2.

Figure 4 displays the averaged performance of the top-5 methods as a function of different levels of “jaggedness” of the tested distributions. More precisely, we compute the jaggedness of a distribution $p_σ$ as:

$$J(p_{\sigma}) = \frac{1}{2} \sum_{i=2}^{n-1} \left( -p_{\sigma}(y_{i-1}) + 2p_{\sigma}(y_i) - p_{\sigma}(y_{i+1}) \right)^2 \quad (4)$$
<figure><img src="image_21.png" /><figcaption>Fig. 4: Quantification performance as a function of the distribution “jaggedness”.</figcaption></figure>

The plot reveals that all methods degrade their performance as the distributions become “jaggy”. However, UniOviedo(Team1), Lamarr, and UniOviedo(Team2) seem to behave more robustly across the entire spectrum. That Lamarr and UniOviedo(Team2) performed well in T3 was to be expected, as both methods have been designed with ordinal considerations in mind. Also, UniOviedo(Team1) performed well in T3, despite the fact that this method is not specifically suited for ordinal data. Notwithstanding this, note that UniOviedo(Team1) is trained using the validation data as well, which inherently showcase many plausible patterns of ordinal distributions, so one might expect that this method learns to handle the ordinality which resides in the distributions. In contrast, other methods such as PCC and KDEy, which are completely agnostic to the ordinal nature of the data, tend to perform worse in difficult scenarios. The fact that, notwithstanding this, PCC and KDEy rank second and third in the table is a consequence of the fact that most of the samples display low levels of jaggedness.

## 4.4 Task T4: Binary Quantification under Covariate Shift

Table 6 reports the results obtained for T4. In this case, the Lamarr team scored the best result in terms of the official evaluation measure (RAE), while SLD obtained the best AE. Somehow surprisingly, though, the RAE score obtained by UniOviedo(Team1) is not statistically significantly different from the Lamarr's score, even though UniOviedo(Team1) ranked 5th. This may be explained by the variance of their system, which is markedly higher than that of methods SLD, DM, and KDEy, which occupy the 2nd, 3rd, and 4th positions in the rank, respectively.

As recalled from Section 2.8, task T4 is not only characterized by covariate shift but also by a shift in the priors. It is thus interesting to disentangle the
<table><caption>Table 6: Results of Task T4, binary quantification under covariate shift.</caption><thead><tr><th>Rank</th><th>Run</th><th>RAE</th><th>AE</th></tr></thead><tbody><tr><td>1</td><td>Lamarr</td><td><strong>0.10930</strong> ± <strong>0.33394</strong></td><td>0.02191 ± 0.01765</td></tr><tr><td>2</td><td>SLD</td><td>0.11497 ± 0.39872</td><td><strong>0.02013</strong> ± <strong>0.01594</strong></td></tr><tr><td>3</td><td>DM</td><td>0.11559 ± 0.32287</td><td>0.02341 ± 0.01853</td></tr><tr><td>4</td><td>KDE<sub>y</sub></td><td>0.11804 ± 0.32916</td><td>0.02380 ± 0.01865</td></tr><tr><td>5</td><td>UniOviedo(Team1)</td><td>0.12975<sup>‡</sup> ± 0.42556</td><td>0.02122 ± 0.01617</td></tr><tr><td>6</td><td>ACC</td><td>0.26187 ± 1.09686</td><td>0.03054 ± 0.02454</td></tr><tr><td>7</td><td>PACC</td><td>0.28918 ± 1.33046</td><td>0.03078 ± 0.02480</td></tr><tr><td>8</td><td>CC</td><td>1.11975 ± 4.33704</td><td>0.08292 ± 0.04915</td></tr><tr><td>9</td><td>PCC</td><td>1.45553 ± 5.63934</td><td>0.10735 ± 0.06316</td></tr><tr><td>10</td><td>TeamCUFE</td><td>2.49902 ± 9.93390</td><td>0.23017 ± 0.16531</td></tr></tbody></table>

systems' performance in terms of both types of shifts, separately. Figure 5 displays the performance of the methods (on logarithmic scale) as a function of the amount of prior shift. The trends we observe are, by and large, in line with those of T1. Figure 6 instead displays the distribution of the errors for the top-5 methods as a function of the prevalence of the Books domain in the test samples, therefore effectively reflecting the amount of covariate shift with respect to the training distribution. This plot shows a weak tendency to improve as the prevalence of Books in the test samples increases, thereby approximating the training mixture (made of 90% Books and 10% Electronics) and reducing the amount of covariate shift. This tendency is more evident for Lamarr, the best performer system for this task. One possible reason why the covariate shift has a weak effect on the results may be that the embeddings were generated using the ELECTRA-Small model [10], which is specifically trained to capture sentiment polarity, rather than topical information, which may therefore be obscured.

Interestingly, PCC has obtained a poor score. This is relevant since this method is considered to behave robustly in the presence of covariate shift. However, it is also known that its performance degrades when some shift in the priors is also at play [24], as is the case for T4. Figure 7 compares the performance of PCC against Lamarr, the top performer in Table 6, proving that PCC's performance in terms of RAE is out of scale.

Figure 8 reports the diagonal plot for T4. Interestingly enough, most methods seem to perform very well notwithstanding the fact that the underlying distributions are affected by covariate shift as well. As witnessed for T1, CC, PCC, and TeamCUFE struggle to obtain good predictions for the entire spectrum, since these methods seem to be strongly biased toward the center of the distribution.
<figure><img src="image_22.png" /><figcaption>Fig. 5: Error-by-shift plot for T4.</figcaption></figure>

# 5 Final remarks

In this edition of LeQua, we have observed several tendencies worth remark-
ing. First, that the “symmetric approach” adopted by UniOviedo(Team1) shows
promising performance across many different scenarios and involving differently
characterised types of dataset shift. This symmetric approach regards the prob-
lem as a regression one, in which both the training instances and the test in-
stances are composed by bags of individual datapoints. Such an approach thus
requires many bags, each labelled with its class prevalence values. The team
used (among other things) the validation samples otherwise provided for model
selection. Given that such samples were representative of the type of shift at play
in each task, the model was able to learn the particularities of each problem.
UniOviedo(Team1) thus leveraged more information during the training phase
than the competitors; however, this strategy is not unfair, as the validation sam-
ples were made available to all participation teams. Yet another differentiating
aspect of this method has to do with its ability to optimise specific loss func-
tions. This capability allowed UniOviedo(Team1) to tune their models for the
very same evaluation measures used to rank the participating systems. Perhaps
the most important differentiating aspect of this method is the fact that no
classifier is involved. This appears promising from the point of view of the Vap-
nik's principle (outlined in Section 1), since the method is trained to solve the
quantification problem *directly*.

One notable observation is that the binary quantification problem (exempli-
fied by T1) is almost a “solved problem”, with most methods performing remark-
ably well in this case. In contrast, the multiclass quantification problem remains
significantly more challenging, leaving substantial room for improvement.

Compared to the previous edition, we observed many methods clearly out-
performing SLD. This is noteworthy, since SLD is widely considered a very hard-
to-beat system [1,37,38] and was one of the top-performing methods in the past
LeQua 2022 edition.
<figure><img src="image_23.png" /><figcaption>Fig. 6: Errors of the top-5 methods at different proportions of book and electronics domains.</figcaption></figure>

<figure><img src="image_24.png" /><figcaption>Fig. 7: A comparison of the errors produced by PCC and the top performer method Lamarr, at different proportions of book and electronics domains.</figcaption></figure>
<figure><img src="image_25.png" /><figcaption>Fig. 8: Diagonal plot for T4.</figcaption></figure>

Another important point concerns the difficulty in simulating meaningful levels of prior shift in ordinal problems. Plausible ordinal distributions impose certain constraints (e.g., smoothing requirements) that conflict with protocols designed to artificially alter them. In future editions, we plan to explore more sophisticated mechanisms to challenge participant systems with more abrupt shifting conditions in ordinal problems.

Looking back, we believe that using a sentiment-specific feature extractor reduced the impact of the covariate-shift effect introduced in task T4. For future editions, we plan to incorporate alternative representation mechanisms that better account for the controlled mixture of domains.

To conclude, we believe that LeQua 2024 has provided participating teams with the opportunity to stress-test their cutting-edge systems in a controlled setting, offering valuable insights to the community.

## Acknowledgments

This work has been supported by the SoBigData++ project, funded by the European Commission (Grant 871042) under the H2020 Programme INFRAIA-2019-1, and by the SOBIGDATA.IT and FAIR projects funded by the European Commission under the NextGenerationEU program. This work has also been funded by the QuaDaSh project (P2022TB5JF) "Finanziato dall'Unione europea —Next Generation EU, Missione 4 Componente 2 CUP B53D23026250001".
The authors' opinions do not necessarily reflect those of the European Commission.

## References

1. Alexandari, A., Kundaje, A., Shrikumar, A.: Maximum likelihood with bias-corrected calibration is hard-to-beat at label shift adaptation. In: Proceedings of the 37th International Conference on Machine Learning (ICML 2020). pp. 222–232. Virtual Event (2020)

2. Bella, A., Ferri, C., Hernández-Orallo, J., Ramírez-Quintana, M.J.: Quantification via probability estimators. In: Proceedings of the 11th IEEE International Conference on Data Mining (ICDM 2010). pp. 737–742. Sydney, AU (2010). https://doi.org/10.1109/icdm.2010.75

3. Bunse, M., Moreo, A., Sebastiani, F., Senz, M.: Regularization-based methods for ordinal quantification. arXiv:2310.09210 [cs.LG] (2023)

4. Bunse, M., Morik, K.: Unification of algorithms for quantification and unfolding. In: Proceedings of the 2nd International Workshop on Learning to Quantify (LQ 2022). pp. 1–10. Grenoble, IT (2022)

5. Card, D., Smith, N.A.: The importance of calibration for estimating proportions from annotations. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics (HLT-NAACL 2018). pp. 1636–1646. New Orleans, US (2018). https://doi.org/10.18653/v1/n18-1148

6. Castaño, A., Alonso, J., González, P., Pérez, P., del Coz, J.J.: QuantificationLib: A python library for quantification and prevalence estimation. SoftwareX **26**, 101728 (2024)

7. Castaño, A., Alonso, J., González, P., del Coz, J.J.: An equivalence analysis of binary quantification methods. In: Proceedings of the 37th AAAI Conference on Artificial Intelligence (AAAI-23). pp. 6944–6952. Washington, US (2023)

8. Castaño, A., González, P., González, J.A., del Coz, J.J.: Matching distributions algorithms based on the Earth Mover's Distance for ordinal quantification. IEEE Transactions On Neural Networks and Learning Systems (2022). https://doi.org/10.1109/TNNLS.2022.3179355, forthcoming

9. Castaño, A., González, P., González, J.A., del Coz, J.J.: Matching distributions algorithms based on the Earth mover's distance for ordinal quantification. IEEE Transactions on Neural Networks and Learning Systems **35**(1), 1050–1061 (2024). https://doi.org/10.1109/TNNLS.2022.3179355

10. Clark, K., Luong, M.T., Le, Q.V., Manning, C.D.: ELECTRA: Pre-training text encoders as discriminators rather than generators. In: Proceedings of the 8th International Conference on Learning Representations (ICLR 2020). Addis Ababa, ET (2020), https://openreview.net/pdf?id=r1xMH1BtvB

11. Donyavi, Z., Li, F., Batista, G.: Ensemble Learning to Quantify: The CSE UNSW Team at LeQua 2024. In: Working Notes of the Learning to Quantify: Methods and Applications (LQ 2024) workshop, co-located at the European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML-PKDD 2024), Vilnius, Lituania. Vilnius, LI (2024)

12. Donyavi, Z., Serapio, A., Batista, G.: MC-SQ: A highly accurate ensemble for multi-class quantification. In: Proceedings of the 23rd SIAM International Conference on Data Mining (SDM 2023). pp. 622–630. Minneapolis, US (2023). https://doi.org/10.1137/1.9781611977653.ch70
13. Dussap, B., Blanchard, G., Chérief-Abdellatif, B.: Label shift quantification with robustness guarantees via distribution feature matching. In: Proceedings of the 34th European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML / PKDD 2023). pp. 69–85. Torino, IT (2023). https://doi.org/10.1007/978-3-031-43424-2_5

14. Esuli, A., Fabris, A., Moreo, A., Sebastiani, F.: Learning to quantify. Springer Nature, Cham, CH (2023). https://doi.org/10.1007/978-3-031-20467-8

15. Esuli, A., Molinari, A., Sebastiani, F.: A critical reassessment of the Saerens-Latinne-Decaestecker algorithm for posterior probability adjustment. ACM Transactions on Information Systems 39(2), Article 19 (2021). https://doi.org/10.1145/3433164

16. Esuli, A., Moreo, A., Sebastiani, F., Sperduti, G.: A concise overview of LeQua 2022: Learning to quantify. In: Proceedings of the 13th International Conference of the CLEF Association (CLEF 2022). pp. 362–381. Bologna, IT (2022). https://doi.org/10.1007/978-3-031-13643-6_23

17. Esuli, A., Moreo, A., Sebastiani, F., Sperduti, G.: A detailed overview of LeQua 2022: Learning to quantify. In: Working Notes of the 13th Conference and Labs of the Evaluation Forum (CLEF 2022). Bologna, IT (2022)

18. Firat, A.: Unified framework for quantification (2016), arXiv:1606.00868v1 [cs.LG] 2 Jun 2016

19. Forman, G.: Counting positives accurately despite inaccurate classification. In: Proceedings of the 16th European Conference on Machine Learning (ECML 2005). pp. 564–575. Porto, PT (2005). https://doi.org/10.1007/11564096_55

20. Forman, G.: BNS feature scaling: An improved representation over TF.IDF for SVM text classification. In: Proceedings of the 17th ACM Conference on Information and Knowledge Management (CIKM 2008). pp. 263–270. Napa Valley, US (2008)

21. Forman, G.: Quantifying counts and costs via classification. Data Mining and Knowledge Discovery 17(2), 164–206 (2008). https://doi.org/10.1007/s10618-008-0097-y

22. González, P., Castaño, A., Chawla, N.V., del Coz, J.J.: A review on quantification learning. ACM Computing Surveys 50(5), 74:1–74:40 (2017). https://doi.org/10.1145/3117807

23. González, P., Moreo, A., Sebastiani, F.: Binary quantification and dataset shift: An experimental investigation. Data Mining and Knowledge Discovery 38(4), 1670–1712 (2024). https://doi.org/10.1007/s10618-024-01014-1

24. González, P., Moreo, A., Sebastiani, F.: Binary quantification and dataset shift: an experimental investigation. Data Mining and Knowledge Discovery pp. 1–43 (2024)

25. Kawakubo, H., Du Plessis, M.C., Sugiyama, M.: Computationally efficient class-prior estimation under class balance change using energy distance. IEICE Transactions on Information and Systems 99, 176–186 (2016)

26. Kloos, K.: UniLeiden at LeQua2024: Evaluating Continuous Sweep and Comparison Using Underlying Classifiers. In: Working Notes of the Learning to Quantify: Methods and Applications (LQ 2024) workshop, co-located at the European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML-PKDD 2024), Vilnius, Lithuania. Vilnius, LI (2024)

27. Kloos, K., Karch, J.D., Meertens, Q.A., de Rooij, M.: Continuous Sweep: An improved, binary quantifier. arXiv:2308.08387 [stat.ML] (2023)
28. Lotz, T., Bunse, M.: Lamarr at LeQua2024: Regularized Soft-Max Likelihood Maximization. In: Working Notes of the Learning to Quantify: Methods and Applications (LQ 2024) workshop, co-located at the European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML-PKDD 2024), Vilnius, Lituania. Vilnius, LI (2024)

29. Luth, L., Daniel, O., Gomes, G., Maletzke, A.: UniOeste at LeQua 2024: Combining the top-ranked quantifiers. In: Working Notes of the Learning to Quantify: Methods and Applications (LQ 2024) workshop, co-located at the European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML-PKDD 2024), Vilnius, Lituania. Vilnius, LI (2024)

30. Maletzke, A., Moreira dos Reis, D., Cherman, E., Batista, G.: DyS: A framework for mixture models in quantification. In: Proceedings of the 33rd AAAI Conference on Artificial Intelligence (AAAI 2019). pp. 4552–4560. Honolulu, US (2019). https://doi.org/10.1609/aaai.v33i01.33014552

31. Moreno-Torres, J.G., Raeder, T., Alaíz-Rodríguez, R., Chawla, N.V., Herrera, F.: A unifying view on dataset shift in classification. Pattern Recognition 45(1), 521–530 (2012). https://doi.org/10.1016/j.patcog.2011.06.019

32. Moreo, A., Esuli, A., Sebastiani, F.: QuaPy: A Python-based framework for quantification. In: Proceedings of the 30th ACM International Conference on Knowledge Management (CIKM 2021). pp. 4534–4543. Gold Coast, AU (2021). https://doi.org/10.1145/3459637.3482015

33. Moreo, A., Francisco, M., Sebastiani, F.: Multi-label quantification. ACM Transactions on Knowledge Discovery and Data 18(1), Article 4 (2023). https://doi.org/10.1145/3606264

34. Moreo, A., González, P., del Coz, J.J.: Kernel density estimation for multiclass quantification. arXiv:2401.00490 [cs.LG] (2023). https://doi.org/10.48550/arXiv.2401.00490

35. Moreo, A., González, P., del Coz, J.J.: Kernel density estimation for multiclass quantification (2024), https://arxiv.org/abs/2401.00490

36. Moreo, A., González, P., del Coz, J.J.: Kernel density estimation for multiclass quantification. Machine Learning (2024), forthcoming

37. Moreo, A., Sebastiani, F.: Re-assessing the “classify and count” quantification method. In: Proceedings of the 43rd European Conference on Information Retrieval (ECIR 2021). vol. II, pp. 75–91. Lucca, IT (2021). https://doi.org/10.1007/978-3-030-72240-1_6

38. Moreo, A., Sebastiani, F.: Tweet sentiment quantification: An experimental re-evaluation. PLOS ONE 17(9), 1–23 (September 2022). https://doi.org/10.1371/journal.pone.0263449

39. Pennington, J., Socher, R., Manning, C.D.: Glove: Global vectors for word representation. In: Proceedings of the 12th Conference on Empirical Methods in Natural Language Processing (EMNLP 2014). pp. 1532–1543. Doha, QA (2014)

40. Pérez-Mon, O., González, P.: UniOvi Team at LeQua 2024: Quantification via Gaussian Latent Space Representations. In: Working Notes of the Learning to Quantify: Methods and Applications (LQ 2024) workshop, co-located at the European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML-PKDD 2024), Vilnius, Lituania. Vilnius, LI (2024)

41. Quiñonero-Candela, J., Sugiyama, M., Schwaighofer, A., Lawrence, N.D. (eds.): Dataset shift in machine learning. The MIT Press, Cambridge, US (2009). https://doi.org/10.7551/mitpress/9780262170055.001.0001

42. Saerens, M., Decaestecker, C.: Decision-making with unknown priors in supervised classification (2010), unpublished manuscript
43. Sauerens, M., Latinne, P., Decaestecker, C.: Adjusting the outputs of a classifier to new a priori probabilities: A simple procedure. Neural Computation 14(1), 21–41 (2002). https://doi.org/10.1162/089976602753284446

44. Sakai, T.: Comparing two binned probability distributions for information access evaluation. In: Proceedings of the 41st International ACM Conference on Research and Development in Information Retrieval (SIGIR 2018). pp. 1073–1076. Ann Arbor, US (2018). https://doi.org/10.1145/3209978.3210073

45. Schumacher, T., Strohmaier, M., Lemmerich, F.: A comparative evaluation of quantification methods. arXiv preprint arXiv:2103.03223 (2021)

46. Sebastiani, F.: Evaluation measures for quantification: An axiomatic approach. Information Retrieval Journal 23(3), 255–288 (2020). https://doi.org/10.1007/s10791-019-09363-y

47. Smith, N.A., Tromble, R.W.: Sampling uniformly from the unit simplex. Tech. rep., Johns Hopkins University (2004), https://www.cs.cmu.edu/~nasmith/papers/ smith+tromble.tr04.pdf

48. Tasche, D.: Class prior estimation under covariate shift: No problem? In: Proceedings of the 2nd International Workshop on Learning to Quantify (LQ 2022). pp. 11–26. Grenoble, IT (2022)

49. Vapnik, V.: Statistical learning theory. Wiley, New York, US (1998)

50. Werman, M., Peleg, S., Rosenfeld, A.: A distance metric for multidimensional histograms. Computer Vision, Graphics, and Image Processing 32, 328–336 (1985)
# UniLeiden at LeQua2024: Evaluating Continuous Sweep and Comparison Using Underlying Classifiers

Kevin Kloos

University of Leiden, NL

**Abstract.** We compared the continuous sweep quantifier with median sweep, SLD, and DyS using Task T1 of the LeQua2024 competition data. We fitted 100 different underlying support vector machines and evaluated all quantifiers on all models. Continuous sweep is outperformed by SLD and DyS if a well-performing underlying classifier exists. Using worse quantifiers, continuous sweep and median sweep appeared to be more stable than DyS and SLD. We compared these findings with existing results in quantification literature.

## 1 Introduction

Quantification Learning is a task that is focused on predicting the prevalence of a data set rather than individually labelling each observation [1, 2]. In earlier years, quantification has been a side product of classification, but over the last decades, quantification learning has been developed to a standalone field with sophisticated tasks and methods. One of those tasks are the LeQua competitions which are designed to evaluate and compare quantifiers with each other. The LeQua competitions contains various tasks regarding quantification learning [3]. In this paper, we focus on task T1 of the LeQua2024 competition. This task concerns binary quantifiers from which the data is affected by prior-probability shift.

In this paper, we compare the continuous sweep quantifier with median sweep, SLD, and DyS. All quantifiers use a support vector machine with a radial basis function as an underlying classifier. We not only investigate the raw performance of the quantifiers, but also investigate the relationship with the underlying classifiers. We finalize the paper with a discussion about the competition and a generalization to other studies.

## 2 Methods

In this study, we compare our four quantifiers. These four quantifiers need an underlying classifier to compute probabilities for each observation. In this section, we explain how we computed the underlying classifier and elaborate on the
four quantifiers. Extra emphasis is put on the Continuous Sweep quantifier, because the author of this paper is one of the developers of Continuous Sweep and it is the most unknown quantifier out of the four. Moreover, we submitted the prevalences extracted from Continuous Sweep to the LeQua2024 competition.

**Underlying classifier** The underlying classifier of all quantifiers is a support vector machine using a radial basis function. Using the kernlab package supported by the Tidymodels library [4], we can tune this support vector machine with two hyperparameters: the cost (C) and the radial basis function sigma (σ). Moreover, the data is preprocessed by normalizing all predictor variables. A max entropy grid is used to find 100 pairs of hyperparameters that optimally cover the hyperparameterspace. We therefore fit 100 SVMs with different hyperparameters. Using the SVM, we can predict the probabilties of observations that belong to the positive class. With 5-fold cross validation, we also estimate the probabilties of all observations in the training data, which will be used to fit the quantifiers.

**Median Sweep (MS)** Median sweep is an ensemble quantifier in the group *Classify, Count, and Correct* [5]. First, we use the SVM to compute probabilities of all observations in the test data. Accordingly, we compute an adjusted count estimate for every probability that occurs in the test data. The adjusted count estimates are computed using true and false positive rate estimated by the cross validated probabilities from the training data. Consequently, we discard unreliable estimates, that is, every adjusted count estimate where the difference between the true and false positive rates are smaller than 0.25. Finally, we compute the median of the remaining adjusted count estimates as our median sweep prevalence estimate.

**Continuous Sweep (CS)** Continuous sweep is a quantifier that is similar to median sweep [6]. More details of Continuous Sweep can be found in [6], but we provide a short explanation of the method. Continuous sweep and median sweep are different in two characteristics. First, continuous sweep uses continuous functions to estimate the true and false positive rates. In this task, we used the kernel cumulative density function from R's *ks* package to compute the true and false positive rates with the cross-validated training data. The continuous functions of the true and false positive functions enabled us to integrate the adjusted count estimates with respect to the estimated probabilities. Therefore, the second change is that we compute the mean of all adjusted count estimates using integration, instead of computing the median. Moreover, we were able to improve the procedure of discarding unreliable adjusted count estimates. We derived derivations for the bias and variance of continuous sweep which enabled us to optimize the value of the minimal difference between true and false positive rates (i.e., $p^Δ$) that an adjusted count estimate is determined as "reliable". The reliable prevalence estimates are located between $\theta_l$ and $\theta_r$. The final continuous
sweep estimate is the mean area under the curve of the adjusted count function between $\theta_l$ and $\theta_r$.

**SLD** The SLD algorithm is an expectation maximization approach to the quantification task [7]. The prevalence of the training data is used as a starting point to update the test prevalence iteratively by an optimal Bayes classifier until convergence is reached. The stopping criterion is a prevalence difference of 0.0001 between two updates or a maximum of 1000 iterations.

**DyS** The DyS algorithm is an algorithm that matches histograms to find the prevalence of a test set [8]. First, the cross-validated probabilities of the training set are used to construct histograms of the probabilities in the positive and negative class. Moreover, the probabilities of the test set are used to make a histogram for the test data. Using a tenary search, we aim to find the optimal mixture between the positive and negative class probabilities to match the test histogram as good as possible. The measure we used to describe the difference between histogram is the hellinger distance. Moreover, default values of 8 bins are used to construct the histograms and the stopping criterion is a prevalence difference of 0.0001 between two updates.

# 3 Results

We fitted 100 different SVM-RBF models with different hyperparameters using a max entropy grid. Out of these 100 models, 76 converged. We compared the performance of the four quantifiers with 1) their best MAE overall, and 2) using the model with the lowest ROC-AUC. From Table 1, we see that SLD has the lowest MAE out of the four models in the two scenarios. Moreover, the differences between the minimum MAE with the MAE of the model with the lowest ROC-AUC are small across the quantifiers.

We also compared the performance of the classifiers against their quantification performance, both within as well as between quantifiers. For all 76 converged models, we evaluated the ROC-AUC of the classifier against the cross-validated probabilities from the training data and we computed the MAE of all four quantifiers on each of the models. Two models had a ROC lower than 0.8, which means that those models perform very bad compared to the other models. Those two models are excluded from the illustrations. In Figure 1, we see that the best classifiers are embraced by all quantifiers. SLD and DyS perform better than CS and MS using the best classifiers. However, these two classifiers are less stable if the classifiers perform worse. For a ROC-AUC around [0.90 – 0.92], SLD had a remarkable drop in performance. Moreover, out of the 76 converged models, SLD had a MAE lower than 0.05 for 22 models, whereas DyS, MS, and CS had a MAE lower than 0.05 for 47, 45, and 44 respectively. For classifier with a low ROC-AUC, CS and MS outperformed SLD and DyS, but the MAE's respectively are fairly high compared to the good classifiers. Moreover, we extract from Figure 1 that DyS performed either good or very bad for worse classifiers.
<table><caption>**Table 1.** The MAE of four quantifiers evaluated on 1000 development sets. At the top, the lowest MAE of every quantifier across all models. At the bottom, the MAE of every quantifier based on the largest ROC-AUC. The columns of the table denote the quantifier, the model number, the cost hyperparameter (C), the rbf sigma hyperparameter ($\sigma$), the ROC-AUC metric of the model, and the MAE of the quantifier given the classifier.</caption><thead><tr><th>Quantifier</th><th>Model no.</th><th>C</th><th>σ</th><th>ROC-AUC</th><th>MAE</th></tr></thead><tbody><tr><td>Continuous Sweep</td><td>77</td><td>4.436</td><td>3.28 × 10<sup>-4</sup></td><td>0.9177</td><td><strong>0.0241</strong></td></tr><tr><td>Median Sweep</td><td>74</td><td>0.236</td><td>3.68 × 10<sup>-3</sup></td><td>0.9099</td><td>0.0267</td></tr><tr><td>SLD</td><td>77</td><td>4.436</td><td>3.28 × 10<sup>-4</sup></td><td>0.9177</td><td><strong>0.0205</strong></td></tr><tr><td>DyS</td><td>77</td><td>4.436</td><td>3.28 × 10<sup>-4</sup></td><td>0.9177</td><td>0.0223</td></tr><tr><td>Continuous Sweep</td><td>78</td><td>25.1</td><td>4.45 × 10<sup>-4</sup></td><td>0.9198</td><td><strong>0.0246</strong></td></tr><tr><td>Median Sweep</td><td>78</td><td>25.1</td><td>4.45 × 10<sup>-4</sup></td><td>0.9198</td><td>0.0291</td></tr><tr><td>SLD</td><td>78</td><td>25.1</td><td>4.45 × 10<sup>-4</sup></td><td>0.9198</td><td><strong>0.0209</strong></td></tr><tr><td>DyS</td><td>78</td><td>25.1</td><td>4.45 × 10<sup>-4</sup></td><td>0.9198</td><td>0.0224</td></tr></tbody></table>

<figure><img src="image_26.png" /><figcaption>**Fig. 1.** Plot that compares the MAE of the four quantifiers against the ROC-AUC. On the left, we compare the MAE against the ROC-AUC in general for all models. On the left, we compare the difference between the MAE of the displayed quantifier against Continuous Sweep, where a value higher than zero indicates a higher MAE for the respective quantifier.</figcaption></figure>
# 4 Discussion

Our quantifier, Continuous Sweep, underperforms against good baseline quantifiers like SLD and DyS in the LeQua2024 competition. If a good classifier can be obtained from the training data, SLD and DyS performed well. Similar results have been found in the previous LeQua2022 competition, where DyS and SLD were good baseline classifiers [3]. If the underlying classifier performs worse, the ensemble methods such as Continuous Sweep and Median Sweep had a lower performance drop than, and outperformed, SLD and DyS. These findings could be translated to the elaborative comparison of [9], where median sweep generally outperformed SLD and DyS for datasets with a binary target variable. These datasets used were usually much smaller than the datasets of the LeQua competitions. Whereas the LeQua competition contained a training set of 5000 observations with 1000 development sets containing 250 observations, the datasets used in [9] were smaller baseline UCI or Kaggle datasets only splitted into a training and a test set.

In future research, it could be interesting to investigate the performance of quantifiers when it is difficult to construct a good (probabilistic) classifier. Moreover, it could be interesting to investigate whether some decision rules could be developed to choose a suitable quantifier based on the characteristics of an underlying classifier. For example, if one has an excellent classifier, we could safely apply SLD, whereas a worse classifier might be better used by a quantifier like continuous sweep. Last, it might also be interesting to investigate the performance of direct learners when it is difficult to construct a good classifier.

# References

1. González P, Castaño A, Chawla NV, del Coz JJ. A Review on Quantification Learning. ACM Computing Surveys. 2017;50(5):74:1-74:40.

2. Esuli A, Fabris A, Moreo A, Sebastiani F. Learning to Quantify. 1st ed. The Information Retrieval Series. Cham: Springer Nature; 2023.

3. Esuli A, Moreo A, Sebastiani F, Sperduti G. A Detailed Overview of LeQua@CLEF 2022: Learning to Quantify; 2022. Available from: https://api.semanticscholar.org/CorpusID:251471941.

4. Kuhn M, Wickham H. Tidymodels: a collection of packages for modeling and machine learning using tidyverse principles; 2020. Available from: https://www.tidymodels.org.

5. Forman G. Quantifying Counts and Costs via Classification. Data Mining and Knowledge Discovery. 2008;17(2):164-206.

6. Kloos K, Karch JD, Meertens QA, de Rooij M. Continuous Sweep: an improved, binary quantifier; 2023. Available from: https://arxiv.org/abs/2308.08387.

7. Saerens M, Latinne P, Decaestecker C. Adjusting the Outputs of a Classifier to New a Priori Probabilities: A Simple Procedure. Neural Computation. 2002;14(1):21-41.

8. Maletzke A, dos Reis D, Cherman E, Batista G. DyS: A Framework for Mixture Models in Quantification. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 33; 2019. p. 4552-60.

9. Schumacher T, Strohmaier M, Lemmerich F. A Comparative Evaluation of Quantification Methods; 2023.
Ensemble Learning to Quantify: The CSE UNSW
Team at LeQua 2024

Zahra Donyavi, Feiyu Li, and Gustavo Batista

University of New South Wales, Sydney, Australia
{z.donyavi, feiyu.li, g.batista}@unsw.edu.au

**Abstract.** LeQua 2024 is a data challenge to facilitate the comparative evaluation of quantification methods for class-prior estimation, also known as quantification or learning to quantify. The challenge focuses on training predictors, termed "quantifiers," to estimate the relative frequencies of classes within sets of unlabeled data points. Notably, the datasets are affected by class prevalence shifts, exhibiting prevalences in the test set that differ from the training set. We propose two ensemble methods, Multiple Classifiers - Single Quantifier (MC-SQ) and Single Classifier - Multiple Quantifiers (SC-MQ), for binary and multi-class quantification tasks. Additionally, we introduce EMQ-ini, a new variation of the Expectation-Maximization algorithm for Quantification (EMQ) method. This variation uses the predicted target prior from the quantifier Generalized Probabilistic Adjusted Classify & Count (GPACC) as the initial point of log-likelihood maximization. We use EMQ-ini as one of the base quantifiers of SC-MQ. Our MC-SQ method ranked first in Mean Relative Absolute Error (MRAE), the official competition performance measure, and second in Absolute Error (AE) on the binary quantification task. Our SC-MQ method ranked third in MRAE and first in AE for the multi-class quantification task.

**Keywords:** Prevalence estimation · Target prevalence shift · Quantification.

# 1 Introduction

Quantification learning is used in many real-world scenarios where the objective is to predict the behavior of groups. It is particularly useful in sentiment analysis, which tracks how overall opinions about products, people, or organizations change over time [1]. Instead of classifying individual behaviors, quantification focuses on estimating the distribution of opinions, providing insights into broader trends in public sentiment.

The simplest quantification approach, known as *Classify & Count* (CC), directly applies classification to quantification problems. However, this method suffers from systematic bias in which the error increases linearly as we approach the more skewed class distributions [2]. To address this, researchers have proposed novel quantification methods to provide more accurate estimates of class distributions in the presence of class prevalence shifts.
LeQua 2024 is a competition that challenges participants to evaluate various techniques for binary, multi-class, and ordinal quantification tasks using real-world Amazon product review datasets. The challenge encompasses four tasks:

**Task T1** evaluates binary quantifiers on data affected by prior probability shift (label shift), akin to Task T1A of LeQua 2022.

**Task T2** assesses single-label multi-class quantifiers operating on data points belonging to one of $L > 2$ classes, with data affected by prior probability shift, similar to Task T1B of LeQua 2022.

**Task T3** new to LeQua 2024, evaluates ordinal quantifiers handling a set of $L > 2$ ordered classes, also involving data affected by prior probability shifts.

**Task T4** another new addition, evaluates binary quantifiers on data affected by covariate shifts.

Our contributions focus on Tasks T1 and T2. For the binary quantification task T1, we employ *Multiple Classifiers - Single Quantifier* (MC-SQ) [3], an ensemble method that achieved the top rank in MRAE. MC-SQ leverages multiple classifiers combined with a single quantifier. For the multi-class quantification task T2, we introduce *Single Classifier - Multiple Quantifiers* (SC-MQ), an ensemble approach that secured the third rank in MRAE. SC-MQ combines a single classifier with multiple quantifiers, including our proposed *Expectation-Maximization Quantifier with Initialization Adaptation* (EMQ-ini) method.

The performance of our ensemble methods can be attributed to the combination of multiple classifiers and quantifiers, leveraging the strengths of diverse models while mitigating individual weaknesses. These ensemble approaches enhance overall quantification accuracy. Furthermore, extensive hyperparameter tuning optimized the performance of each component within the ensembles, contributing to the top-ranking results.

This paper is organized as follows: Section 2 describes the classification and quantification methods employed in our ensemble approaches, MC-SQ and SC-MQ. Section 3 details the evaluation process and comprehensive hyperparameter tuning strategy. Finally, Section 4 concludes our work and presents directions for future research.

# 2 Methods

This section describes our proposed ensemble approaches, MC-SQ and SC-MQ, employed for the binary quantification task (T1) and multi-class quantification task (T2).

## 2.1 MC-SQ Approach for T1

For T1, we employ MC-SQ, an ensemble approach that previously achieved the top rank in T1B for the LeQua 2022 competition, based on our post-competition experiments [3]. Figure 1 illustrates the MC-SQ architecture, which consists of
an ensemble of six pairs of classifiers and quantifiers. We introduce diversity by varying the base classifiers while keeping the base quantifier fixed. The chosen quantifier is Distribution y-Similarity (DyS) [4] as it has been recognized as one of the top-performing quantifiers for binary problems in the comparative study conducted by [5].

<figure><img src="image_27.png" /><figcaption>Fig. 1. Schematic of the proposed MC-SQ ensemble approach.</figcaption></figure>

Our approach encompasses the following classifiers: Logistic Regression (LR), Linear Discriminant Analysis (LDA), Support Vector Machines (SVM), Light Gradient Boosting Machines (LGBM), Gradient Boosting (GB), and CatBoost (CB). The rationale behind selecting these algorithms stems from their diverse learning paradigms and their frequent success in various Machine Learning applications. As initially proposed, MC-SQ [3] also incorporates Random Forest and Naive Bayes as base classifiers. However, an extensive evaluation of the competition dataset showed that pairing these classifiers with DyS exhibited higher quantification error rates on the validation data. Consequently, we removed them from our ensemble and introduced CatBoost to maintain diversity among the base classifiers while leveraging its strength in quantification tasks.

We conduct comprehensive hyperparameter tuning for each classifier and quantifier combination to optimise our ensemble approach. This process systematically explores the parameters space to identify the configurations that yielded the best quantification performance on the validation set. Section 3 provides fur-
ther details on our methodology and experimental setup, including the compre-
hensive hyperparameter tuning process employed to optimize the performance
of our ensemble approach.

## 2.2 SC-MQ Approach for T2

For the T2 task, we introduce SC-MQ, an ensemble approach that combines di-
verse quantification algorithms with a single base classifier. We evaluated various
classifiers as potential base classifiers by measuring the quantification error over
the validation set. Our analysis showed that logistic regression (LR), as the base
classifier, coupled with the assessed quantifiers, yielded the lowest quantification
error on the validation set. Consequently, we selected LR as the base classifier for
our ensemble, aiming to optimize overall quantification performance. By fixing
the *Single Classifier* as Logistic Regression, we aim to minimize the variability
introduced by the classification component. This allows the quantification algo-
rithms to be the primary source of diversity within the ensemble. Figure 2 shows
the architecture of our proposed SC-MQ method.

<figure><img src="image_28.png" /><figcaption>Fig. 2. Schematic of the proposed SC-MQ ensemble approach.</figcaption></figure>

We employ four quantification methods in our experiments: *Energy Distance* (EDy) [6], *Kernel Density Estimation* (KDEy) [7], *Generalized Probabilistic Adjusted Classify & Count* (GPACC) [8], and our newly proposed EMQ-ini. We select these methods based on our prior knowledge of quantification techniques and their ability to handle multi-class problems. Each algorithm represents a distinct approach to quantification. For KDEy, we specifically utilize the Maximum
Likelihood (ML) variation, as it showed the best performance for multi-class datasets among all KDEy variants in the original study [7].

Complementing these established quantifiers, we introduce EMQ-ini, a novel variation of the EMQ method [9] we are currently working on. EMQ-ini uses the predicted target priors from GPACC as the initial point of log-likelihood maximization. This competition provides an opportunity to test our idea and evaluate the performance of EMQ-ini on the quantification tasks in a real-world setting. We discuss the EMQ-ini method in detail in the next section.

## 2.3 EMQ-ini

EMQ-ini is a new quantification method designed to enhance the performance of traditional EM-based quantifiers. It leverages an initial estimate from GPACC to provide a more informed starting point for the EM process, improving convergence speed and accuracy, especially in scenarios where class distribution shifts are significant between the training and target domains.

Alg. 1 provides a detailed description of EMQ-ini. Suppose we have $\mathbf{y} = \{y_i\}_{i=1}^L$ as the set of labels, and $\mathbf{x}$ is the unlabelled test set with $N$ instances sampled from the target domain. EMQ-ini takes three inputs: $\hat{p}_{ini}(\mathbf{y})$, which is an estimate of the target prevalence from GPACC, used as an initiation for EM; $p_t(\mathbf{y})$, as the class prevalence from a training set; and $\hat{p}_t(\mathbf{y}|\mathbf{x})$ is the estimate posterior class probabilities (scores) from a classifier trained on training set sampled from the source domain.

EMQ-ini updates the scores before applying the EM iterations. This update is based on the Bayes' rule, similar to the *E step* in EMQ. It begins by computing the prevalence ratio $\mathbf{r}$, which is the element-wise division of prevalence estimate $\hat{p}_{ini}(\mathbf{y})$ by the training prevalence $\hat{p}_t(\mathbf{y})$. This ratio is then used to compute the scores $\hat{p}_{ini}(\mathbf{y}|\mathbf{x})$.

The main iterative process of the EMQ-ini is similar to EMQ and consists of the following steps:

- **E-step:** In this step, the algorithm computes the updated posterior probabilities $\hat{p}^{(s)}(\mathbf{y}|\mathbf{x})$ for the current iteration $s$. This is done by adjusting the initial posterior probabilities $\hat{p}_{ini}(\mathbf{y}|\mathbf{x})$ based on the ratio of the current prevalence estimate $\hat{p}^{(s)}(\mathbf{y})$ to the initial prevalence estimate $\hat{p}_{ini}(\mathbf{y})$, using Bayes' rule.

- **M-step:** In this step, the algorithm updates the prevalence estimate $\hat{p}^{(s+1)}(\mathbf{y})$ for the next iteration $s+1$. This is done by taking the average of the updated posterior probabilities $\hat{p}^{(s)}(\mathbf{y}|\mathbf{x})$ across all instances in the dataset.

The E-step and M-step are iteratively performed until a stopping condition is met (e.g., convergence or maximum iterations reached).

After the iterative process, the final prevalence estimate $\hat{p}(\mathbf{y})$ is set to the last computed prevalence estimate $\hat{p}^{(s)}(\mathbf{y})$.
<caption>**Algorithm 1:** EM Quantifier with Initialization Adaptation.</caption>

**Input:** $\hat{p}_{ini}(\mathbf{y})$, $p_t(\mathbf{y})$, $\hat{p}_t(\mathbf{y}|\mathbf{x})$
**Output:** $\hat{p}(\mathbf{y})$
Prevalence ratio: $\mathbf{r} \leftarrow \frac{\hat{p}_{ini}(\mathbf{y})}{p_t(\mathbf{y})}$;
Updated posterior: $\hat{p}_{ini}(\mathbf{y}|\mathbf{x}) \leftarrow \left[ \frac{r_i \cdot \hat{p}_t(y_i|\mathbf{x})}{\sum_{j=1}^{L} r_j \cdot \hat{p}_t(y_j|\mathbf{x})} \right]_{\text{for } i \leftarrow 1 \text{ to } L}$;
State: $s \leftarrow 0$;
Target prevalence estimate in state $s$: $\hat{p}^{(s)}(\mathbf{y}) \leftarrow \hat{p}_{ini}(\mathbf{y})$;
**while** stopping condition = false **do**
    E step: $\hat{p}^{(s)}(\mathbf{y}|\mathbf{x}) \leftarrow \left[ \frac{\frac{\hat{p}^{(s)}(y_i)}{\hat{p}_{ini}(y_i)} \cdot \hat{p}_{ini}(y_i|\mathbf{x})}{\sum_{j=1}^{L} \frac{\hat{p}^{(s)}(y_j)}{\hat{p}_{ini}(y_j)} \cdot \hat{p}_{ini}(y_j|\mathbf{x})} \right]_{\text{for } i \leftarrow 1 \text{ to } L}$;
    M step: $\hat{p}^{(s+1)}(\mathbf{y}) \leftarrow \frac{1}{N} \sum_{\mathbf{x} \in \mathbf{X}} \hat{p}^{(s)}(\mathbf{y}|\mathbf{x})$;
    $s \leftarrow s + 1$;
    $\hat{p}(\mathbf{y}) \leftarrow \hat{p}^{(s)}(\mathbf{y})$;
    **return** $\hat{p}(\mathbf{y})$;

## 3 Evaluation

The performance of the proposed methods is evaluated on both the validation and test sets for Tasks T1 and T2. Tables 1 and 3 present the results in terms of Mean Absolute Error (MAE) and Mean Relative Absolute Error (MRAE) for the validation and test sets, respectively.

For the binary quantification task T1, Table 1 compares the performance of the individual classifiers coupled with the DyS quantifier and the proposed MC-SQ ensemble method. Among the individual methods, LR-DyS and the MC-SQ ensemble achieved the best MAE of 0.0206 on both the validation and test sets. However, MC-SQ outperformed LR-DyS regarding MRAE, achieving the lowest scores of 0.0869 and 0.0981 on the validation and test sets, respectively.

The parameters for the classifiers and quantifiers used in the T1 task were selected using grid search and optimization to minimize the MRAE metric. This process was facilitated by the QuaPy library [10]. Table 2 presents the selected parameters obtained through this optimization procedure. Notable parameters include the regularization and gamma parameters for Logistic Regression and Support Vector Machines and the number of bins (nbins) used by the DyS quantifier, which varied across the different methods. Any other parameters or hyperparameters not explicitly mentioned in the parameters table are set to their respective default values as defined by the implemented methods.

For the multi-class quantification task T2, Table 3 compares the performance of the proposed SC-MQ ensemble with its individual components: EDy, EMQ-ini,
<table><caption>**Table 1.** Performance comparison of methods on validation and test sets for T1.</caption><thead><tr><th rowspan="2">Method</th><th colspan="2">Validation set</th><th colspan="2">Test set</th></tr><tr><th>MAE</th><th>MRAE</th><th>MAE</th><th>MRAE</th></tr></thead><tbody><tr><td>LR-DyS</td><td><strong>0.0206</strong></td><td>0.0910</td><td><strong>0.0206</strong></td><td>0.1024</td></tr><tr><td>LDA-DyS</td><td>0.0218</td><td>0.0987</td><td>0.0218</td><td>0.1121</td></tr><tr><td>SVM-DyS</td><td>0.0213</td><td>0.1023</td><td>0.0217</td><td>0.1026</td></tr><tr><td>LGBM-DyS</td><td>0.0253</td><td>0.1034</td><td>0.0255</td><td>0.1220</td></tr><tr><td>GB-DyS</td><td>0.0258</td><td>0.1052</td><td>0.0258</td><td>0.1176</td></tr><tr><td>CB-DyS</td><td>0.0233</td><td>0.0954</td><td>0.0233</td><td>0.1145</td></tr><tr><td>MC-SQ</td><td><strong>0.0206</strong></td><td><strong>0.0869</strong></td><td><strong>0.0206</strong></td><td><strong>0.0981</strong></td></tr></tbody></table>

<table><caption>**Table 2.** Classifier and Quantifier selected parameters for T1.</caption><thead><tr><th>Method</th><th>Classifier</th><th>Quantifier</th></tr></thead><tbody><tr><td>LR-DyS</td><td>C = 10, class-weight = balanced</td><td>nbins = 40</td></tr><tr><td>LDA-DyS</td><td>None</td><td>nbins = 30</td></tr><tr><td>SVM-DyS</td><td>C = 24.1967, gamma = 0.0114</td><td>nbins = 30</td></tr><tr><td>LGBM-DyS</td><td>None</td><td>nbins = 30</td></tr><tr><td>GB-DyS</td><td>None</td><td>nbins = 16</td></tr><tr><td>CB-DyS</td><td>Depth = 2, learning_rate = 0.1, l2_leaf_reg = nbins = 18<br>7, iterations = 900</td><td></td></tr></tbody></table>

KDEy, and GPACC. The SC-MQ ensemble achieved the best performance, with MAE scores of 0.0129 and 0.0127 on the validation and test sets, respectively. It also obtained the lowest MRAE scores of 1.1160 and 1.0786 on the validation and test sets.

<table><caption>**Table 3.** Performance comparison of methods on validation and test sets for T2.</caption><thead><tr><th rowspan="2">Method</th><th colspan="2">Validation set</th><th colspan="2">Test set</th></tr><tr><th>MAE</th><th>MRAE</th><th>MAE</th><th>MRAE</th></tr></thead><tbody><tr><td>EDy</td><td>0.0137</td><td>1.3053</td><td>0.0135</td><td>1.2390</td></tr><tr><td>EMQ-ini</td><td>0.0139</td><td>1.1351</td><td>0.0137</td><td>1.1038</td></tr><tr><td>KDEy</td><td>0.0179</td><td>1.4542</td><td>0.0176</td><td>1.4355</td></tr><tr><td>GPACC</td><td>0.0155</td><td>1.2021</td><td>0.0155</td><td>1.1950</td></tr><tr><td>SC-MQ</td><td><strong>0.0129</strong></td><td><strong>1.1160</strong></td><td><strong>0.0127</strong></td><td><strong>1.0786</strong></td></tr></tbody></table>

The performance of SC-MQ can be attributed to the combination of a robust base classifier (Logistic Regression) with a diverse set of quantification algorithms, including the novel EMQ-ini method proposed in this work. Among the individual quantifiers, EMQ-ini was the best-performing single quantifier, outperforming other methods like EDy, KDEy, and GPACC. By leveraging the strengths of multiple quantifiers, with EMQ-ini being the strongest contributor while maintaining a consistent base classifier, the ensemble could effectively capture the diverse characteristics of the multi-class quantification problem.
Table 4 presents the selected parameters using the grid search and minimizing the MRAE for the classifier and quantifiers used in the T2 task. Notable parameters include the regularization parameter for Logistic Regression, the bandwidth parameter for KDEy, and the solver used by GPACC. Additionally, EMQ-ini utilized the exact training prevalences during the initialization step.

<table><caption>**Table 4.** Classifier and Quantifier selected parameters for T2.</caption><thead><tr><th>Method Classifier</th><th>Quantifier</th></tr></thead><tbody><tr><td>EDy C = 1, class-weight = balanced</td><td>None</td></tr><tr><td>KDEy C = 100, class-weight = None</td><td>bandwidth = 0.14</td></tr><tr><td>GPACC C = 0.1, class-weight = balanced</td><td>solver = minimize</td></tr><tr><td>EMQ-ini C = 1, class-weight = None</td><td>exact _train _prev = True</td></tr></tbody></table>

## 4 Conclusion

The results demonstrate the effectiveness of the proposed ensemble approaches, MC-SQ and SC-MQ, in addressing the binary and multi-class quantification tasks, respectively. By combining diverse classifiers and quantifiers, these methods could leverage the strengths of individual components while mitigating their weaknesses, leading to improved quantification performance on both tasks.

For future work, we will investigate EMQ-ini in more depth, focusing on the effect of the initial point of maximum likelihood optimization on this method. Additionally, we will evaluate EMQ-ini on various datasets to ensure its generality and robustness across different scenarios.

## References

1. A. Moreo and F. Sebastiani, "Tweet sentiment quantification: An experimental re-evaluation," *PLoS One*, vol. 17(9), 2022.

2. G. Forman, "Quantifying counts and costs via classification," *Data Min Knowl Discov*, vol. 17, no. 2, pp. 164-206, 2008.

3. Z. Donyavi, A. Serapio, and G. Batista, "Mc-sq: A highly accurate ensemble for multi-class quantification," in *Proceedings of the 2023 SIAM International Conference on Data Mining (SDM)*. SIAM, 2023, pp. 622-630.

4. A. Maletzke, D. dos Reis, E. Cherman, and G. Batista, "Dys: a framework for mixture models in quantification," in *AAAI Conference*, vol. 33, no. 01, 2019, pp. 4552-4560.

5. T. Schumacher, M. Strohmaier, and F. Lemmerich, "A comparative evaluation of quantification methods," *arXiv preprint arXiv:2103.03223*, 2021.

6. J. J. del Coz, "Unioviedo (team2) at lequa 2022: Comparison of traditional quantifiers and a new method based on energy distance." in *CLEF (Working Notes)*, 2022, pp. 1869-1874.
7. A. Moreo, P. González, and J. J. del Coz, "Kernel density estimation for multiclass quantification," *arXiv preprint arXiv:2401.00490*, 2023.

8. A. Firat, "Unified framework for quantification," *arXiv preprint arXiv:1606.00868*, 2016.

9. M. Saeiens, P. Latinne, and C. Decaestecker, "Adjusting the outputs of a classifier to new a priori probabilities: a simple procedure," *Neural computation*, vol. 14, no. 1, pp. 21–41, 2002.

10. A. Moreo, A. Esuli, and F. Sebastiani, "Quapy: a python-based framework for quantification," in *Proceedings of the 30th ACM International Conference on Information & Knowledge Management*, 2021, pp. 4534–4543.
# Lamarr at LeQua2024: Regularized Soft-Max Likelihood Maximization

Tobias Lotz and Mirko Bunse

Lamarr Institute for Machine Learning and Artificial Intelligence,
44227 Dortmund, Germany
{tobias.lotz,mirko.bunse}@cs.tu-dortmund.de

**Abstract.** As members of the Lamarr Institute, we participated in the open LeQua2024 competition. The goal in this competition was to predict the prevalences of classes in unlabeled sets of data, given a labeled training set. Our submission builds on the regularized maximization of a likelihood function with constraints that are implemented through a soft-max operator. Ultimately, this method ranked in the top three across all four disciplines of LeQua2024; most notably, we achieved the first place in discipline T4, a binary quantification task with covariate shift. In this paper, we detail our approach to the competition.

**Keywords:** Quantification · prior probability shift · covariate shift.

## 1 Introduction

LeQua2024¹ was a competition hosted for the evaluation of quantification methods. These methods estimate class prevalences in unlabeled sets of data, i.e., they estimate how often each class appears in each data set [4]. To learn the correspondence between class labels and feature vectors, a labeled training set is provided. Unlike classification, quantification is not concerned with predicting the label of each individual data item; what matters are aggregate predictions for sets of data items [5]. These predictions are complicated by shifts between the training distribution and the target distributions.

LeQua2024 consists of four disciplines that are separately evaluated. Each discipline represents a different quantification setting. Our team from the Lamarr Institute ranked in the top three across all disciplines.

**T1** Binary quantification with prior probability shift (3rd place).

**T2** Multi-class quantification with prior probability shift (2nd place).

**T3** Ordinal quantification with prior probability shift (2nd place).

**T4** Binary quantification with covariate shift (1st place).

In Sec. 2, we introduce the quantification method that we used across all disciplines. Sec. 3 details our optimization of the method's hyper-parameters. We conclude with Sec. 4.

<footnote n="1">¹ See https://lequa2024.github.io</footnote>
## 2 Method

Let $C$ be the number of classes and let $\mathcal{P} = \{\mathbf{p} \ge 0 : 1 = \sum_{i=1}^{C} \mathbf{p}_i\}$ be the set of valid class prevalence vectors. Making a prediction, such that $\hat{\mathbf{p}}_i$ is an estimate of $\mathbb{P}(Y = i)$ in set unlabeled data items, can be realized through the constrained minimization of a loss function $\ell: \mathcal{P} \to \mathbb{R}$,

$$ \hat{\mathbf{p}} = \arg\min_{\mathbf{p} \in \mathcal{P}} \ell(\mathbf{p}) \qquad (1) $$

where $\ell$ is typically defined in terms of an unlabeled data set $\mathcal{D} = \{x \in \mathcal{X}\}$.

Our approach to all disciplines of LeQua2024 evolves around three aspects: the choice of the loss function, the implementation of the $\mathbf{p} \in \mathcal{P}$ constraint, and regularization. Regularization is the only aspect through which we adapt our method to the particularities of each discipline.

**Loss Function** We employ the negative log-likelihood loss proposed by Alexandari et al. [1]. This choice is motivated by an exceptional performance recently reported for this loss function in combination with kernel density estimates [6] and by the winning performance of a closely related method in the first edition of LeQua in 2022 [7]. We opted against kernel densities because our initial experiments did not verify a performance improvement due to them.

**Constraints** We ensure that $\hat{\mathbf{p}} \in \mathcal{P}$ by optimizing over a latent vector $\mathbf{l} \in \mathbb{R}^{C-1}$ that we feed through a soft-max operator $\sigma: \mathbb{R}^{C-1} \to \mathcal{P}$ [2].

**Regularization** In T3, we promote solutions that we deem ordinally plausible [3]. In all other disciplines, we promote uniform solutions, assuming that the testing protocol of the competition exhibits a slight preference towards these outcomes. The impact of regularization is controlled through a hyper-parameter $\tau \ge 0$.

Combining all of the above aspects, our prediction $\hat{\mathbf{p}} = \sigma(\hat{\mathbf{l}}) \in \mathcal{P}$ is obtained after optimizing

$$ \hat{\mathbf{l}} = \arg\min_{\mathbf{l} \in \mathbb{R}^{C-1}} - \sum_{i=1}^{C} \log \sum_{x \in \mathcal{D}} \frac{\hat{\mathbb{P}}(Y=i | X=x)}{\hat{\mathbb{P}}(Y=i)} \cdot [\sigma(\mathbf{l})]_i + r(\mathbf{l}) \quad (2) $$

$$ \text{where } [\sigma(\mathbf{l})]_i = \begin{cases} \frac{1}{1+\sum_{j=1}^{C-1} \exp(\mathbf{l}_j)} & \text{if } i=1 \\ \frac{\exp(\mathbf{l}_i)}{1+\sum_{j=1}^{C-1} \exp(\mathbf{l}_j)} & \text{else} \end{cases} $$

$$ \text{and } r(\mathbf{l}) = \begin{cases} \tau \cdot \sum_{i=1}^{C-1} ([\sigma(\mathbf{l})]_i - [\sigma(\mathbf{l})]_{i+1})^2 & \text{for T1, T2, T4} \\ \tau \cdot \sum_{i=1}^{C-2} ([\sigma(\mathbf{l})]_i - 2[\sigma(\mathbf{l})]_{i+1} + [\sigma(\mathbf{l})]_{i+2})^2 & \text{for T3} \end{cases} $$
We estimate the posteriors $\hat{P}(Y = i | X = x)$ through a multi layer perceptron (MLP) classifier, except for discipline T4, where a logistic regression outperformed the MLP on the validation set. The prior $\hat{P}(Y = i)$ is estimated on the training set, which is also used to train the classifier within each discipline.

The numerical optimization of the loss function in Eq. 2 is realized through an unconstrained Newton conjugate gradient trust-region method [9], as it is implemented in the SciPy package [8]. The full implementation of our approach is publicly available on GitHub<fnref n="2" />

## 3 Hyper-Parameter Optimization

Each discipline of LeQua2024 provides, in addition to a labeled training set, a validation set for hyper-parameter optimization. This validation set consists of multiple sets $\mathcal{D}$ of data items, each sampled with a discipline-specific type of distribution shift. Performance evaluations on the validation data act as an estimate of the final performance on the test data; for the latter, the ground-truth remained hidden throughout the competition. Participants were able to choose those hyper-parameters that perform best during validation.

In order to optimize our hyper-parameter selection, we employed a coarse-to-fine grid-search adaptation strategy, starting from heuristically chosen starting grids. For numeric parameters, if the optimal value is located at the smallest or largest value in the grid, we shift it in such a way, that the currently optimal value now lies at the center. Otherwise, we decrease the difference of the candidate values, in order to allow finer updates to further improve the performance.

The final hyper-parameters that we selected for the different classifiers can be seen in tables 1 and 2. We note that regularization only had a minor impact on the validation performance of our method.

<table><caption>**Table 1.** Final hyper-parameters used for MLP classifiers.</caption><thead><tr><th>task</th><th>hidden layer size</th><th>activation</th><th>alpha</th><th>learning rate</th><th>solver</th><th>τ</th></tr></thead><tbody><tr><td>T1</td><td>512</td><td>tanh</td><td>1e-1</td><td>1e-3</td><td>sgd</td><td>0</td></tr><tr><td>T2</td><td>512</td><td>tanh</td><td>1e-1</td><td>1e-5</td><td>adam</td><td>0</td></tr><tr><td>T3</td><td>320</td><td>tanh</td><td>1e-6</td><td>1e-3</td><td>adam</td><td>1e-3</td></tr></tbody></table>

<table><caption>**Table 2.** Final hyper-parameters used for logistic regressions.</caption><thead><tr><th>task</th><th>C</th><th>class_weight</th><th>τ</th></tr></thead><tbody><tr><td>T4</td><td>0.43571</td><td>None</td><td>1e-5</td></tr></tbody></table>

<footnote n="2">² See https://github.com/tobiaslotz/lequa2024</footnote>
# 4 Conclusion

Our participation in LeQua2024 evolves around a maximum likelihood estimate with constraints that are implemented through a soft-max operator. We employed this estimate across all four disciplines of LeQua2024, with a discipline-specific regularization that only played a minor role on quantification performance. Our method achieved top-ranking results throughout the competition.

The reasons for this outcome—and their implications on future research—remain yet to be discussed. Two essential prerequisites for this discussion are i) specific information about the submissions of the other teams and ii) specific information about the pre-processing of LeQua’s data. Lacking both prerequisites at the moment, we are looking forward to the conclusions that are to be drawn by LeQua’s organizers.

# References

1. Alexandari, A., Kundaje, A., Shrikumar, A.: Maximum likelihood with bias-corrected calibration is hard-to-beat at label shift adaptation. In: Int. Conf. on Mach. Learn. pp. 222–232 (2020), http://proceedings.mlr.press/v119/alexandari20a.html

2. Bunse, M.: On multi-class extensions of adjusted classify and count. In: Int. Worksh. on Learn. to Quantify: Meth. and Appl. pp. 43–50 (2022), https://lq-2022.github.io/proceedings/CompleteVolume.pdf

3. Bunse, M., Moreo, A., Sebastiani, F., Senz, M.: Ordinal quantification through regularization. In: Europ. Conf. on Mach. Learn. and Knowl. Discov. in Databases. pp. 36–52. Springer (2022). https://doi.org/10.1007/978-3-031-26419-1_3

4. Esuli, A., Fabris, A., Moreo, A., Sebastiani, F.: Learning to Quantify. Inform. Retrieval. Series, Springer (2023). https://doi.org/10.1007/978-3-031-20467-8

5. Forman, G.: Quantifying counts and costs via classification. Data Mining and Knowl. Discov. pp. 164–206 (2008). https://doi.org/10.1007/s10618-008-0097-y

6. Moreo, A., González, P., del Coz, J.J.: Kernel density estimation for multiclass quantification (2024), https://arxiv.org/abs/2401.00490

7. Senz, M., Bunse, M.: DortmundAI at LeQua 2022: Regularized SLD. In: Conf. and Labs of the Eval. Forum. pp. 1911–1915 (2022), http://ceur-ws.org/Vol-3180/paper-152.pdf

8. Virtanen, P., et al.: SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Meth. pp. 261–272 (2020). https://doi.org/10.1038/s41592-019-0686-2

9. Wright, S.J., Nocedal, J.: Numerical optimization. Operations Res. and Financial Engin., Springer, 2nd edn. (2006). https://doi.org/10.1007/978-0-387-40065-5
# UniOvi Team at LeQua 2024: Quantification via Gaussian Latent Space Representations

Olaya Pérez-Mon[0000-0002-4527-6698] and Pablo González[0000-0002-9250-0920]

Artificial Intelligence Center, University of Oviedo, Gijón, Asturias, Spain
U0257319@uniovi.es
gonzalezgpablo@uniovi.es

**Abstract.** Most traditional quantification methods often depend on prior probability shift assumptions to develop models that use classifiers for optimal prevalence estimation. In contrast, the idea of this team was to introduce an end-to-end neural network that utilizes Gaussian distributions to achieve invariant sample representations. This paper describes this deep learning architecture and its application to the LeQua 2024 quantification competition, specifically, addressing tasks T2, T3, and T4.

## 1 Motivation

Our main goal in this competition was to evaluate the performance of a novel deep-learning representation layer for quantification. This layer is based on modelling latent spaces using Gaussian distributions, to obtain an invariant representation of bags of examples. Using this layer in a deep neural network, with an architecture suited for quantification [1], we can undertake prevalence estimation tasks and compare the performance of our method against some of the most popular quantification algorithms, which served as baselines in the competition, as well as against the other participants.

We focused on tackling tasks T2 and T3, both complex multiclass problems, as well as T4, which, despite being a binary problem, presents the added challenge of covariate shift. We opted not to participate in task T1, a binary quantification problem where traditional quantification methods are highly effective and much simpler (using our approach for this task might have been seen as overkill). Our solution delivered consistently strong results in the multiclass tasks (T2 and T3), achieving a gold medal in both tasks. In the binary task with covariate shift, T4, we achieved fifth place. The setup and methodology employed to tackle the competition tasks will be detailed in Section 2.

## 2 Method

For this competition, we used a deep neural network designed to address quantification problems. Our approach builds on the same architecture as Deep
Quantification Network (DQN) [1], but instead of using pooling layers for rep-
resenting the bags, we use Gaussian distributions. The architecture consists of
three main components: i) a feature extraction module, which is adapted to
the specific task, it projects features into latent spaces. ii) a bag representation
module that summarizes the feature vectors into a numerical representation of
each bag. In our case, Gaussian distributions are used to model the latent space.
And, iii) a quantification module, a combination of dense linear layers, that learns
the relationship between bag representations and bag prevalences, outputting
prevalence estimates due to a softmax activation function.

This architecture is flexible, allowing the optimization of different loss func-
tions, such as the Relative Absolute Error (RAE) for tasks T2 and T4, and the
Normalized Match Distance (NMD) for T3. In addition, it can be trained using
bags labelled by prevalence, with or without individual example labels.

## 2.1 Layer details

Our method introduces a novel approach to generating invariant bag features using learnable multivariate Gaussian distributions. This representation layer models the latent space, to which the bag examples are projected, by covering it with Gaussian distributions, each defined by a mean vector and a covariance matrix, which are learnable parameters of the network. Each example projected in the latent space is evaluated against the Gaussian distributions by calculating their likelihoods. The mean likelihood across all examples in a bag then shows how well each Gaussian represents the entire bag, resulting in a compact and invariant representation in the latent space.

In order to capture more useful information about the bags, we propose an extension of the network incorporating multiple latent space representations, each modelled with its own Gaussian distributions. To favor learning different latent spaces with different information, we use the Centered Kernel Alignment (CKA) score (for more information, please, see [2]). Since this metric measures the similarity between latent spaces, we incorporated it into the loss function we are minimizing to make them as distinct and informative as possible.

Another important aspect of our approach is that the network can be trained
using only bags labelled by prevalence, but can also benefit when utilizing
individual labelled examples, adding a classification layer to the feature extraction
module that allows the network to learn representations, combining quantification
and classification losses during training. It is also possible to train the net by
combining these two techniques.

When labelled bags are limited, we use data augmentation techniques to
generate new bags. There are two options: i) when only individual labels are
available, we generate new bags using the APP protocol [3] that allows us to
generate bags with a desired prevalence and, ii) when only bag prevalences are
available, we can mix two real bags producing a new augmented bag whose
prevalence is the average of the original bags. When possible, we can combine
both strategies.
# 3 Experiments

The datasets used in this study correspond to the T2, T3 and T4 tasks from the LeQua 2024 competition. T2 is a multiclass problem with prior probability shift, T3 is a multiclass ordinal quantification problem, and T4 is a binary problem with covariate shift.

The official loss function for T2 and T4 is the Relative Absolute Error (RAE) and Normalized Mean Distance (NMD) for task T3 (see [4] for more information on quantification loss functions).

The experimental setup was tailored to optimize the network according to the official loss function for each task. Our quantification method's flexibility allows for various training data configurations, as explained in Section 2. Specifically, for T2 and T4, half of the training bags were generated using APP from example-labelled data, while the other half were labelled by prevalence (with 700 out of 1,000 bags used for training and 300 for validation). To prevent overfitting, the labelled data was augmented by randomly mixing real bags, as described in Section 2.

For T3, the network was trained exclusively on bags generated using APP from the example-labeled dataset, with the 1,000 training bags used for validation and early stopping. In this case, the label information was incorporated into the network helping its convergence.

The number of Gaussian distributions to cover the latent space was fixed to 100. For T2, the most challenging task, we used 18 different latent spaces, each in 5 dimensions. The number of latent spaces for T3 was set to 9, while the other parameters remained unchanged. The limitation in the number of latent spaces was given due to memory limitations in the GPU used for training (12Gb). For T4, we utilized 1 latent space with 17 dimensions and 156 Gaussians. For T4, these parameter values were found using Optuna [5]. For T2 and T3 parameters were chosen manually, without using any automated procedure.

## 3.1 Results

In this section, we present the performance of our proposed method on the three tasks: T2, T3, and T4.

For the T2 dataset, see Table 1, our method significantly outperforms all baseline methods as well as other participants' ones, in terms of the optimized loss function, Relative Absolute Error (RAE). The performance gap is significant, demonstrating the effectiveness of our approach in handling this complex multiclass problem with prior probability shift.

The T3 dataset presents a multiclass ordinal problem with five classes, where the objective loss function to optimize is the NMD, see Table 1. In this case, our method also demonstrates superior performance, although the margin of improvement is narrower compared to T2. Nevertheless, our approach proves to be more effective in dealing with the complexities of ordinal quantification.

Finally, for the T4 dataset, see Table 1, our method was outperformed by other teams' approaches. In this case, our thought was that the ability to train
<table><caption>Table 1: Top five competitors for each of the tasks in which we competed.</caption><thead><tr><th>Ranking</th><th>T2 (RAE)</th><th>T3 (NMD)</th><th>T4 (RAE)</th></tr></thead><tbody><tr><td>1</td><td>0.9217 (Ours)</td><td>0.0644 (Ours)</td><td>0.1093 (tobiaslotz)</td></tr><tr><td>2</td><td>1.0302 (tobiaslotz)</td><td>0.0659 (tobiaslotz)</td><td>0.1150 (EMQ)</td></tr><tr><td>3</td><td>1.0786 (hustav)</td><td>0.0668 (PCC)</td><td>0.1156 (DistMatching-y)</td></tr><tr><td>4</td><td>1.1616 (EMQ)</td><td>0.0690 (KDEy)</td><td>0.1180 (KDEy)</td></tr><tr><td>5</td><td>1.1942 (PACC)</td><td>0.0721 (juanjodelcoz)</td><td>0.1298 (Ours)</td></tr></tbody></table>

the network using bags with a particular type of shift might prove an advantage when this shift is reproduced in the test bags. In practice, quantification methods using an underlying classifier and designed for prior probability shift, performed substantially well in this task, with errors very similar to those obtained by teams in task T1.

## 4 Conclusions

We have reached several interesting conclusions. Firstly, our method has demonstrated strong adaptability across a variety of quantification scenarios, achieving robust results in the presence of prior probability shift, covariate shift and, even in ordinal quantification problems. It is particularly effective in handling multiclass problems, which are often more complex and demanding.

Additionally, our method can optimize different loss functions, such as RAE and NMD, delivering strong results in both cases. Furthermore, our approach is flexible regarding training data. It can handle situations where data is labelled by individual examples, where bags are labelled by prevalence, or when there is a combination of both cases.
References

1. Lei Qi, Mohammed Khaleel, Wallapak Tavanapong, Adisak Sukul, and David Pe-
terson. A framework for deep quantification learning. In *Machine Learning and
*Knowledge Discovery in Databases: European Conference, ECML PKDD 2020, Ghent,
Belgium, September 14-18, 2020, Proceedings, Part I*, pages 232-248. Springer, 2021.

2. Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited, 2019.

3. Andrea Esuli, Alessandro Fabris, Alejandro Moreo, and Fabrizio Sebastiani. *Learning to Quantify*. Springer, Cham, CH, 2023.

4. Fabrizio Sebastiani. Evaluation measures for quantification: An axiomatic approach. *Information Retrieval Journal*, 23(3):255-288, 2020.

5. Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: A next-generation hyperparameter optimization framework. In *Proceedings of the 25rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining*, 2019.
# UNIOESTE at LeQua 2024: Combining the top-ranked quantifiers

Luiz Luth, Daniel Ojeda, Guilherme Gomes, and Andre Maletzke

Western Paraná State University, Foz do Iguaçu, Brazil
{luiz.junior90, daniel.ojeda, guilherme.gomes, andre.maletzke}@unioeste.br

**Abstract. Keywords:** Counting · Class prevalence shift · label shift

## 1 Introduction

In recent years, the problem of class prior estimation (also known as quantification or learning to quantify) has gained significant attention due to its importance in real-world applications where estimating the distribution of classes in unlabelled data is crucial. The challenge extends beyond classifying individual instances to accurately predicting the proportion of different classes in a dataset, a task complicated by prior probability shift (or label shift), where the class distribution in the test data differs from the training data, making standard supervised learning approaches less effective.

The LeQua 2024 competition addresses these challenges through several tasks, and our proposal specifically aims to tackle Task T1. This task focuses on evaluating binary quantifiers, which are tasked with predicting the relative frequencies of a class and its complement under conditions of prior probability shift.

Our proposal for Task T1 of the LeQua 2024 competition addresses the challenge of predicting class proportions under prior probability shift using a straightforward and effective strategy. Instead of relying on a single quantifier, we propose to run several quantifiers on the target dataset. Each quantifier generates a prediction for the class proportions, and we then rank these quantifiers based on their performance. Once ranked, we select the top-performing quantifiers and form the final prediction by averaging the predictions of the top-ranked models. This approach leverages the strengths of multiple quantification methods, increasing robustness and reducing the reliance on any one model. By combining the predictions of the best-performing quantifiers, we aim to produce more reliable and accurate estimates of the class proportions, effectively tackling the prior probability shift that characterizes Task T1.

## 2 Method

This section describes the method applied to obtain predictions for Task T1. First, we evaluated a set of base classifiers for Task T1, including CatBoost [1],
Random Forests [2], Support Vector Machines [3], and XGBClassifier [4]. We conducted a grid search for each classifier to determine the best hyperparameters, as detailed in Table 1.

<table><caption>**Table 1.** Parameter grids for different models</caption><thead><tr><th>Model</th><th>Parameter</th><th>Values</th></tr></thead><tbody><tr><td rowspan="4">XGBoost</td><td>n_estimators</td><td>{100, 200, 300}</td></tr><tr><td>gamma</td><td>{0.01, 0.1}</td></tr><tr><td>max_depth</td><td>{3, 6, 9, 12}</td></tr><tr><td>learning_rate</td><td>{0.001, 0.01, 0.1, 1}</td></tr><tr><td rowspan="3">CatBoost</td><td>depth</td><td>{4, 5, 6, 7, 8, 9, 10}</td></tr><tr><td>learning_rate</td><td>{0.01, 0.05, 0.2, 0.5, 0.8, 1.0}</td></tr><tr><td>iterations</td><td>{10, 20, 50, 100}</td></tr><tr><td rowspan="3">RandomForest</td><td>n_estimators</td><td>{100, 200, 300, 400, 500, 600, 700, 800}</td></tr><tr><td>criterion</td><td>{gini, entropy}</td></tr><tr><td>max_depth</td><td>{-1, 10, 50, 100}</td></tr><tr><td rowspan="2">SVC</td><td>kernel</td><td>{linear, poly, rbf, sigmoid}</td></tr><tr><td>gamma</td><td>{scale, auto}</td></tr></tbody></table>

Furthermore, to account for label shift, we applied the correction technique proposed by Lipton et al. (2018) [5], which adjusts the predictions to mitigate the impact of prior probability shift in the test data.

Next, we employed several well-established quantifiers to generate the class proportion estimates. We selected the best-performing quantifiers for binary problems as described by Schumacher et al. (2021) [6]. We used the following quantifiers: DyS, HDy, SORD, CC, ACC, MS, and EMQ, each with their respective default parameters.

Finally, to generate the predictions, we ranked all the quantifiers and measured the Mean Absolute Error (MAE) for each individual. We also evaluated the performance by combining the quantifiers in groups of three and five, calculating the average prediction for each group. We then compared the performance of the individual quantifiers and their combinations, selecting the best-performing setup based on these results to predict the test set. This approach ensured that we identified the most robust configuration for accurate quantification under the conditions of prior probability shift.

## 3 Results

The results of our experiments indicated that the best-performing setup for Task T1 was a combination of both an individual classifier and a group of quantifiers. Specifically, the SVC classifier with `gamma = auto` and `kernel = poly` yielded the most accurate results among the base classifiers. In terms of quantifiers, the combination of DyS, HDy, and EMQ proved to be the most effective, providing the lowest Mean Absolute Error when their predictions were averaged. This
combined approach outperformed other configurations, and thus it was selected
to predict the test set, ensuring robust and accurate quantification under the
conditions of a prior probability shift.

4 Conclusion

Our approach to Task T1 of the LeQua 2024 competition effectively addressed the challenge of prior probability shift by combining multiple quantification techniques and leveraging an optimized classifier. To further enhance our results, we applied a correction technique that adjusts predictions to mitigate the impact of prior probability shift in the test data, ensuring more reliable class proportion estimates. By averaging the predictions of the top quantifiers and applying the correction, we successfully selected the best-performing setup, which proved to be effective in handling distributional shifts. This strategy highlights the importance of combining classifier performance, quantification methods, and correction techniques to achieve high predictive accuracy under challenging conditions.

References

1. L. Prokhorenkova, G. Gusev, A. Vorobev, A. V. Dorogush, and A. Gulin, "Catboost: unbiased boosting with categorical features, 2017," *arXiv preprint arXiv:1706.09516*, vol. 201, 2017.

2. L. Breiman, "Random forests," *Machine learning*, vol. 45, pp. 5–32, 2001.

3. C. Cortes, "Support-vector networks," *Machine Learning*, 1995.

4. T. Chen and C. Guestrin, "Xgboost: A scalable tree boosting system," in *Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining*, 2016, pp. 785–794.

5. Z. Lipton, Y.-X. Wang, and A. Smola, "Detecting and correcting for label shift with black box predictors," in *International conference on machine learning*. PMLR, 2018, pp. 3122–3130.

6. T. Schumacher, M. Strohmaier, and F. Lemmerich, "A comparative evaluation of quantification methods," *arXiv preprint arXiv:2103.03223*, 2021.