Counterfactual Voting Adjustment for Quality Assessment and Fairer Voting in Online Platforms with Helpfulness Evaluation
Authors: Chang Liu
Yixin Wang
Moontae Lee
Venue: ICML 2025 poster (2025)
Keywords: helpfulness voting
question answers
position bias
herding bias
online evaluation
causal effect
Primary area: applications->social_sciences

Abstract: Efficient access to high-quality information is vital for online platforms. To promote more useful information, users not only create new content but also evaluate existing content, often through helpfulness voting. Although aggregated votes help service providers rank their user content, these votes are often biased by disparate accessibility per position and the cascaded influence of prior votes. For a fairer assessment of information quality, we propose the Counterfactual Voting Adjustment (CVA), a causal framework that accounts for the context in which individual votes are cast. Through preliminary and semi-synthetic experiments, we show that CVA effectively models the position and herding biases, accurately recovering the predefined content quality. In a real experiment, we demonstrate that reranking content based on the learned quality by CVA exhibits stronger alignment with both user sentiment and quality evaluation assessed by GPT-4o, outperforming system rankings based on aggregated votes and model-based rerankings without causal inference. Beyond the individual quality inference, our embeddings offer comparative insights into the behavioral dynamics of expert user groups across 120 major StackExchange communities.

Counterfactual Voting Adjustment for Quality Assessment and Fairer Voting in Online Platforms with Helpfulness Evaluation
Chang Liu Yixin Wang?
Moontae Lee 1
Abstract
answers in Q&A communities help build large-scale knowl- edge repositories through the wisdom of crowds (Wang et al., 2013). For instance, StackOverflow - the largest iconic community within the StackExchange expert Q&A forums - -alone recorded over 24 million questions and 36 million answers (sta). This paper identifies these customer reviews and user answers as key examples of user-generated re- sponses provided for various products and questions.
Efficient access to high-quality information is vi- for online platforms. To promote more useful information, users not only create new content but also evaluate existing content, often through helpfulness voting. Although aggregated votes help service providers rank their user content, these votes are often biased by disparate accessi- bility per position and the cascaded influence of prior votes. For a fairer assessment of informa- tion quality, we propose the Counterfactual Voting Adjustment (CVA), causal framework that ac- counts for the context in which individual votes are cast. Through preliminary and semi-synthetic experiments, we show that CVA effectively mod- els the position and herding biases, accurately recovering the predefined content quality. In a real experiment, we demonstrate that reranking content based on the learned quality by CVA ex- hibits stronger alignment with both user sentiment and quality evaluation assessed by GPT-4o, out- performing system rankings based on aggregated votes and model-based rerankings without causal inference. Beyond the individual quality infer- ence, our embeddings offer comparative insights into the behavioral dynamics of expert user groups across 120 major StackExchange communities.
Volume does not always indicate quality. Although user- generated responses are abundant, their sheer size and di- versity make it difficult to identify genuinely helpful infor- mation. To address this challenge, many service providers frequently implement binary voting systems, allowing users to rate individual responses helpful or not. However, even when aggregated, these helpfulness votes may fail to accurately capture the true quality of the responses. Due to the limited perception of human users (Joachims et al., 2007). more accessible responses displayed at the top tend to attract more votes, leading to position bias. Furthermore, users often conform to majority opinion: the visible count of prior votes can significantly steer subsequent evaluations (Lee et al., 2016), amplifying herding bias. Over time, these biases mutually reinforce a cascading effect: favor- ing content that gains early popularity and overshadowing potentially superior yet less accessible future content.
To fairly assess information quality while mitigating posi- tion and herding biases, we should be capable of addressing two counterfactual questions: What if the content were dis- played in a different position, and what if the content had an equal number of positive and negative votes? In interactive systems, however, it is infeasible to create parallel universes where the same content is evenly distributed across all po- sitions with balanced number votes. To overcome this challenge. we introduce the Counterfactual Voting Ad- justment (CVA), causal framework that systematically accounts for the contextual factors influencing individual votes. Unlike existing models that rely solely on static data, CVA leverages voting trajectories to simultancously adjust position and herding biases, offering principled approach to inferring the causal effect of helpfulness voting on con- tent quality. This framework facilitates fairer rankings that reduce the undue influence of early accessibility and social
1. Introduction
bser-generated content is influential information for deci- sion-making and knowledge discovery. Customer reviews on merchant products and services are essential elements of electronic Word-OF-Mouth (eWOM) (Babić Rosario al., 2016), where 97% of customers rely on online reviews for their daily shopping decisions (Murphy, 2017). User
'Department of Information and Decision Sciences, University of Illinois Chicago, Chicago. United States Uniiversity of Michi- gan, Ann Arbor, MI, United States 'LG Al Research, Scoul, Korea. Correspondence to: Moontae Lee <moootaa@uuicc @di..
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Counterfactual Voting Adjustment for Quality Assessment and Fairer Voting in Online Platforms with Helpfuiness Evaluation
conformity, ultimately leading to a reliable identification of high-quality content among user-generated responses.
provided a comparative analysis of deep learning models, finding that CNN outperforms traditional classifiers, RNNs, and XGBoost. Kastrati et al. (2024) used deep learning to identify helpful reviews and integrated textual and meta-data attributes. Researchers in the business field also leverage computational models to improve the performance of help- fulness prediction. Tay et al. (2020) proposed Dirichlet distribution- aggregation models. Deng et al. (2020) divided voting processes into initial and cumulative stages, with heuristic and systematic cues influencing these stages differently. Du et al. (2021) proposed an end-to-end neu- ral architecture to capture the missing interaction between reviews and their contextual neighbors. Mitra & Jenamani (2021) proposed multi- pustii-pespecctive models combining lexi- cal, structural, and sequential features. Wang et al. (2021) applied random forests and gradient boosting techniques to analyze helpfulness on the Steam gaming platform. Lee et al. (2021) found that XGBoost outperformed other models for predicting review helpfulness in restaurant data. Wang et al. (2020a) proposed Bayesian ranking techniques that balance early and late-posted reviews, only ensuring fairer visibility distribution without considering herding bias. Dev et al. (2019) proved causal effects of different impression signals, such as aggregate vote thus far and position of content, by adopting an instrumental variable (IV) framework. How- ever, their approach is unable to isolate social influence bias with position bias.
Extensive validations and analyses confirm the effective- ness of our CVA framework. First, preliminary experiments demonstrate how CVA mitigates position and herding biases, inferring information quality as desired. Second, a semi- synthetic dataset, designed to closely emulate real commu- nity statistics, shows that CVA achieves the strongest align- ment with prior content quality. Third, since the true quality of information in real StackExchange data is unknown, we leverage GPT-4o, a Large Language Model (LLM), to eval- uate sentiment in user comments and content quality by LLM-as-a-judge like human experts (Chiang & Lee, 2023). The results prove that our CVA re-rankings significantly outperform both existing system rankings and model-based re- re-rankings that lack causal inference. Fourth, community embeddings reveal varying degrees of behavioral biases, generalizing our findings on representative communities across the major StackExchange forums. Finally, through a case study illustrating how CVA answers to the key coun- terfactual questions, we diseuss its potential benefits for service providers as well as users.
2. Related Work
Our work expands recent studies in position and herding bias verification, helpfulness voting modeling, and approaches for mitigating biases and inferring information quality.
Mitigating the biases. The Chinese Voting Process (CVP) proposed a generative model that infers the underlying qual- ity of individual responses by de- de-biasing the herding bias (Lee et al., 2016). While the CVP utilizes trajectories of voting to re-weight the importance of individual votes per different contexts, this model cannot de-bias the position bias together with the herding bias. Note that it is important to recognize the observational nature of our dataset. Even if we reconstruct all trajectories of writing new responses and voting existing responses similar to the CVP, our dataset consists only of the votes that have been cast to existing answers. To address this, some studies consider both whether to vote and helpfulness vote ratio as outcomes (Kuan et al., 2015; Wu et al., 2021). They and CVP both proposed a two-stage model that cannot mitigate the posi- tion bias and herding bias at the same time. CVP also did a cross-community comparison. Helpfulness votes represent the subjective valuation of the information judged by others, and they are also the aggregated perceived utility of the information (Kuan et al., 2015). Due to its subjectivity, the sensitivities to biases vary among different communities.
We aim to simultaneously mitigate position and herding bias and uncover fairer quality estimates of responses rather than just quantify biases and predict the next votes.
Previous studies proved the existence of the position and herding biases. Wan (2015) identified the Matthew effect, where early-posted reviews garner more votes due to visibil- ity, and the Ratchet effect, which sustains their dominance through cumulative feedback loops. (Zhou & Guo, 2017: Alzate Barricarte et al., 2024; Sipos et al., 2014) also ver- ified that reviews ranked higher always get higher review visibility. (Risselada et al., 2018: Tseng et al., 2023) verified social conformity and demonstrated herd behavior in online voting systems, revealing that prior votes strongly influence subsequent decisions. Guo et al. (2019) used neurological tools, showing that users experience positive neurological feedback when their votes align with majority opinions, sug- gesting cognitive biases toward popular reviews. Liu et al. (2007) mixes these two biases into winner circle bias.
Modeling the helpfulness voting. In the Computer Science field, modern studies predominantly employ machine learn- ing and deep learning techniques to predict the helpfulness voting score using data as snapshot. Yin et al. (2021) proposed semi-supervised learning framework utilizing unlabeled reviews. Palahan (2023); Sharma et al. (2023)
To address biases and confounders in causal inference for recommendation studies, exposure bias widely exists due to self-selection through user exposure or ranking policy of the systems itself. Liang et al. (2016a;b); Wang et al. (2020b) model user exposure in recommendation as a latent variable
2

---

Counterfactual Voting Adjustment for Quality Assessment and Fairer Voting in Online Platforms with Helpfulness Evaluation
Bupunts
Watton
fm
Degrae Herding Blas
salesferes
doulding
medhnica
Abyder
JEA
MIDAA
Hembyyy
S gamin
Sexpmn
partiaas
nulsccovartry banch
Lnis nets" 10
Bith cockrethoog
Bur
mide sittuuttu woriouddng..eta
whw meth midlunei
2598 Mi
20 Sensitivity to Position Bias
2.4
3,5
Figure I: Analyses of different voting dynamics across the 120 largest communities on StackExchange. First, math-oriented communities such as mathoverflow..et and stats exhibit high conformity but are less influenced by trends due to the exact and provable nature of their knowledge content. Second, computer-related communities like reactjs(SOF) and askubuntu are particularly responsive to trending and most useful answers because they share up-to-date and practical experiences. Third, users in meta communities are minimally conforming and barely sensitive to trending information, rather featuring diverse opinion as purposed. Finally. politics and philosophy resemble religious communities with their low conformity but a high sensitivity to popular discussions, likely driven by their personal and societal beliefs. Interestingly, bitcoin emerges as a kind of new language that uniquely intersects gaming aspects.
and formalize the problem with empirical risk minimization methods to infer the value of exposure from data (Krauth et al., 2022; Mendler- Dünner et al., 2022). Schnabel al. (2016) followed a similar process but adopted Bayesian perspective. Both of them fitted the exposure model and action model separately. However, the explicit exposure data they used is hard to obtain.
approach for bias correction and propose an algorithm for a more faithful quality estimation.
Why take causal approach response quality estimation? The key observation is that the helpfulness votes of re- sponse are consequence of both the quality of the response and its other extrinsic factors (i.e., the response location and its existing vote distribution). In this sense, the helpfulness votes will be a more faithful quality estimator if we live in a fictitious world where all responses are placed at the same location (e.g., ranked the top) and share the same distribu- tion of existing votes (e.g., an even distribution of positive and negative votes). That is, for each response, we hope to infer how users would vote counterfactually if it were ranked at the top and has an even existing distribution. One can thus use this counterfactual vote estimate to mitigate the herding and position bias in response quality estimation.
Existing methods are unable to answer the two counterfac- tual questions introduced earlier using only observational data. Our CVA addresses this by combining causal and behavioral modules, as described in the next section.
3. Counterfactual Voting Adjustment
While helpfulness votes are often correlated with the qual- ity of responses, they suffer from both herding bias and position bias. The same response may receive different help- fulness votes if the voting user sees different distribution of existing votes; the votes may also differ even if the same response was placed at different positions. How can we miti- gate these biases? In this section, we take a causal inference
Setup and notations. We begin by describing the inter- active voting setup. In interactive voting, users vote on questions sequentially. At each time step, user decides whether to choose response to read and vote (or otherwise write a new response) and then decides to vote positively
3

---

Counterfactual Voting Adjustment for Quality Assessment and Fairer Voting in Online Platforms with Helpfulness Evaluation
Time
via some function f.
RM
M
R'
M
P(V = 11= Clg)
B'
where Gi (Bi,, Mi MB, D includes all other factors that drive the position bias and herding bias. We only con- sider the relative length of the responses Li as the main factor of Bi which is one of our limitations. Our frame- work can easily incorporate additional factors when they are observable in the future.
VM
D
E'
A'
Q
V
Regarding the sequential decision-making of the voting pro- cess, the current vote (V') and current summary of existing votes (R") will directly affect the next round M*+ at time + 1. Variables in red or blue frames are observable, while those in black frames are unobservable.
Figure 2: A causal graph for interactive voting that meets our assumptions. At each time step I, observable variables include each answer's is existing votes (M), summary statis- tic of M (R), displayed rank (D), and pre-reading content- related features (B). These affect exposure (unobservable E). If a user chooses to read the answer, post-reading features (unobservable A) are revealed. Together with B and other unobservable factors (U), they shape the perceived quality of the answer (Q), which drives whether it receives posi- tive or negative vote (V). Over time, current V and R update the future M.
Interactive voting as causal inference. To estimate the counterfactual votes for each response, we next frame the users' voting behavior as a causal problem at one certain time The counterfactual votes of interest describe "what would the votes of the responses be if it were displayed at different rank and/or received different previous votes." The treatment is thus the existing votes Mi, and its display rank Diy, and the outcome is the voting behavior (positive or negative vote). We denote the potential vote of response as V2 (gly) at time if its existing votes were mł, and its display rank were d' while the text stays as is, i.e. g (bz mlg. diy)-
or negatively on the chosen response. Both actions can be affected by the existing votes of each response and its displayed rank. More formally, at time t, for question i, user chooses a response from existing responses to work with as the exposure E (we assume that once a response is chosen, it must be read and voted); we denote the total number of existing responses as J!- This exposure is based on three types of observations. Observation is the existing votes, where we denote -a E f ") as the number of positive and negative votes for the jth response at time f. As a summary statistic of M, we fur- ther denote the perceived positive vote ratio at time t as Rh Observation 2 is the currently displayed rank of the responses, denoted as Di E {l...., J J. where D = means top rank. Observation are some celtent-related features before the responses being actually read which we denoted B E (1,...,Ji, }.
To perform causal inference for V (m, d), we rely key ignorability condition (Imbens & Rubin, 2015),
Vi (efy)) L My Digl
(I)
This assumption says that the only confounder, namely the variable that affects both the extrinsic factors of the responses (e.g., the display rank and the existing votes of responses) and the users' voting behavior, is the response's By itself. There are no other factors that can affect both simultancously. This is a common assumption in interactive voting in that a user makes a voting decision based on either the extrinsic factors M,j. Di or the response's ' Bay.
This ignorability assumption holds under, for example, the causal graph in Figure 2. (Many other causal graphs also satisfy this assumption.) At the time step t. a user decides what to vote next given presentational features before the response is read (B), previous votes (M), and currently dis- played rank (D) as input from the system. So M directly decides the presented votes summary R, such as positive vote ratio and vote difference. M also affects the displayed rank D at each time since the platform ranks the responses based on vote difference by default. The content-related presentational factor B is a confounder; it affects both the treatments M and D, but also the outcome V. Moreover, we assume that conditioning on B will block all backdoor
At time f, user chooses to work with Ed-th response. She either chooses to read and vote on the E = jthh response out of J total responses or chooses to write the new response J +1 1. she chooses to work with an existing response, then she can perceive some other content-related features after the response being read, denoted as Aly. This A', along with Bi, and other unobservable but independent features U, will affect the perceived quality of the chosen response at this time (Q3)) by the user. Finally, this user decides to vote positively on the chosen E = jthh response with a probability that depends on the perceived quality of the chosen response at this time Qi and other factors G2
4

---

Counterfactual Voting Adjustment for Quality Assessment and Fairer Voting in Online Platforms with Helpfulness Evaluation
paths (Pearl, 2010), hence satisfying the ignorability as- sumption Equation (1) which is one of our limitations. Wip- ing out the effect of M and D on the voting behavior V makes V reflect the true response quality Q.
the extent of the position bias effect: B3 close to 0 implies insensitivity position bias.
To quantify the herding bias, we estimate the odds between the probability of agreeing with the majority opinion and the probability of disagreeing with the majority opinion via geometric mean across all time steps and for all ques- tions. Further details about the computation of the degree of herding bias are in Appendix B.
Given this ignorability condition, together with other stan- dard assumptions in causal inference (namely, positivity and stable unit treatment value assumption (Imbens & Rubin, 2015), one is greenlighted to estimate the counterfactual votes using observational data,
4. Preliminary Experiments
(())(()))BB P(V IM,, = mly. Dij = dig. Bly),
While the Chinese Voting Process (CVP) accounts for posi- tion and herding biases in its generative process (Lee et al., 2016), it lacks an explicit mechanism to mitigate position bias, thereby learning its model parameters for position and herding biases independently. In contrast, our Coun- terfactual Voting Adjustment (CVA) mitigates both biases by leveraging voting trajectories and backdoor adjustment, jointly learning its model parameters without any indepen- dence assumption. This section demonstrates how CVA contextualizes individual votes to infer information quality under each bias.
where ViJ. Mi Di Ri B are all observable user behav- iors and response features.
To estimate the counterfactual vote estimates, we posit model of positive voting, following Lee et al. (2016):
V Gij Bern + + 3(
1
1 D (2)
for all j.j.t, L, based on the observational data. We fit the model by maximizing the ligeliill with 12- regularization:
4.1. Mitigating the position bias
T max log
II 2-2
1 logit (aly +AR tu, Lig +1( 1 + D 2
1)
10
I.
To alleviate position bias, votes cast at higher ranks (easy votes) should be down-weighted, while those at lower ranks (hard votes) should be up- up-weighted. Consider the position bias examples la and 1b in Table 1. We assume that two an- swers exist for the same question: answer A is consistently ranked higher than answer B. In 1 la, both answers receive 3 positive votes, but since Answer A gains these votes ear- lier, it remains ranked higher than Answer B. Their voting sequences are identical, meaning that herding bias is fixed, Their key difference is that all votes for A occur at rank 1, whereas those for B occur at rank 2. Since votes at lower ranks are harder to obtain, up-weighting their importance causes Answer B to be evaluated more favorably than An- swer A. The symmetric example 1b replaces positive votes with negative ones, leading to Answer B being assessed more negatively than Answer A.
where 0- = HHgh,,, X, (mi}, 3). After training, we are able to get community-leve parameters, including the coeffi- cients/ À and 3; and question level parameter ;y; and answer level parameter- the response quality of each answer d/y- We finally produce the response quality estimate:
Φiy Σ EVV ( x P(mllg tiy ddn
where gy = (B) miy. dty).
This estimate performs counterfactual voting adjustment since it integrates out the existing vote distribution mi, and display rank d' - out their impacts on the quality estimate-while fixing the response text. This estimator is similar to the backdoor adjustment estimator (or regres- sion adjustment estimator) (Pearl, 2010). It turns out this estimator is the optimal estimator that satisfies counterfac- tual invariance to existing voting distribution and display rank while minimizing the KL divergence to the original votes under the product distribution. (We expand on this discussion in Appendix A).
We learn o and of by training them together. Following the convention of CVP, we dropped the first vote of each answer from the training data as it is an arbitrary vote given no context in our setting. So, the quality of each answer at the time of its first vote cannot be estimated. We trained 4 times using the data before T each time. Because this toy example not real, there's no relative length data involved. The result of the first toy example for position bias is in (a) of Figure 3. Although the total vote of both answers is 3, at the end time, the quality of answer B (0.486) is greater than the quality of answer A (0.466). The result of the second toy example for position bias is in the (b) of Figure 3.
Measurement of position bias and herding bias. Given the learned parameters for each community, we can quantify the sensitivities to position and herding bias. The displayed- 1 rank related term in the model is B( (4bb)) So 8 reflects
5

---

Counterfactual Voting Adjustment for Quality Assessment and Fairer Voting in Online Platforms with Helpfulness Evaluation
Table I: Preliminary Experiment Settings: Examples la and lb assume two answers of a question where A ranks above B. Both receive identical sequences of three positive (1a) or negative (lb) votes. Example 2 assumes two equally ranked answers of a question having the same vote score (0) but different vote sequences.
Example
la: for position bias
Example lb: for position bias
Example 2: for herding bias
Time
2
4
5
1
2
3
4
Time
2
3
5
Answer (rank 1)
+
+
+
Answer
+
Answer (rank
+
+
+
+
0.549
-03
0.354
-32
3.49 467
0.308
0.600
2011
.0204
-0 353
33
1.
1-1
1-3
1-2
1-3
6-5
(a) Learndd quality over time for Example la
(b) Leared quality over time for Example 1b
(c) Leamed quality over time for Example 2
Figure 3: Preliminary Experiment Results: Subfigures (a) and (b) demonstrate that CVA mitigates position bias by upweighting the votes received at a lower-ranked position. By time step 6, the learned quality of answer B is more positive than that of answer in (a), and answer B's learned quality is more negative than answer A's in (b). Subfigure (c) illustrates that CVA mitigates Herding bias by upweighting votes that go against the prevailing majority opinion. At time step the learned quality of answer A is lower than that of answer B.
Although the total vote of both answers is -3, the quality of answer B (-0.486) is more negative than the quality of answer A (-0.467) at the end time. These results satisfy our expectations, and the position bias has been mitigated.
alternating votes but remains less negative than Answer A. At the final timestamp, Answer A (-0.146) is more negative than Answer B (-0.118), confirming that herding bias has been mitigated.
4.2. Mitigating the herding bias
5. Experimental Results
To reduce herding bias, votes that follow the majority should be down-weighted, while those opposing the major- ity should be we-weighted. Consider Example for herding bias in Table 1, where two answers are displayed at the same rank. Answer A receives three consecutive positive votes followed by three negative votes, while Answer B gains alternating votes. Although both answers have a net vote difference score of zero, their vote trajectories differ, imply- ing different quality assessments. For Answer A, its second and third positive votes reinforce the majority and should be down-we weiihwe,g while the subsequent negative votes counter the prevailing trend and should be up-weighted, making its final quality more negative. Answer B, experiencing less dominant majority effect, should have a slightly higher qual- ity than Answer A.
In our study, introduce and implement our CVA algo- rithm. After training the model with a customized neural net- work, we performed rigorous testing on both semi-synthetic dataset and the real StackExchagne dataset, which has been an anonymized compilation of all user-contributed content across the StackExchangg network since its launch in 2010. We adopt the dataset version published on October 5, 2022. The network consists of 180 Q&A communities and their corresponding META forums, where users discuss policies and logistics pertaining to their respective communities. The preprocessing details are in Appendix C. After preprocess- ing, we do the experiments on the largest 120 communities. The statisties of 8 representative communities of StackEx- change are in Table 2. "reactjs(SOF)" is sub-community of StackOverflow about "reacjs."
We fix the rank of answers A and B both as rank to learn LA and of separately. We also omitted the first vote of each answer due to its arbitrariness. As shown in Figure 3(c), Answer A's quality initially rises with positive votes but drops sharply with the first negative vote, reflecting strong shift against the majority. Answer B 's quality fluctuates with
5.1. Semi-synthetic experiment
We generate a semi-synthetic dataset using the proposed model, with detailed data generation steps provided in Ap- pendix D. Using the dataset's 's ground truth qualities, we
6

---

Counterfactual Voting Adjustment for Quality Assessment and Fairer Voting in Online Platforms with Helpfulness Evaluation
Table 2: Result Summary of 8 Representative Communities: In most cases, CVA's quality-based ranking aligns more closely with the true quality ranking than CVP or vote dite-difeeeence ranking, as measured by Kendall's Tau rank correlation (KT, higher is better) and Sum of Squared Residuals (Res, lower is better).
Statistic
Semi-Synthheicc
Real (commmet sentiment)
Real (GPTI44 helpfulness)
Comun
#Questions
#Answers
#Voes
#Comments
vonsDiif
CVP
CVA
vose
CVP
CVA
voteDiff
CVP
CM
KT
0,3035
0,9089
0.4106
0,0383
0.1301
0.1524
0.0805
0.1575
0,1819
2,445
10.301
212,015
15,607
Res
17015
13458
6716
7514
6651
0.2620
0.3603
0.3691
0,0950
0.1180
0,0580
0.1116
101,799
57,581
Res
9337
9098
6245
6222
6122
5753
KT
0.2774
0.3750
0.3902
0.1034
0.0579
0.1160
0.1055
0.0076
0.1495
mathoverlow
998
8,733
119,383
49,002
Res
10438
8232
8024
8280
8528
7940
8408
8961
7679
KT
0.3909
0,0640
0.4659
0,0788
0.1275
0.1317
0.1461
0.1936
0.2037
eoheory
325
2,460
33,098
3,808
Res 2306
1924
1892
2032
1872
1856
1936
1766
1736
KT
0,3779
0.4883
0.4880
0,0255
0,1470
0.1748
0.0824
0.2138
0.2573
politics
1,161
5,029
113,378
31,121
Reh 7118
5485
5359
969]
8102
7775
9114
7463
6865
KT
0,3484
0,3473
0.3567
0,0668
0,07768
0.0898
0.1475
0.1530
0.1654
philosophy
1,500
5.991
39,079
30.451
Res 6822
6643
6566
62.90
6009
6067
5622
5813
5652
KT
0.4469
0.5521
0.5360
0.1082
0.1356
0.1411
0.1259
0.1538
0.1775
mathmetta
271
2.59
46,158
13,056
Res 2253
1860
1866
3831
3465
3499"
3764
3466
3425
KT
0,4837
0.5633
0.5640
0.1664
0.1075
0.1070
0.1797
0.1509
0,1567
codegolf
173
2,878
38,421
13,713
Riss
1675
1671
3453
3450
3430
3396
significance level of CVA better than vote
1Ip 0.05)
"P 0.00)
(p: (1.011)
1or 0.0001)
2
2
3600 Ceneenn rani2scora
vota diff rankzscore
1
CVA rankZscore
1
0
-1
-1
re51d
residual
Asiidaal53599.
i
-2
quality rankZscore
quality rankZscore
true quality rankZscore
Figure 4: Validation on semi-synthetic data (Politics community):: Dots represent answers. X-axis: true quality rank; Y-axis: rank by vote difference score (left), CVP learned quality (middle), or CVA learned quality (right). Closer alignment to the diagonal indicates better performance CVA shows the strongest alignment.
assess whether our model can recover a ranking closely aligned with the ground truth quality-basedd ranking.
system vote difference- ranking and the CVP-learned quality ranking. Additional example community plots provided in Appendix E.
Residual to diagonal line comparison. For each response (answer), we rank based on system vote difference scores, model-icarned quality, and predefined true quality, then normalize ranks into z-scores for comparability across ques- tions. better ranking shows a stronger correlation with the true ranking. We visualize answers as scatter points, with the x-axis representing true rank z-scores and the y-axis representing the model's 's rank z-scores. The association is measured using the sum of squared residuals from the diag- onal line (x = y), as the ideal ranking aligns perfectly with the true ranking. The results for the synthetic data of the politics community are shown in Figure 4. A smaller resid- ual to the diagonal line indicates better performance. Our proposed model achieves the lowest sum of squared residu- als, demonstrating significant improvement over both the
Rank Correlation comparison. For each question, we have rankings of its answers using different models. We compute the Kendall's т coefficient (rank correlation) between a model's ranking and the true ranking. Then, average over questions. The results for synthetic data of 8 representative communities are shown in Table 2. Our model outperforms both the system vote difference ranking algorithm (voteDiff) and CVP for all 8 communities.
5.2. Real experiment
Since the ground truth qualities of StackExchange answers are unknown, we use two proxies. The first is comment sentiment, as writing comments requires more effort than
7

---

Counterfactual Voting Adjustment for Quality Assessment and Fairer Voting in Online Platforms with Helpfulness Evaluation
rankZscor
quality raniZscooe
voto dir rankzscare
1
1
0
re51
1151
residual-7775
-2
-2
-2
-2
-2
-1
i
sentiment rank2score
sentiment ran eankooe
sentiment ran a2cooee
2
rankZscore
CUA CVA quality rank-sccre
vote dITY ranaanaaacc
1
1
0
0
-1
-1
residual-0114.5
residual-7462.7
residual-6864.6
-2
2
helptuiness rank2score
helptuiness rank2score
helpfuiness rank2score
Figure 5: Validation on real data (Politics community): Dots represent answers. X-axis (first row): comment sentiment as
true quality; X-axis (second row): GPT estimated helpfulness score as true quality; Y-axis: similar to Figure 4. CVA (right) shows the best alignment to the diagonal line with the least residuals.
voting and reflects independent user opinions, making it less biased and closer to true quality. Additionally, Stack- Exhange hired moderators to review the comments and hide trivial comments since 2016. The quality of the com- ments is reliable. The second is a helpfulness evaluation by GPT-4o, which assesses every single answer in zero- shot manner given its questions and comments. Previous study (Kamalloo & Rafici, 2023) found that GPT-4 achieves state-of-the-arr art on an established benchmark NQ-OPEN for open-domain question answering tasks, and zero-shot prompting method can be a reasonable substitute for human evaluation. Given GPT-4o's advancements, we consider it reliable proxy for ground truth. The prompt example for GPT-4o evaluation is in Appendix F.
results for real data from the politics community are in Figure 5 (second row). Our model achieves the lowest sum of squared residuals (6865) compared to CVP (7463) and vofeDiff (9115). Across all representative communities except codegolf.meta's 's KT and philosophy's 's Res, our model performs the best. Additional plots are in Appendix G.
Community-level winrate. As shown in Table 3, among the largest 120 communities, there are 94 (78.3%) com- munities whose CVA ranking has a smaller residual than both system vote difference-based ranking and CVP quality- based ranking; and there are 89 (74.2%) communities whose CVA ranking has higher KT than both system vote differ- ence based ranking and CVP quality based ranking when using sentiment of comments ground truth. There 93 (77.5%) communities' residuals performing the best, and there are 90 (75%) communities' KT performing the best when using the helpfulness score evaluated by GPT-4o as ground truth.
Similarly, as for semi-synthetic experiments, we demon- strate our CVA's advantage with less residual to the diagonal line and higher Kendall rank correlation with true ranking.
Comment Sentiment as ground truth. The results for real data from the polities community are shown in Figure (first row), with different y-axes: vote difference-based ranking (left), CVP-leamed quality (middle), and CVA-learned qual- ity (right). Our model achieves the lowest sum of squared residuals (7776) compared to CVP (8102) and vifeDiff (9691). Across all 8 representative communities except math.meta's Res, our model performs best as shown in Ta- ble 2. A possible reason that some meta communities suffer much fewer position biases, so the proposed CVA didn't show its advantage over CVP, but still better than vote difference score. Kendall's 7 rank correlation results in Table 2 also show that CVA consistently achieves the highest correlation across all 8 communities.
Table 3: Three types of win rates for the largest 120 com- munities with real data. The entries are the percentages of communities showing that CVA is better than vote differ- ence score (left), CVA is better than CVP (middle), and CVA is better than both (right)
CVA better than
CVA better than
CVA better than
CVP
84.17%
90.00%
78.33 %
sentimen: comments trath
79.17%
86.67%
74.17% 17%
helpfulness score of GPT
81.67%
94.17%
77.50%
ground truch
80.83%
91.67%
75.0 %
GPT-4o evaluated Helpfulness as ground fruth. The
A Advantages over CVP. While the proposed CVVA does not
8

---

Counterfactual Voting Adjustment for Quality Assessment and Fairer Voting in Online Platforms with Helpfulness Evaluation
always significantly outperform CVP, it offers three key ad- vantages. Firstly, CVP paper didn't 't provide causal effect estimation. Secondly, CVA is able to mitigate position and herding bias at the same time, while CVP can't. 't. Thirdly, CVA is better than CVP in evaluations, especially for the communities with high position bias, since CVP doesn't con- sider the rank position when predicting the vote. For those communities with much less position bias, the advantage of CVA won't be shown significantly.
Vote Probuu
rank
Figure 6: Predict positive vote probabilities given different vote ratios and ranks (politics community). Probabilities are higher when the majority opinion is positive (red) than neutral (blue) or negative (green), and decrease with lower ranks.
6. Analysis and Discussion
In this section, we extensively apply our framework to an- alyze user voting behavior across 120 communities for broader generalization. Additionally, we present case study illustrating how our CVA framework addresses the two key counterfactual questions.
of b also reflects the position bias. Figure 6 is for politics community. See other community figures in Appendix H.
6.1. Cross-community analysis
We train the largest 120 communities in the StackExchange with the data till October 2022. For 17 communities with enormous data sizes like StackOverflow and MathOverflow, we randomly sampled partial questions due to the limitation of our computational resources. With the learned parame- ters in Section 3, we compute the sensitivities to position bias and the degrees of herding bias separately for each community and map these two behavioral coefficients into the 2D map as shown in Figure 1. The MEDIAN point is the median value of the measurements of two biases. It divides the map into 4 quadrants. The communities in sim- ilar regions share similar voting behaviors and tendencies. Detailed analyses can be found in the caption of Figure 1.
1 p(V+)= = rank + + c
(3)
7. Conclusion
Helpfulness votes are often less accurate due to position bias and herding bias. Our CVA framework mitigates these bi- ases by accounting for the sequence of votes and underlying causal effects, enabling fairer estimation of response qual- ity. Compared to system vote score-basee rankings and the CVP model, CVA 's quality-based rankings show stronger alignment with true quality proxies: comment sentiments and GPT-4o helpfulness evaluations.
6.2. Revisiting the key counterfactual questions
This study offers several managerial insights. Platforms can improve answer rankings by recovering true quality, turning high-quality question- pairs into valuable knowledge assets. Better rankings reduce users' effort in finding useful content and help recognize contributors whose answers are underappreciated by biased votes.
To answer "What if the information were presented at dif- ferent ranks?", we compute a bunch of probabilities of re- ceiving a positive vote (P(V-)) )) given different ranks using Equation (2), since we learned the quality and coefficients. The descending speed of P(V+) from top rank lower rank reflects the position bias. Similarly, to answer "What if the information had received even votes?", we force the vote ratio feature as even votes (previous positive vote count negative vote count) to show a non-herding biased P(V+) for each rank (blue bars). For any response at a certain rank, the P(V+) in the positive mood (previous positive votes are more than negative votes as in red bars) should be higher than that in the neutral mood (blue bars). Vice versa, the P(V+) in the negative mood (green bars) should be lower than that in the neutral mood. These differences reflect the herding bias. We average the P(V+) across all the answers in a community. 'b ro.rell VoteRaaio VoteoeaaiiooppsMMooo posMood', re.rralloool VoteR- atio negMood' and 'b even VoteRatio" are fitted parameters of simplified pawer-law function as Equation (3). The value
CVA is adaptable to other platforms, such as product or service review sites, as long as vote history is available. The framework can also integrate richer presentational features, such as images, code, or author reputation.
This research has limitations, mainly due to assumptions made for backdoor adjustment. We assume that perceived answer quality affects only the next vote, not prior ones, and that all relevant confounders are included. Cases where answers were read but not voted on are excluded due to unavailable reading data. However, the framework can be extended to address these issues as more data becomes avail- able. Future work will explore these aspects and aim to bridge the gap between GPT-4o and human evaluation.

---

Counterfactual Voting Adjustment for Quality Assessment and Fairer Voting in Online Platforms with Helpfulness Evaluation
Impact Statement
Imbens, G. W. and Rubin, D. B. Causal inference in stafis- tics, social, and biomedical sciences. Cambridge Univer- sity Press, 2015.
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
Joachims, T., Granka, L., Pan, B., Hembrooke, H., Radlin- ski, F., and Gay, G. Evaluating the accuracy of im- plicit feedback from clicks and query reformulations in web search. ACM Transactions on Information Systems (TOIS), 25(2):7-es, 2007.
Acknowledgments
YW was supported in part by the Office of Naval Re- search under grant 100014-23-1-1
the National Science Foundation under grant No. 2231174, No. 2310831, No. 2428059, No. 2435696, No. 2440954, and Michigan Institute for Data Science Propelling Original Data Science (PODS) grant.
Kamalloo, Dziri, C. and Rafiii. Evaluating open-domain question answering in the era of large language mod- cls. Association for Computational Linguistics, 1(Long Papers):5591-5660 2023.
Kastrati, Z., Fatima, S., Kurti, A., Daudpota, M., and Imran, A. S. Analyzing and predicting the helpfulness of reviews in moocs context using deep learning. Procedia
References
Computer Science, 246:772-781, 2024.
All Sites Stack Exchange stackexchange.com. https: //stackexchange com/ sites. [Accessed 03-06- 2025].
Krauth, K., Wang, Y., and Jordan, M. Breaking feedback loops in recommender systems with causal inference. ACM Transactions Recommender Systems, 2022.
Alzate Barricarte, M., Arce Urriza, M. and Cebol- lada Calvo, J. Is review visibility fostering helpful votes? the role of review rank and review characteristics in the adoption of information. Computers in Human Behavior, 2024.
Kuan, K. K., Hui, K.-L.,, Prasarmphanich, P., and Lai, H.-Y. What makes a review voted? an empirical investigation of review voting in online review systems. Journal of the Association for Information Systems, 16(1):1, 2015.
Babić Rosario, A., Sotgiu, F., De Valck, K., and Bijmolt, T. H. The effect of electronic word of mouth sales: A meta-analytic review of platform. product, and metric factors. Journal of Marketing Research, 53(3):297-318, 2016.
Lee, M., Jin, S. H., and Mimno, D. Beyond exchangeabil- ity: The chinese voting process. Advances in Neural Information Processing Systems, 29:4934 4944,494222 2016.
Lee, M., Kwon, W., and Back, K.-J. Artificial intelligence for hospitality big data analytics: developing a predic- tion model of restaurant review helpfulness for customer decision decision-making. International Journal of Contemporary Hospitality Management, 33(6):2117-211 2021.
Chiang, C.-H. and Lee, H.-y. Can large language models be an alternative to human evaluations? arXiv preprint arXiv:2305.01997, 2023.
Deng, W. Yi, M., and Lu, Y. Vote or not? how various infor- mation cues affect helpfulness voting of online reviews. Online Information Review, 44(4):787-805 2020.
Liang, D., Charlin, L.,, and Blei, D. M. Causal inference for recommendation. In Causation: Foundation to Applica- tion, Workshop UAI AUAI, 2016a.
Dev, H., Karahalios, K., and Sundaram, H. Quantifying voter biases online platforms: An instrumental vari- able approach. ACM on Human-Computer Interaction, 3 (CSCW):1-27, 2019.
Liang, D., Charlin, L., McInerney, J., and Blei, D. M. Mod- eling user exposure in recommendation. In International Conference on World Wide Web, pp. 951-961, 2016b.
Liu, J. Cao, Y., Lin, C.-Y,, Huang, Y., and Zhou, M. Low-quality product review detection in opinion sum- marization. In Joint Conference on Empirical Methods in Natural Language Processing and Computational Natu- ral Language Learning (EMNLP--CNILL)) pp. 334-342, 2007.
Du, J., Rong, J., Wang, H., and Zhang, Y. Neighbor-aware review helpfulness prediction. Decision Support Systems, 2021.
Guo, D., Zhao, Y., Zhang, L.,, Wen, X., and Yin, C. Confor- mity feedback in an online review helpfulness evaluation task leads to less negative feedback-related negativity amplitudes and more positive p300 amplitudes. Journal of Neuroscience, Psychology, and Economics, 12(2):73, 2019.
Mendler-Dünner, C., Ding, F., and Wang, Y. Anticipating performativity by predicting from predictions. Advances in Neural Information Processing Systems, 35:31171- 31 31185, 2022.
10

---

Counterfactual Voting Adjustment for Quality Assessment and Fairer Voting in Online Platforms with Helpfulness Evaluation
Mitra, S. and Jenamani, M. Helpfulness of online consumer reviews: A pusteepers approach. Information Pro- cessing & Management, 5803)102533, 2021.
Wang, Y., Liang, D., Charlin, L., and Blei, D. M. Causal inference for recommender systems. In ACM Conference Recommender Systems, pp. 426-431, 2020b.
Murphy, R. Local consumer review survey 2017, Jun 2017. /earn// Local-come www.-ooonuuuueeee www.ewewww... ooraa--ooossseerrrrrreee-e---s suweeyy-wws
Wang. Y., Sridhar, D., and Blei, Adjusting machine learn- ing decisions for equal opportunity and counterfactual fairness. Transactions on Machine Learning Research, 2023.
URL BRRLattpsss/www.biihhh.... cow.aaa..oomaaaaaaaaeeeeeee c/
Palahan, S. Comparative analysis of deep learning models for predicting online review helpfulness. In Asia Confer- ence on Computer Vision, Image Processing and Pattern Recognition, pp. 1-5, 2023.
Wang. Z., Chang. V., and Horvath, G. Explaining and predicting helpfulness and funniness of online reviews the steam platform. Journal Global Information Management (JGIM), 29(6):1-23, 2021.
Pearl, J. Causal inference. Causality; objectives and assess- ment, pp. 39-58, 2010.
Wu, C.,, Mai, F,, and Li, The effect of content depth and deviation on online review helpfulness: Evidence from double-hurdle model. Information & Management, 58(2): 103408, 2021.
Risselada, H., Vries, I.., and Verstappen, M. The impact of social influence the perceived helpfulness of online consumer reviews. European Journal of Marketing, 2018.
Yin, H., Hu, Z., Peng, Y., Wang, Z., Xu, G., and Xu, Deep semi-supervised learning with weight map for re- view helpfulness prediction. Computer Science and In- formation Systems, 18(4):1159-111 1174, 2021.
Schnabel, T., Swaminathan, A., Singh, A., Chandak, N., and Joachims, T. Recommendations as treatments: Debiasing learning and evaluation. In International Conference on Machine Learning, pp. 1670-1679, PMLR, 2016.
Zhou, S. and Guo, B. The order effect on online review helpfulness: A social influence perspective. Decision Support Systems, 93:77-87, 2017.
Sharma, S. P., Singh, L., and Tiwari, R. Prediction of cus- tomer review's 's helpfulness based on feature engineering driven deep learning model. International Journal of Software Innovation (IJSI), 11(1):1-16, 2023.
Sipos, R., Ghosh, A., and Joachims, T. Was this review helpful to you? it depends! context and voting patterns in online content. In International Conference on World Wide Web, pp. 337-348, 2014.
Tay, W., Zhang, X., and Karimi, S. Beyond mean rating: Probabilistic aggregation of star ratings based on helpful- ness. Journal of the Association for Information Science and Technology. 71 7999 2020.
Tseng, S.-L., Lu, S., Weathers, D., and Grover, V. How product review voting is influenced by existing votes, consumer involvement, review valence, and review diag- nosticity. Decision Support Systems, 172: 13381, 2023.
Wan, The matthew effect in social commerce: the case of online review helpfulness. Electronic Markets, 25; 313-324, 2015.
Wang. G., Gill, K., Mohanlal, M., Zheng, H., and Zhao, B. Y. Wisdom in the social crowd: an analysis of quora. International Conference on World Wide Web, pp. 1341- 1352, 2013.
Wang, J.-N., Du, J., and Chiu, Y.L.. Can online user re- views be more helpful? evaluating and improving ranking approaches. Information & Management, 10388,, 2020a.

---

Counterfactual Voting Adjustment for Quality Assessment and Fairer Voting in Online Platforms with Helpfulness Evaluation
A. CVA: Inferring response quality from helpfulness votes
Given the counterfactual vote estimate, how can we form an estimate of the response quality? One criterion that the predicted response quality Qiy (0ly) be invariant to the extrinsic factors:
(igg - in
in (.
(4)
where _ indicates equal in distribution. Moreover, the predicted response quality shall recover the users' voting behavior as much as possible.
To this end, we find the optimal prediction that is closest to the mean of ViJ. (dty) while satisfying counterfactual invariance:
T
B (i dy dd.
(5)
where Ø = (Bij mly. d.)). P(m dzp) is the population distribution of the existing votes and display rank at time t for the jthh response across all questions. The optimality of predicted response quality is due to Theorem of (Wang et al., 2023), while following the same distribution what presentation configurations m,y dig the response was assigned to. Notation-wise, T' is the total number of voting actions, so Qi calculates the total sum of average potential positive votes if the users were shown random configurations of past voting history and at random display rank.
Finally, one can estimate potential voting behavior Qi following the fitted parametric voting behavior model:
/ EVV ddnt Ft, Ji, (sigmoid( + ARR + v, + 3(
1
))}.
D
where one plugs in the estimate of 2, population distribution of the voting trajectories at time t for the jth response). This is the CVA estimate.
13,,0, Vi, Z. from parametric model fit. The expectation is taken over all items (i.e., the
B. Computation of the degree of herding bias
As given in Section 3, the sensitivity to herding bias can be computed as follows:
pft.
(ugrec. 11/m
degree of herding bias
(9)
p(t) (against majority vote) 1
The annotation i in the equation indicates a question, and the time tick has a subscript because it's a relative time associated with each question. n is the total number of votes. The ratios of agree majority vote probability over against majority vote probability at all time ticks for all questions are summarized by geometric mean. To generalize this equation using the probability of vote P(vote+) at time tick and the probability of down-vote P(vote-) a time tick, the degree of herding bias can be computed as below:
P ++1 ld X², vi, 34) yolt))a where (4))) P +1 Olat,, M: v/, 3r)
[1 when nij (st.) > n
+(da) > nij (sts)
+ +(2))
(10)
-1 when mij
The annotation +(0) means the number of positive votes (up-vote) given to the j'h answer of question before time tick /; ((4) is the number of negative votes (down-vote) given to the jth answer of question before time 4.
nij
and nij
The dynamic probabilities of uptvote and down-vote overtime from the whole community view are acquired as below:
" D 170) ))
P(u lag X', "(, = sigmoid (iyy + RR, + vi + 3'3( (
(11)
12

---

Counterfactual Voting Adjustment for Quality Assessment and Fairer Voting in Online Platforms with Helpfulness Evaluation
C. Data pre-processing
For real data experiments, we extract data from Posts, Votes, and PostHistory for each community. Among 8 types of Posts, we primarily focus on two main types: "Question" and "Answer". For each community, we extract all the questions and compile them chronologically with respect their creation time. We exclude the questions that have ever been closed locked because closed questions no longer answerable, and locked questions cannot be modified. Then, for each question, we gather all corresponding answers in their chronological order of creation. But we filter out questions with less than 5 answers, aside from the answer accepted by the question author.
For each question, we then organize all votes cast for its answers according to the timestamp of each vote, assigning each vote relative time index to facilitate cross-question comparisons and integrations. This processing results in the reconstruction of the matrix for each question, whose dimensions are the number of answers by the length of time. The matrix entry the vote: up-vote as 1, down-vote -.. We remove all the votes given to the accepted answers after the answers get accepted. This is because the user interface of StackExchange with accepted answer has changed multiple times. In addition, the fixed top placement of the accepted answer could inject noisy information. Further, an answer is accepted by the question's 's author when they feel it is good enough. However, It's not necessarily the best answer all the time. Finally, we discard communities with fewer than 100 questions.
D. Semi-synthetic data generation steps
We generate the same number of questions the real data and the same number of events, including answer creation and vote events. At each simulating time, we first choose question according to the distribution of event counts of each question in real data. Secondly, we decide whether to create new answer or vote based on the Chinese Restaurant Process (CRP) of one parameter case, and the parameter computed based on the real data. Next, if choosing to create a new answer, sample a true quality from a normal distribution and use the same relative length as in real data. choosing to vote, we select an existing answer according to the probability, which is the inverse the displayed rank. At last, we decide to vote positive or negative using the proposed model and the learned coefficients from real data.
13

---

Counterfactual Voting Adjustment for Quality Assessment and Fairer Voting in Online Platforms with Helpfulness Evaluation
E. Other example community plots to validate the learned quality on semi-synthetic data
vote air rankzzcom
CVP raniZ
1
CVA rankZscare
1
0
0
residual=17014.
re345uel=3458
residual=13633
-2 -2
-2
-2
2
-2
22
-1
1
true quality rankZscore
true quality rankZscore
true quality rankZscore
(a) reactjs(SOF)
2
vota dift raniz.core
CVA rankZscore
1
-1
-1
-1
residual= 10438.
CVP
residual 6231
residual 8024.4 8024
-2
-2
-2
-2
-2
-1
quality
quality rankZscore
quality
(b) mathoverflow
rankZscore
2
vote dIr rankzscars
1
CVA rankksscore
1
-2
-2
-2
-2
-2
-1
true quality rankZscore
true quality rankZscore
true quality sanke
(c) codegolf.mett
2
CVP qualityyytiiaa. annkszara
vota dIT ranizccre
CVA ranicsccore
Avd
0
-1
-1
-1
1058du
r3Sidual-937..7
residual- 9098.9
i
2
2
2
quality rankZscore
quality
quality
(d) askubuntu
2
CUP quality mankZscore
vate dit manzzcooa
1
CVA rankzscore
1
-1
residual=23055
residual=1923.0
residual- 1891.6
-2
22
-2
-2
-1
true quality rankZscore
true quality rankZscore
true quality rankzscore
(e) estheory
14

---

Counterfactual Voting Adjustment for Quality Assessment and Fairer Voting in Online Platforms with Helpfulness Evaluation
CVP quality rani2
voto dir rankzscors
1
CNA rankisscore
1
0
residual-1899.
1es6dual-1866.5
-2
-2
-2
-2 -1
2
-2
-2
-1
1
true quality rankZscore
true quality rankZscore
true quality rankzscore
math
2
2
vote dmf rankz.com
simenn rankZacors
1
CVA raniZiscore
0
-1
-1
-1
6852
residual 6643..6
residual 6566.0
656660
-2
-2
-2
-2
-1
quality rankZscore
quality
quality
(g) philosophy
F. Prompt template for GPT-4o
For each answer, we ask GPT-4o with the following prompt:
Given the following [QUESTION] and [ANSWER] with 4 COMMENTS from [COMMENT-I] [COMMENT-4], Do the following tasks.
(1) Estimate sentiment score from -1 to for each comment. as the most positive and as the most negative. Response in new line only with the sentiment scores and separate them with commas, such as "0.1,-0.3.0.9" (2) Estimate score from -1 1 for the [ANSWER] about how helpful it is to the [QUESTION] considering all the COMMENTS. the most helpful and -1 the most non-helpful. Response in line only with the helpfulness score, such "0.5".
*[QUESTION]:
Can someone suggest good book for teaching myself about Lie groups? study algebraic geometry and commutative algebra, and like lots of examples. Thanks.
*[ANSWER]:
I like Humphreys' book, Introduction to Lie Algebras and Representation Theory, which is short and sweet, but doesn't 't really talk about Lie groups (just Lie algebras). I also sometimes find myself looking through Knapp's Lie Groups: Beyond an Introduction. If the material was covered in the Spring 2006 Lie groups course at Berkeley, then prefer the presentation this guy's notes.
"[COMMENT-I]:
Before the 2006 course, there was Allen Knutson's 2001 course, from which there are several sets of notes, e.g. URL
*[COMMENT-2]:
Also, Theo Johnson- -reyd has some notes from Mark Haiman's 's Fall 2008 course here: URL
15

---

Counterfactual Voting Adjustment for Quality Assessment and Fairer Voting in Online Platforms with Helpfulness Evaluation
*[COMMENT-3]:
Finding reasonably elementary book on Lie groups with lots of examples is challenging. What makes the sub- ject attractive is that it's the crossroads for many subjects. My book definitely wasn't about Lie groups (and has too few examples) but does get somewhat into "modern" representation theory. Knapp is reliable but somewhat advanced. Fulton-Harris is also not a Lie group book and doesn't t introduce infinite dimensional representations, but cov- ers lot of concrete classical examples plus symmetric groups. Free online notes can safe starting point, but shop around.
*[COMMENT-4]: @ Anton You and Theo should be both be very proud of your TeX-ed notes, particularly Lie theory.
G. Other example community plots to validate the learned quality on real data
2
rankZscare
CMA qualltt rankZscore
Vosa dM mani2score
1
-1
-1
-1
residual-7935
-7516
residual 6508.6
-2
-2
-2
-2
sentiment rankZscore
sentmeent kZscore
sentiment ancoee
2-
2
vota ditt c25cor
CVP rankZscare
CUA quality rankZsccore
1
1
-1
-1
-1
residual=7514.2
residual=6551.4
residual 6299..
-2
-2
-2
helpfuiness rankZscore
helpfulness rank2score
helpfulness rankzscore
(a) reactjs(SOF)
CVP quality mniizscore
CVA quality rankksscore
voze dIr rankzscors
1
1
0
0
-1
8258
residual-8578
r
-2
-2
-2
-2
-2
-2
-1
sentiment rankZscore
sentiment ran ankoZoore
sentiment ran a2core
2
rankZscore
CVA quality rankZscare
vate dIf ranzzccree
1
1
-1
residual-8408-
residual=8961..
residual=7679,
-2
-2
-2
-2
helptuiness ankZscore
helpruiness rankZscore
helptuiness rankzscore
mathoverflow
16

---

Counterfactual Voting Adjustment for Quality Assessment and Fairer Voting in Online Platforms with Helpfulness Evaluation
rankZscom
CUA quality raniZsccoe
vodo dir ranisscars
1
1
3569.
residual-354B.
-2
-2
-2
-2
-2
-1
i
sentiment rankZscore
sentiment ran antoose
sentiment ran ranoZee
vate dIfY ranaaaccrc
mankZiccre
CUA quality rank-sscoee
1
1
-1
-1
residual-37123
residual-3554
residual-
-2
-2
-2
2
helptuiness cank2score
helpfulness
helpfuiness rankZscore
(c) codegolf meta
2
Vote dIft ranizscore
CVP raricssoore
CVA queand Ciienn rankcsccre
1
0
0
-1
-1
residual=6769.22
residual= 6245.
residual 6222.3
-2
-1
-2
-2
-1
sentiment
sentiment
sentiment raniZsccree
2
CVP quality ranicsscrre
CMA qualitty quallty rankisscore
vate dit manizzcara
0
0
-1
6651
residual-6122.
residual
-2
-2
-2
-2
1
2
-2
-1
-2
-1
1
helpfulness rankZscore
helpfulness rank2score
helpfulness rank
(d) askubuntu
2
2
CVP ranksscare
CUA quality rankZscoree
vota dIt mankzz.coee
1
1
-1
-1
-1
residual-2031.
residual- 1872.
residual- 1856.5
-2
-2
-2
rankkecore
sentiment rasizee
rankZscore
2
2
vote DITY aanizscara
CVP mrncZccora
CVA quality raricZscore
1
1
1
195
residual 1766.4
residual 1
-2
helpfulness rankZscore
helpfulness rankZscore
helpfulness
(e) cstheory
17

---

Counterfactual Voting Adjustment for Quality Assessment and Fairer Voting in Online Platforms with Helpfulness Evaluation
rankZscom
CUA qualitylttyyy raniZscome
vodo dir ranisscrrs
1
1
385
-439d02l-3499.5
-2
-2
-2
-2
-2
-1
i
sentiment rankZscore
sentiment rankZstore
sentiment ran annoZee
vote dITY ran ranaaaacar
mankZiscor
CUA quality rank-sccree
1
1
-1
-1
residual-3764.
residual-3465.
residual- 3425.2
-2
-2
-2
2
helptuiness a2roes
helptuiness rankZscore
helpfuiness rankZscore
(f)
math.meta
2
Vote dIt ranizscore
CVP aaricssoor
CVA. queen canicsccom
1
0
0
-1
-1
6251
60688
residual 6067 6067.1
-2
-1
-2
-2
-1
sentiment
sentiment
raniZsccre
2
CVP quality ranicsscore
CWA Somiing rankZscoree
vate dR manizzccrr
1
0
0
-1
-1
S651
residual-
residual
-2
-2
-2
-2
2
-2
-1
-2
-1
helpfulness rankZscore
helpfulness rank2score
helpfulness rankZscore
(g) philosophy
18

---

Counterfactual Voting Adjustment for Quality Assessment and Fairer Voting in Online Platforms with Helpfulness Evaluation
H. Other example community plots to answer the counterfactual questions
be Me 04
Vote Probability
Probablity
044
rank
(SOF)
(b) mathoverflow
1.
Vote Probability babiitty
Vote Probability
87
rank
rank
(c) codegolf.meta
(d) askubuntu
carsery
maturmeia
Vote Probability
Vote Probability
85
0,1
rank
rank
(e) esthcory
math.meta
p8 kegply
nahotshato, sieMood evarsssoiaati
Vote Jote:0 Probability
rank (g) philosophy
19


Reviews:
[{"review_id": "3HwEpiCrj5", "reviewer": "Reviewer_Xff6", "summary": "This paper develops a model of voting (for example, StackOverflow up/down votes) that measures and removes the effect of position bias (voters are more likely to vote on already highly-ranked items) and herding bias (voters are likely to agree with the existing consensus). This allows a better estimate of item quality than what would be naively estimated from the available votes.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s positive evaluation and recognition of our work’s strengths and its practical value. Below we respond to the questions and comments:\n\n**Re the comparison with the existing model:**\n\nThe proposed CVA shows three main advantages compared to existing CVP: Firstly, the CVP paper didn’t provide causal effect estimation. Secondly, the CVA is able to mitigate position and herding bias at the same time while CVP can’t. Thirdly, the CVA is better than CVP in evaluations especially for the communities with high position bias, since CVP didn’t consider the rank position when predicting the vote.  For those communities with much less position bias, the advantage of CVA won’t be shown significantly.\n\n**Question: If GPT4o can be considered ground truth, what’s the value of this approach?**\n\nAlthough GPT-4o is a good ground truth proxy, it still needs to be validated by human evaluation, especially for different expertises of different communities, which will be done in our future work. Additionally, our work can not only provide a fairer quality estimation, but also give measurements of position and herding bias which are valuable insights about the community behavior trends for the platforms.", "claims_and_evidence": "This paper makes a convincing case that the proposed method is both necessary to solve a problem not addressed by existing methods (which can't jointly mitigate position and herding bias) and makes progress towards a solution.", "methods_and_evaluation_criteria": "The method is a nice application of generative modeling to this problem. The evaluation datasets make sense, given the lack of available ground truth comment qualities. The improvements shown in these evaluations are modest, but the method tends to do as well as or better than the existing Chinese Voting Process model.", "theoretical_claims": "I didn't check the correctness of the theoretical claims.", "experimental_designs_or_analyses": "Yes, they seem sound given the difficulty of finding ground truth comment qualities.", "relation_to_broader_scientific_literature": "This paper provides a modest improvement over prior models of voting in these contexts.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "NA", "other_comments_or_suggestions": "NA", "questions_for_authors": "If GPT4o can be considered ground truth, what’s the value of this approach?\n\n“thereby learning its model parameters for position and herding biases independently” I think it would be useful to cover this in more detail, since this is the key advantage over the state of the art.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4rYgjCx3GU", "reviewer": "Reviewer_qZQL", "summary": "This paper proposes Counterfactual Voting Adjustment (CVA), a causal inference framework to mitigate position bias (content visibility due to ranking) and herding bias (influence of prior votes) in online voting systems. By modeling counterfactual scenarios where votes are cast under neutral visibility and balanced prior signals, CVA disentangles true content quality from biases. Key contributions include a causal framework leveraging voting trajectories and backdoor adjustment, empirical validation on semi-synthetic and real-world StackExchange data, and cross-community analysis revealing varying bias patterns (e.g., technical communities exhibit stronger herding bias, while others show position bias dominance).\nExperiments demonstrate CVA’s superiority over baselines (e.g., raw votes, Chinese Voting Process) using metrics like Kendall’s τ correlation and alignment with ground truth approximated via comment sentiment and GPT-4o evaluations. The framework enables platforms to rerank content for fairer quality estimation, enhancing user experience and information reliability. Integration of GPT-4o as a proxy for human evaluation highlights adaptability to modern AI tools, while insights from 120 StackExchange communities underscore its practical utility in diverse contexts.\nStrengths: Novel causal design, robust validation, and actionable cross-community insights.\nWeaknesses: Limited discussion on computational scalability and real-time deployment challenges. Overall, the work addresses critical biases in online systems with methodological rigor and empirical grounding.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s positive evaluation and recognition of our work’s strengths. Below we respond to the questions/comments:\n\n**a. Regarding the ground truth (proxy reliance),**\n\ni. For human judgments, due to the different topics discussed in different communities, to hire specific groups of human experts for each community is a non-trivial work, and the huge volume of questions of StackExchange increases the cost of acquiring human judgment data. We plan to tackle these difficulties and fill the gap between GPT-4o evaluation and human evaluation in our future work.\n\nii. For StackExchange’s expert labels, if you mean the ‘accepted answer’ labels, we didn’t leverage it mainly because that an answer is accepted by the author of the question when he/she feels it is good enough and then it will stay on the top rank. However, It’s not necessarily the best answer for all the time. \n\niii. For details about how LLM is used, the prompt for an example question is provided in the Appendix I. We use temperature=0 and we will release our codes for parsing the LLM responses. \n\n**b. Regarding the causal framework,**\n\ni. For the causal identification assumption and robust/sensitivity analysis, we don’t allow unobservable confounders, and we will acknowledge in the paper that this assumption can be a limitation of our approach. \n\nWhen hidden confounders are present, indeed one could develop alternative causal identification strategies like proxy variables to aid identification, or perform sensitivity analysis to unobserved confounding, assuming a certain level of unobserved confounding is present. However, these strategies like proxy variables need customized development and we leave them to future work.\n\nThis work also connects to invariant causal prediction; it constructs a latent information quality assessment that is invariant to changes in display rank and voting sequences. We will discuss these connections in the paper.\n\nii. For other key parametric modeling assumptions, it is indeed true that the model could be misspecified and may not reflect real world nonlinear dynamics fully. In response, we have stressed test our model under a series of different model misspecification scenarios (e.g. missing nonlinear terms). We found that, despite model misspecification, CVA outperforms baseline method CVP in its accuracy of assessing information quality. We will include these results in the paper, and discuss the potential issue of model misspecification.\n\niii. For other baselines like IPW and doubly robust AIPW estimators, while these estimators like IPW can potentially be applied to handle position bias, it cannot handle position bias and herding bias simultaneously. In particular, handling herding bias involves careful modeling of the whole sequence of votes. These obstacles make it difficult for us to compare CVA with baselines like IPW and doubly robust AIPW estimators.\n\n**c. Regarding synthetic data generation,**\nthe herding bias mechanisms used for synthetic data generation are mainly based on the Chinese restaurant process which has solid theoretical foundations. \n\n**d. Regarding including more dataset,** our method needs the historical trajectory of vote data (not snapshot static data), and we can only get this kind of data from StackExchange for free. We will consider doing the experiments when other datasets are available.\n\n**e. Regarding how to select the regularization parameters,** we do cross-validation within the search space (λ ∈ [0.1,0.2, 0.3,0.4, 0.5,0.6, 0.7,0.8,0.9,1, 2, 3,4,5, 6, 7,8,9,10,20, 30,40,50,60, 70,80,90,100, 200, 300, 400, 500, 600, 700, 800, 900,1000]). The validation metric we used is the voting prediction accuracy.\n\n**f. Regarding the term inconsistency of “quality”,** we will unify the term use of quality. Our quality indicates the user's perceived quality at a time, and we assume the users in the same community have the same preference. \n\n**g. Regarding the reproducibility,** we will release our codebase.\n\n**h. Regarding the Ethical & Societal Considerations,** we acknowledge that our paper does not fully address potential societal impact from two angles. First, our findings highlight the risk of information monopoly due to the design of a helpfulness voting mechanism that seeks efficient wisdom-of-crowds, thus suppressing potentially more valuable information often contributed later by other users. Our findings urge that 1) human users should critically interpret online information and 2) system providers need to employ a causal framework to mitigate biases in social decision making processes.\n\n**i. Regarding the writing,** we explained why causal and behavioral modules are jointly necessary in the third paragraph of Introduction. We will improve the transition from Sec 2 to Sec 3.\n\n**j. To answer the questions,**\n\n(1) Please see the replies for b. the causal framework\n\n(2) Please see the replies for a. the ground truth (proxy reliance) and d. more dataset", "claims_and_evidence": "The claims made in the submission are largely supported by clear and convincing evidence, though minor limitations exist. Here are potential limitations:\n1. Real-world experiments use comment sentiment and GPT-4o evaluations as proxies, not direct human judgments. Semi-synthetic data with known ground truth partially addresses this, but the absence of human validation remains a caveat.\n2.The causal graph (Figure 2) assumes covariates X block all confounders, which depends on correct model specification. While standard in causal inference, sensitivity analyses or robustness checks could strengthen confidence.\n3.Experiments are limited to StackExchange communities. The framework is theoretically adaptable (as noted in §7), but empirical validation on other platforms (e.g., product reviews) is absent.", "methods_and_evaluation_criteria": "The methods and evaluation criteria are well-suited to the problem, leveraging causal inference principles and rigorous experimentation. The use of semi-synthetic data with ground truth and real-world proxies (comment sentiment, GPT-4o) provides balanced validation. While limitations exist (e.g., proxy reliance, unmeasured confounders), the approach is methodologically sound and addresses the core challenge of bias mitigation in online voting systems. Future work could expand validation to other platforms and incorporate human evaluations to further solidify claims.", "theoretical_claims": "1.The paper’s theoretical claims—identifiability of content quality parameters, counterfactual invariance, and estimator consistency—require stronger formal grounding. While the proposed CVA framework aligns with causal principles (e.g., backdoor adjustment), key assumptions (e.g., linear position/herding bias models, no unobserved confounders) lack validation. For instance, the identifiability of quality parameters hinges on parametric assumptions like β/(1+D)\\beta/(1+D)β/(1+D) for position bias, which may not reflect real-world nonlinear dynamics (e.g., exponential decay with rank). Similarly, claims of counterfactual invariance and MLE consistency lack rigorous proofs or sensitivity analyses (e.g., robustness to model misspecification). A formal treatment using causal frameworks (do-calculus) and asymptotic theory (Van der Vaart regularity conditions) is needed to strengthen these arguments.\n2. The empirical validation relies heavily on semi-synthetic data and GPT-4o as a proxy for ground truth, raising concerns about real-world applicability. While results show improved correlation metrics (e.g., Kendall’s τ\\tauτ), the synthetic data generation process may oversimplify bias mechanisms (e.g., predefined herding rules). Furthermore, GPT-4o’s alignment with human judgment remains unverified, and confounding factors (e.g., content freshness, user expertise) are not controlled. For broader validity, the authors should test CVA on fully observational data with human-annotated quality labels and address unobserved confounders via sensitivity analyses.", "experimental_designs_or_analyses": "1.The semi-synthetic validation introduces circularity by generating data using the same parametric bias models (e.g., β/(1+D)\\beta/(1+D)β/(1+D) assumed in CVA, potentially inflating performance. Reliance on GPT-4 as a quality proxy further weakens grounding, as its alignment with human judgment is unverified. To strengthen validity, the authors should test CVA on real-world benchmarks with human-annotated labels (e.g., StackExchange “accepted answers”) and replace semi-synthetic data with fully synthetic datasets incorporating nonlinear bias mechanisms.\n2.Key causal baselines (e.g., inverse propensity weighting) are omitted, limiting claims of superiority over state-of-the-art methods. Additionally, unaddressed confounders (e.g., user reputation, content age) and overaggregated community-level results (e.g., masking failure modes in political forums) reduce interpretability. The authors should include advanced causal baselines, report statistical significance (e.g., confidence intervals), and disaggregate results by community type to clarify performance variations.\n3.Insufficient details on hyperparameter tuning (e.g., regularization strength λ\\lambdaλ) and missing sensitivity analyses (e.g., Rosenbaum bounds for unobserved confounding) hinder reproducibility. A code/data release and robustness checks against alternative bias models (e.g., exponential decay for position effects) are critical for practical adoption.", "relation_to_broader_scientific_literature": "1. The paper builds on causal inference and recommendation systems by unifying position/herding biases into a structural framework, advancing beyond correlational models (e.g., Agarwal et al., 2019) and heuristic approaches (e.g., time-decay weighting). While parametric bias modeling (e.g., β/(1+D)\\beta/(1+D)β/(1+D)) aligns with classical exposure literature, its causal formalization of identifiability and invariance lacks rigorous ties to foundational tools like do-calculus (Pearl, 2009) or modern sensitivity analyses (Cinelli et al., 2020). The use of GPT-4 as a quality proxy follows trends in LLM-driven annotation (Gilardi et al., 2023) but overlooks established validation methods (e.g., StackExchange’s expert labels). Though the integration of behavioral assumptions (e.g., modular bias components) is novel, gaps remain in connecting to robust causal estimation techniques (e.g., doubly robust estimators) and addressing regularization impacts on consistency claims.\n2. The work extends causal ML by treating biases as structural parameters rather than confounders, bridging social dynamics (e.g., Muchnik et al., 2013) and identifiability theory. However, its reliance on semi-synthetic data risks circularity compared to real-world benchmarks (Gordon et al., 2019), and the omission of advanced baselines (e.g., inverse propensity weighting) limits practical relevance. Strengthening ties to invariance frameworks (Pfister et al., 2021) and adopting sensitivity analyses for unobserved confounders would better situate it within modern causal literature.", "essential_references_not_discussed": "The paper omits critical context for its methodology. First, it cites Kamalho & Rafiei (2023) to justify GPT-4’s use as a quality evaluator but overlooks Gilardi et al. (2023), which systematically validates LLM-as-a-judge alignment with human judgments—a gap weakening claims about label reliability. Second, while proposing causal adjustments for bias, the work fails to contrast its approach with established propensity-scoring methods (e.g., Agarwal et al., 2019, widely adopted in industry for position bias correction). Addressing these omissions would clarify how CVA advances beyond existing bias-mitigation frameworks.", "other_strengths_and_weaknesses": "The paper demonstrates notable originality through its innovative integration of causal inference frameworks (e.g., do-calculus) with behavioral dynamics like herding and position bias, proposing a modular hybrid model (CVA) that advances interdisciplinary methodology. Its practical relevance is evident in addressing real-world platform challenges (e.g., biased content ranking) while maintaining interpretability via disentangled quality and bias parameters. Methodologically, the semi-synthetic validation and sensitivity analyses enhance robustness, though reliance on LLM-generated labels introduces unquantified risks.\nHowever, theoretical gaps weaken its foundation: the absence of formal identifiability proofs for causal parameters under unobserved confounding and insufficient discussion of regularization-induced biases (e.g., L2 penalty trade-offs). Empirically, the lack of validation on real-world data (e.g., Reddit/Stack Exchange voting histories) and incomplete baseline comparisons (e.g., doubly robust estimators) limit claims of superiority. Technical opacity—particularly in LLM annotation protocols and computational scalability—hinders reproducibility, while ethical implications of deploying LLM-dependent models remain unexplored.\nTo solidify contributions, the authors should: 1) strengthen theoretical grounding (e.g., identifiability proofs), 2) validate on authentic platform datasets with broader baselines, 3) disclose LLM prompt details and code, and 4) address societal risks (e.g., bias amplification). Addressing these would elevate the work from a promising interdisciplinary prototype to a rigorous, deployable solution.", "other_comments_or_suggestions": "1. Writing & Presentation: The term “quality” is used inconsistently. In Section 3.1, it is defined as “intrinsic content value”, but in Section 4.2, it conflates with “user-perceived relevance”. Clarify whether “quality” refers to objective merit, subjective preference, or platform-defined metrics (e.g., Reddit’s upvote/downvote rules). The transition from theory (Section 2) to methodology (Section 3) feels abrupt. Consider adding a brief subsection (e.g., “Motivation for Hybrid Modeling”) to explain why causal and behavioral modules are jointly necessary.\n2. Technical Clarifications:\n(1) The paper states that “regularization parameters were selected via cross-validation” (Section 4.1), but does not specify the search space (e.g., λ ∈ [0.1, 1.0]?) or validation metric (e.g., MSE, likelihood?). Include these details in the appendix.\n(2) The model assumes “no unobserved confounders between position and clicks” (Section 3.2). However, real-world platforms often have hidden confounders (e.g., user fatigue, time-of-day effects). Acknowledge this limitation and discuss potential solutions (e.g., proxy variables).\n(3) While GPT-4 is used to generate “quality” labels, no calibration steps (e.g., temperature scaling, human validation of samples) are mentioned. Add a paragraph on how LLM outputs were standardized (e.g., z-score normalization) to mitigate overconfidence.\n3. Ethical & Societal Considerations: The reliance on GPT-4 for label generation risks inheriting its biases (e.g., gender stereotypes in text, Western-centric perspectives). A fairness audit (e.g., disaggregating performance by content topic or demographic proxies) is critical but missing. Training large hybrid models (CVA + LLM components) may incur high computational costs. Estimate the carbon footprint (e.g., using tools like CodeCarbon) and suggest efficiency optimizations.", "questions_for_authors": "1. The paper applies do-calculus for causal estimation but lacks formal identifiability proofs under unobserved confounders (e.g., user demographics). How do you justify the absence of such proofs, and could instrumental variables or sensitivity analyses (e.g., Rosenbaum bounds) address potential biases? Additionally, L2 regularization may bias causal estimates—did you analyze this trade-off (e.g., bias-variance decomposition), and why prioritize it over doubly robust methods?\n2. Experiments rely on semi-synthetic data with GPT-4 labels but omit validation against real-world benchmarks (e.g., Reddit moderator labels). If such data is unavailable, could quasi-experiments or robustness checks substitute? Further, the LLM annotation protocol lacks transparency: Share prompts, temperature settings, and calibration steps (e.g., human validation) to assess bias risks (e.g., verbosity preference inflating performance).", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "No ethical issues"}}, {"review_id": "aVRsrlu1hD", "reviewer": "Reviewer_F6j8", "summary": "This paper introduces the Counterfactual Voting Adjustment (CVA) framework, designed to address biases in online voting systems that distort information quality assessment. Specifically, CVA targets position bias (where content appearing higher receives more votes) and herding bias (where visible prior votes influence subsequent evaluations).\n\nThe authors argue that traditional aggregated voting systems fail to reflect true content quality due to these biases. CVA adopts a causal inference approach, modeling the counterfactual scenario in which content is displayed at different positions with equalized prior vote counts. By leveraging voting trajectories, CVA corrects these biases and improves content ranking fairness.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s positive evaluation and recognition of our work’s strengths and its practical value. The reviewer raised several crucial questions, to which we respond below.\n\n**a. Re “unobserved confounders”:**\n\nAmong various factors influencing helpfulness evaluation, this paper specifically targets social biases by distangling two primary biases: position and herding bias. These biases often lead to winner-take-all type of information cascades, overshadowing other potentially more valuable information. As shown by our experimental results, our framework can easily incorporate additional factors: any observable feature or additional biases like length-bias. Following the reviewer’s suggestion, our immediate future work will try to improve the framework’s robustness against unobserved confounders.\n\n**b. Re “GPT-4o itself is subject to limitations”**\n\nThis issue is particularly relevant as more scientific papers adopt Large Language Models (LLMs) as their judge mechanism. Our study used GPT-4o for 1) sentiment analysis on user comments that are given to individual answers as well as 2) helpfulness evaluations as proxies of human evaluations. Given that sentiment analysis by LLMs shows near perfect accuracy, strong correlations with comment sentiment first show that our model can successfully mitigate position and herding biases. \n\nFollowing the previous research by Lee et al 2016, we employed the same metric based on the observation that users frequently write their comments to address and discuss positive or negative points behind each answer — content and nuances not apparent in numeric helpfulness scores alone. Next, prior to using GPT-4o for helpfulness scoring, we performed various preliminary experiments. Specifically we chose top, middle(median), and bottom-ranked answers from each question, then trying to optimize prompts so that GPT-4o’s evaluations become approximately consistent with existing ranking of these three answers, whose quality are separately compared by human annotators. While complete agreement between GPT-4o’s rankings and existing rankings is neither expected nor desirable due to inherent biases, we carefully designed our experiments to rigorously test the effectiveness of our framework.\n\n**c. Re “potential social impact”:**\n\nWe acknowledge that our paper does not fully address potential societal impact from two angles. First, our findings highlight the risk of information monopoly due to the design of a helpfulness voting mechanism that seeks efficient wisdom-of-crowds, thus suppressing potentially more valuable information often contributed later by other users. Our findings urge that 1) human users should critically interpret online information and 2) system providers need to employ a causal framework to mitigate biases in social decision making processes.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "Yes", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "The paper is well-written and easy to follow, but some notation can be imrpoved (see below).\n\nThe problem is well-motivated. \n\nThe paper validates CVA through multiple approaches—synthetic data, real-world StackExchange datasets, and GPT-4o evaluations—demonstrating consistent performance gains over existing methods.\n\nRegarding weaknesses,   CVA’s effectiveness relies on the assumption that confounding factors are adequately captured in the observed data. Any unobserved confounders could undermine its reliability. Moreover, while CVA's improved alignment with GPT-4o's judgments is promising, GPT-4o itself is subject to limitations and may not always reflect human expert evaluations.", "other_comments_or_suggestions": "Right column, line 150, instead of  j = 1,...,J_t−1, a better notation is j \\in { 1,...,J_t−1}, and this comment applied at line 156 as well. \n\nThe authors could explain more what M_{i,j}^t denotes. In each round, do multiple users vote on issue i? Similarly, more explanation  is needed regarding what R_i,j^t denotes. An example would be useful.\n\nI also believe that a more detailed impact statement would be useful for this work as if the proposed algorithm is applied in practice, it could have a social impact.", "questions_for_authors": "See above", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper introduces a causal inference framework to mitigate position bias and herding bias in online voting systems. All the reviewers agreed that the work is interesting, and the empirical evaluation is thorough.
Decision: Accept (poster)