While using reward confidence as a proxy for preference differences can offer a practical and efficient way to estimate preference discrepancies without exhaustive human annotations, there are potential limitations and drawbacks to consider. One limitation is the reliance on the reward model's internal confidence estimates, which may not always accurately reflect the true nuances of human preferences. This could lead to biases or inaccuracies in the estimation of preference differences. To address this, it is essential to validate the effectiveness of the reward confidence approach through rigorous testing and validation on diverse datasets with varying levels of complexity and ambiguity. Additionally, incorporating mechanisms for recalibration or fine-tuning of the reward confidence estimates based on feedback from human evaluators can help improve the accuracy and reliability of the approach.