SlowGuess/NeurIPS_2023_OCR huggingface dataset:
While the proposed strategy does not require modification of the original inference process of the Language Model (LM), it still necessitates the logits or probabilities of the next token. As a result, it may be necessary to approximate the probabilities in some completely black-box LM services, such as GPT-4.
B Clarifications After Rebuttal
Q: Computational cost of using  $G$ -fair Prompting becomes non-trivial as  $N$  goes larger.
(1) We acknowledge the concern regarding the computational cost of G-fair Prompting as N increases. However, we would like to emphasize that the worst-case computation cost of  $1/2 * N^2$  is not a common scenario. In practice, only a few demonstrations are required. For example, in GPT-4 Technical Report [24], the number of demonstrations is fixed at 5. As a result, the overall computation cost remains acceptable, especially when compared to other existing methods [15] among the typical methods detailed in the related work.
(2) Moreover, our method does not require re-selection of demonstrations for different test samples. Consequently, the inference cost for selecting demonstrations is minimized, particularly when dealing with a larger number of test samples.
(3) In the worst-case scenario, even with an infinitely large training set, our model can efficiently perform Greedy search on a subset, consistently outperforming random demonstration. The additional computational cost depends on the size of the search space that users want to explore.
(4) In contrary, the naive strategy, i.e., enumerating all possible situations, has an exponential time complexity of  $O(n!)$ . The smaller training set used in our experiments is due to considering the computational cost of the comparative methods in the report, rather than any limitations of the proposed method.
Q: One can think of even better algorithm that can show benchmark performance that provides upper bound against G-Prompting.
(1) The proposed simple version T-fair-prompting serves as a time-efficient alternative of G-fair-prompting, while still maintaining competitive performance.
(2) Compared with T-fair-prompting, G-fair-prompting offers the advantage of automatically determining the number of demonstrations, providing a more natural and flexible approach.
(3) Therefore, a tradeoff between performance and time cost can be achieved by transitioning between T-fair-prompting and G-fair-prompting. Specifically, we can select the demonstrations order as same as T-fair-prompting and then decide the number of demonstrations by the strategy of G-fair-prompting. The computation cost upper bound of this strategy will be the lower bound of G-fair-prompting while its fairness lower bound will be the upper bound of T-fair-prompting.
Q: Under the proposed problem formulation, there will be a natural extension to test both different choices of few-shot examples and all possible ordering of them.
(1) As shown in Fig.2, we have tested different choices of few-shot examples (e.g., [x1, x2]/[x3, x4]) and all possible orderings of them (e.g., [x1, x2, x4]/[x2, x4, x4]). In total, 64 different possible situations were considered when the training set size was 4.
(2) Notably, G-fair-Prompting performs remarkably close to the best performance obtained through enumeration, highlighting the efficacy of our proposed method.
Q: How to choose the semantic-free token without any guidance?
(1) Following previous work [18], three different semantic-free prompts are selected, i.e., ['N/A', ' ', '']. The fairness is then calculated as the average of them.
(2) These tokens were chosen based on their guaranteed performance in previous work [18]. We will provide further clarification on this selection in the final version of our paper.
Q: Would be the proposed method helpful for long-form QA?
While the current version of our method is designed for classification tasks, we acknowledge the potential for its extension to long-form QA scenarios.
(1) The limitation preventing direct deployment in generation tasks lies in the absence of an appropriate method to measure the fairness of sentence-level generation.
(2) Similarly, for measuring the calibration of large language models, such as ECE, it can only be calculated for classification tasks [24].
(3) However, we believe that once the community develops techniques to measure the predictive bias of sentences, our proposed strategies can be naturally extended to long sentence generation tasks. This presents an exciting avenue for future research.
Q: What does "total cost" in table "Dataset descriptions" refer to?
Table 6: Dataset descriptions.
Corpus	Task	Classes	Domain	Total Cost1
SST-2	sentiment	2	movie reviews	over 60 GPU hours
TREC	QA/QC	6	open domain	over 220 GPU hours
AGNews	topic	4	news	over 250 GPU hours
CoLA	acceptability	2	misc.	over 160 GPU hours
RTE2	NLI	2	news, Wikipedia	over 110 GPU hours
1 Total Cost=Hours×GPUs. Hardware: BLOOM=A100, LLaMA=V100.
2 Not applicable to LLaMA because of the maximum prompt token limit.
(1) "Total cost" indicates the overall computational cost on each dataset including baselines and the proposed methods.
(2) We report the total computational cost on each dataset to provide a more comprehensive description of the datasets used in this paper. We will clarify this in the final version.
$Q$ : The diversity baseline seems similarly competitive to the proposed approach.
Regarding performance, we would like to clarify the following points:
(1) Our method consistently outperforms the diversity baseline across various scales of the LLaMA family datasets.
(2) The similarity in performance between the diversity method and our method on the Bloom model is attributed to the Bloom model's robustness to the transformations of the demonstration. Even random demonstrations can achieve competitive performance with the Bloom model.
(3) It is essential to note that the baselines, including the diversity method, are the latest methods proposed shortly before submitting this paper. They have demonstrated robust performance on multiple tasks.
Q: Could the authors compare using a strategy such as the proposed one with a large language model vs. fine-tuning with a smaller model?
** Note that although the former strategy requires iterating over the entire dataset, the latter strategy only requires performing inference. It does not incur any space cost for storing gradients nor time cost for updating gradients, making this process highly efficient.
We would like to clarify the advantages of utilizing a large language model and comparing it with fine-tuning a smaller model:
(1) Generalization ability of large models: Large language models (LLMs) accumulate significant prior knowledge during pre-training, resulting in better generalization abilities compared to small models fine-tuned for specific tasks. Even if the performance of a large model is not significantly better than that of a dedicated small model on a specific task, the robustness of the large model in open environments, such as out-of-distribution (OOD) scenarios, is significantly superior to that of a dedicated small model [24].
(2) Multiple tasks: A single large model can perform multiple tasks simultaneously. Compared to fine-tuning a small model for each task, a large model can seamlessly switch between various tasks and effortlessly incorporate new tasks.
(3) Time efficiency: Many open-source large models, such as LLaMA, do not require modifications when encountering new tasks. Even equipped by the methods including baselines in this paper, only a few additional inferences are necessary, without any updates to the model or backpropagation of gradients.
Q: Explanation about the text in lines 293-296 and 300.
(1) Line 293-296 Explanation:
The sentence mentioned in lines 293-296 analyzes why the similarity-guided method achieved the best performance on the topic classification task (AGNews). This method selects unique prompts for different test examples based on the distance metric. Consequently, it chooses demonstrations with labels that are the same as the test samples. Large Language Models (LLMs) tend to predict biased outcomes towards the labels that frequently appear in the context of demonstrations. As a result, the similarity-guided method excels on the topic classification task but may perform poorly on other tasks, such as natural language inference.
(2) Line 300 Explanation:
The Bloom model exhibits high robustness to various demonstrations, even random ones, leading to similar performance results between our method and the compared method. It can be observed that the Bloom model's performance remains competitive regardless of the selected demonstrations.
Q: Discussion about limitations.
Due to space limitations, our discussion on limitations is included in Appendix (A.9 Limitations). While the proposed strategy does not require modification of the original inference process of the Language Model (LM), it still necessitates the logits or probabilities of the next token. As a result, it may be necessary to approximate the probabilities in some completely black-box LM services, such as GPT-4.
Q: Figure 2 shows a non-monotonic trend between the bias and predictive quality.
According to Figure 2, the relationship between predictive bias and predictive quality is not strictly monotonic. Instead, the overall trend shows a significant negative correlation. To avoid any misunderstandings, we will use clearer descriptions in the final version of the paper.
$Q$ : The term fairness is slightly misleading.
The fairness defined in our work is defined from the concept of predictive bias, which may not align precisely with the fairness, such as social bias, that is commonly referred to. It is important to note the distinction between fairness in the context of predictive bias and fairness in the context of social bias. Moreover, we highlight that the proposed metric can be extended to analyze social bias, as social bias can also lead to predictive bias [24].
Q: Contribution and difference from [18].
Our paper shares a similar metric with calibration-before-use [18] to access the predictive bias of a given prompt. However, the differences are significant and listed as follows:
(1) The prior approach aims to use the metric for calibrating the output, which can still be influenced by the quality of the used prompt, as evidenced by Table 3.
(2) In contrast, our research focuses on finding a near-optimal prompt on the original space to improve the model's performance, eliminating the need for any post-adjustment to the output.
(3) Additionally, our paper introduces an empirical validation of the connection between predictive bias and final task performance, as depicted in Fig. 2, which is not explored in [24].
(4) The main contribution of our work lies in analyzing the relationship between predictive bias and performance through extensive experiments, and proposing a Greedy research strategy to solve the NP-hard problem.
(5) Our experiments have revealed that even without calibration, the prompt selected by our method outperforms randomly selected prompts, even with calibration.
(6) Moreover, we have observed that [24] can potentially harm the model's performance, which is a practical concern worth considering when manipulating the model's probability directly.
Q: How sensitive are these strategies to the choice of hyperparameters?
In this paper, we reduce the influence of hyperparameters by minimizing the need for manual hyperparameter tuning.
(1) In G-fair-prompting, the number of demonstrations is automatically determined, indicating that no new demonstrations are added once it is determined that adding more demonstrations does not lead to improvement.
(2) As for the threshold for fairness improvement, we fix it with 0, indicating that new demonstrations are added to the prompt if they are deemed helpful.
(3) While these two aspects could be manually adjusted to make the method more flexible. To ensure the stability of the experimental results, we avoid deliberately setting these hyperparameters.