URG: A Unified Ranking and Generation Method for Ensembling Language Models (2024)

Abstract: Prior research endeavors of the ensemble Large Language Models (LLMs) achieved great success by employing an individual language model (LM) rank before the text generation. However, the use of an individual LM ranker faces two primary challenges: (1) The timeintensive nature of the ranking process, stemming from the comparisons between models; (2) The issue of error propagation arising from the separate ranking and generation models within the framework. In order to overcome these challenges, we propose a novel ensemble framework, namely Unified Ranking and Generation (URG). URG represents an end-toend framework that jointly ranks the outputs of LLMs and generates fine-grained fusion results, via utilizing a dedicated cross-attention-based module and noise mitigation training against irrelevant information stemming from bad ranking results. Through extensive experimentation and evaluation, we demonstrate the efficiency and effectiveness of our framework in both the ranking and generation tasks. With the close coordination of the ranking and generation modules, our end-to-end framework achieves the state-of-the-art (SOTA) performance on these tasks, and exhibits substantial enhancements to any of the ensembled models.

heading: 1 Introduction
text: Large language models (LLMs) have shown a superior performance in storing factual knowledge in their parameters (Chowdhery et al., 2022; Lv et al., 2023; Tang et al., 2023b, 2024; Zhao et al., 2024). These off-the-shelf LLMs exhibit diverse strengths and weaknesses due to variations in data, architectures, and hyperparameters, making them
*Equal contribution. †Corresponding author.
complementary to each other. Therefore, by combining their unique contributions, the ensemble of LLMs can alleviate biases, and errors in individual models, resulting in outputs better aligned with human preferences.
In recent years, the Ensembling of Large Language Models (LLMs) has demonstrated remarkable performance through the judicious incorporation of the diverse strengths of multiple LLMs. A simple approach (Yu et al., 2023a) is to concatenate all candidate results as contextual inputs to a new
4421
generative model for fusion output, but meanwhile this approach also introduces a too lengthy input for the ranker, Typically, the prior ensemble approaches (Salazar et al., 2020; Liu and Liu, 2021; Ravaut et al., 2022; Jiang et al., 2023) employed an individual ranking model ranking response outcomes from multiple LLMs, and then select the top-ranked candidate output as the final ensemble result feeding to a generator. The main difference in the prior works relies on the design of the mechanism in the ranking model. We select the state-ofthe-art model, LLM-Blender (Jiang et al., 2023), as an example. The authors propose a coarse-grained and fine-grained mixed pipeline that firstly makes a pair-wise comparison feature matrix of N LLMs evaluated by a language model as the scorer, and then a generative model fuses to generate the final output from the top-k ranked outputs.
However, training individual ranking models and generators faces two primary challenges. (1) The time-intensive nature of the ranking process: Due to the input limitations of language models, pairwise comparisons are implemented when comparing multiple models. Consequently, when dealing with N LLMs, the time complexity of the process becomes O(N2). In addition, the individual ranking model also incurs additional memory and time costs. (2) The error propagation between ranking and generation: The ranking stage and the generation stage represent two distinct tasks with different inputs and outputs. The textual evaluation knowledge acquired by the ranking model cannot be transferred to the subsequent generator, and vice versa. The text generation objective of the generator is not directly connected to the ranking model. Consequently, the framework may suffer from the introduction of poor candidates (noise) during the ranking stage, which may adversely affect the subsequent generation process.
To address the aforementioned issues, we propose a novel unified ranking and generation method, abbreviated as URG. This method realizes an end-to-end framework with an encoderdecoder-based language model, encompassing both the ranking and generation modules. As depicted in Figure 1, our framework initially gathers the encoded embeddings of candidate responses from N LLMs, along with the encodings of the input x. Subsequently, a cross-attention module is implemented to list-wise form the concatenated features composed by the pair of each N candidate and x, which has merely O(n) time complexity. This
cross-attention module assigns the attention scores to each LLM candidate pair, and aggregates the feature list to predict the tokens in y∗. Therefore, we share the parameters of this cross-attention module with that of the decoder in the generator, which also aims to predict the next token of y∗ based on x with the top k candidates and the history tokens predicted in y∗. Consequently, the decoder learns the representations of N candidates throughout both the ranking and generation stages. In the subsequent generation stage, we further introduce a Kullback-Leibler (KL) loss to contrastively learn the features of positive and negative LLM candidates. By minimizing the KL loss, the decoder (generator) mitigates the adverse effects on the final output y∗ from low-quality candidates (negative). This training objective serves to mitigate the influence of noise introduced during ranking, bridging the gap between the ranking stage and the subsequent generation stage, resulting in a more robust ensemble model. We conduct a series of experiments to analyze the effectiveness and efficicenty of our proposed framework. The experimental results demonstrate that our framework achieves state-of-the-art (SOTA) performance in both the ranking and generation stages. Furthermore, our framework exhibits a substantial improvement in running speed compared to the prior SOTA model, LLM-Blender.
Our contributions can be summairzed as follows: • We propose a novel framework, Unified Ranking
and Generation Method (URG), which integrates the ranking and generation stages within an endto-end framework. • We introduce a KL-loss based mechanism to contrastively learn the features of positive and negative LLM candidates, thereby enhancing the robustness of the ensemble framework against the adverse impact of low-quality LLM candidates introduced during the ranking stage. • We conduct a series of experiments demonstrating that URG significantly reduces running time and achieves state-of-the-art (SOTA) performance according to various automatic metrics and human evaluations.

heading: 2 Related Work
text: Ensemble learning is a widely employed technique that enhances accuracy and resilience in forecasting by merging predictions from multiple models (Sagi and Rokach, 2018; Aniol and Pietron, 2019; Wei
et al., 2023). With the rapid increase in the application of large language models (LLMs) (Liu et al., 2020), the ensemble of LLMs has emerged as a significant topic (Yu et al., 2023b). In light of the prior works. There are two primary approaches for ensembling LLMs: selection-based and generationbased methods. Selection-Based Methods Selection-based methods develop ranking models (Salazar et al., 2020; Liu and Liu, 2021; Ravaut et al., 2022) to compare candidate results generated by the multiple LLMs, selecting the top-1 candidate as the ensemble output. However, these methods are coarse-grained ensembles due to the inherent nature of selection and the limited solution space. As a result, they may limit the ability to harness the advantages of each candidate, potentially hindering the generation of superior outputs. Generation-Based Methods Generation-based methods employ generative models (Yu et al., 2023a) to fuse the candidate results and generate improved output as the final response. However, these methods face challenges in dealing with the surge in computational complexity and inference time with the increase of input text length (Liu et al., 2022; Sun et al., 2024; Tang et al., 2022; Loakman et al., 2023; Goldsack et al., 2023; Tang et al., 2023a; Yang et al., 2024). To address this issue, Jiang et al. (2023) proposes a rank-then-generate pipeline that first adopts a pairwise ranking model to sort outputs from N LLMs and then utilizes a generative model to generate the final output from the K top-ranked outputs. Despite shortening the length of the input text, this model still faces the problems of slow sorting speed caused by pairwise comparisons and the issue of error propagation arising from separate ranking and generation models.

heading: 3 Method
text: Figure 2 illustrate the overall architecture of our proposed URG framwork, which is an end-to-end model.

heading: 3.1 Problem Definition
text: The base task is instruction-following, and we formulate the task as follows: the input text is x, and the output of this task is to generate a response y. The ground-truths in the data are denoted as ŷ. In this paper, we focus on post-hoc ensemble learning, wherein given an input x and N LLMs, {M1, ...,MN}, the task is to combine the candidate
results, Y = {y1, ..., yN}, generated by processing x with N LLMs to produce the integrated final output result y∗. Our goal is to develop a novel unified ranking and generation framework that applies a single language model throughout both the ranking and generation stages, to fine-grained fuse the candidate results outputted by the N LLMs.

heading: 3.2 Encoding Text
text: We separately concatenate the input x with each candidate result yi, forming a unified input sequence with special tokens as separators: 
 <Question> x 
 <response> yi, which is denoted as pi. Subsequently, each input sequence pi is fed into encoder layers to obtain the encoder’s last layer hidden state H ie:
H ie = EncoderLurg 2 (pi) ∈ RTp×h (1)
where h is the hidden dimension, Lurg is the total number of encoder layers and decoder layers, and Tp is the sequence length of pi. By processing candidates independently in the encoder, the computational complexity of the model grows linearly with the number of candidates, as opposed to quadratic growth when concatenating all candidates.

heading: 3.3 Ranking and Filtering Candidates
text: The objective of the ranking stage is to assess the qualities and rank the generated N candidates from LLMs. Only the top k candidates are retained for the subsequent generation stage, mitigating potential issues associated with increased time complexity and noise resulting from inputting an excessive number of candidates into the subsequent generator. The remaining N candidates are regarded as relatively lower-quality instances answering x, particularly the one ranked last, which is also utilized in the following steps for contrastive learning. Why Choose Cross-Attention The standard cross-attention computation for a single head in a transformer is:
Attn(Q,K, V ) = softmax( QKT√
dk )V (2)
where Q = HdWq is the product of the decoder states Hd and the query weight matrix Wq; the keys K = HeWk are the product of the last encoder hidden states He with the key weight matrix Wk; and V = HeWv is similarly the product of He with the value weight matrix Wv.
We denote the attention score between the query Qi at position i, and the key Kj at position j as
αi,j = QiK T j / √ dk. After normalized over the dimension, the attention score is rewritten as:
α̃i,j = softmax(αi,j) = exp(αi,j)
exp( ∑ m(αi,m)) (3)
where 1√ dk is the scaling factor and the dk is the dimension of each transformer head. According to Equation 2 and Equation 3, the new representation can be expressed as a sum of the values, weighted by the attention probabilities, before going through a final linear transformation Wo:
Oi = Wo ∑
j
α̃i,jVj (4)
It is evident that the attention score α̃i,j quantifies the significance of Vj in the computation of the new representation Oi. A lower attention score associated with Vj suggests its diminished importance for Oi. Consequently, by utilizing the aforementioned steps to compute attention scores , the ranking module is trained to discern the qualities of N candidates. How to Utilize Cross-Attention In our paper, the query Q of the cross-attention mechanism is the prediction y∗. K and V are established based on the Ncandidates. Initially, the hidden state of all candidate results output from the last layer of the encoder are concatenated and forwarded to the decoder to generate the first token A:
A = Decoder([H1e ;H 2 e ; . . . ;H N e ]) (5)
where A represents the initial representations of y∗. In this process, we can obtain attention scores α̃l1,j
at each decoder layer l, and then aggregate these scores across all layers and heads to derive α̃mean1,j . Subsequently, we can compute the ranking score sk for each candidate result yk by averaging all the attention scores α̃mean1,j corresponding to the H k e :
sk =
∑b+Tp j=b α̃ mean 1,j
Tp (6)
where b is the start token position of Hke within the concatenated sequence, and Tp is the length of Hke . Consequently, the predicted rank list of candidate results Y is obtained.
The following loss function is used to optimize this rank result:
Lrank = N∑
i=1
N∑
j=1
max(0, si − sj)ri<rj (7)
where ri,rj are the ground-truth scores of yi,yj . The ground-truth score r can be obtained from either the reward model output or human ratings. In this paper, the ranking labels are provided by the MixInstruct dataset, which uses BART scores to rank the candidates. These scores are generated by a BART model that compares candidate outputs with answer responses.

heading: 3.4 Enhancing The Model Robustness
text: Although in an ideal scenario, the top k candidates retained are presumed to exhibit good qualities, practical situations may still entail the presence of low-quality candidates after the ranking stage.
Such low-quality candidates have the potential to exert adverse effects on the final output y∗ generated by the generator. In order to enhance the model’s resilience against irrelevant information stemming from low-quality candidates within the input, we introduce an additional noise mitigation training approach. Here, the worst candidate (ranked last) is incorporated alongside the top k candidates as part of the input, forcing the model to generate outputs that are consistent with those derived from the top-k candidates.
The encoded embeddings of the top-k candidate results, denoted as [H1ste ; . . . ;H k−th e ], are concatenated to form Etk. Furthermore, the encoded embeddings of the lowest-ranked result are appended to Etk, yielding a new concatenated embedding Etk−l = [H1ste ; . . . ;H k−th e ;H last e ]. Given the ground-truth ŷ consisting of L tokens, represented as T = t1, t2, ..., tL, the model generates the output logits using ETtk as the condition. Subsequently, the probabilities of the top-m candidate tokens at each decoder step are normalized using the temperature parameter τ to obtain the probability distribution p:
p(t∗j = Vn|t<j , Etk) = exp(z(Vn)/τ)∑ mexp(z(Vi)/τ) (8)
where the z(Vn) and z(Vi) are the probabilities for the n-th and i-th candidate tokens in the vocabulary, respectively. Inspired by knowledge distillation in neural machine translation tasks, we use top-m candidate tokens at each decoding step, instead of the full distribution, in order to focus on important tokens when calculating the distribution divergence. Inspired by knowledge distillation (Tan et al., 2019; Wei et al., 2021; Lv et al., 2024) in neural machine translation tasks, we use the top-m candidate tokens at each decoding step, instead of the full token distribution. This enables the model to focus on capturing important token information, enhancing its learning capacity. Similarly to the steps producing p, Etk−l is fed into the model to obtain the normalized probability, q(t∗j = Vn|t<j , Etk−l), of the top-m candidate tokens in the new output probability distribution. Finally, we employ the KL (Kullback-Leibler) loss to minimize the divergence between the aforementioned distributions:
Lkl =
L∑
j
m∑
n=1
p(t∗j = Vn|t<j , Etk)×
log p(t∗j = Vn|t<j , Etk) q(t∗j = Vn|t<j , Etk−l)
(9)
Additionally, we employ cross-entropy loss to minimize the disparity between the model’s predicted probabilities and the ground-truth ŷ:
Lce = −( L∑
j=1
log p(tj |T<j , Etk)
+ L∑
j=1
log p(tj |T<j , Etk−l)) (10)
The overall loss function of our approach comprises the rank loss, the KL divergence loss, and the cross-entropy loss:
Ltotal = γLrank + βLkl + Lce (11)
where γ and β represent the weights assigned to the rank loss and KL loss, respectively.

heading: 3.5 Analyzing Computational Complexity
text: We analyze the theoretical time complexity of our proposed method compared to the vanilla method. The more practical computational cost comparison is shown in Table 4.
Suppose the length of the ensembled result is denoted as Ta, and the average length of the candidate result (concatenated with question) is Tp. The time complexity of the pairwise ranking model (Jiang et al., 2023) with L layers is O(N2 · T 2p · d · L), where N, d denote the number of candidate results and the embedding dimensions. For list-wise models (Liu and Liu, 2021), the time complexity is O(N · T 2p · d · L), as each candidate’s score is evaluated only once instead of through pairwise comparisons.
The ranking stage of our URG method firstly utilizes the encoder to encode all the candidate results separately, with a time complexity of O(N · T 2p · d · Lurg2 ), where Lurg is the total number of layers in URG. Both the encoder and decoder have Lurg 2 layers. Then, we concatenate all the embeddings and input them into the decoder to perform a decoder step. The time complexity of this part is O(
Lurg 2 · d · (N · Tp · Ta + T 2a )), where N · Tp · Ta
comes from the cross-attention mechanism, and Ta is the number of decoder step. Since we only executed a single decoder step, Ta = 1, the computation of the decoding phase can be considered negligible compared to the encoding phase. As a result, the overall complexity of the URG ranking stage can be approximated as O(N · T 2p · d · Lurg2 ).

heading: 4 Experiments

heading: 4.1 Datasets and Evaluation
text: MixInstruct MixInstruct was curated by (Jiang et al., 2023) to serve as a benchmark for ensemble models in instruction-following tasks utilizing Large Language Models (LLMs). Table 1 illustrates that this benchmark encompasses a substantial collection of instructional examples sourced primarily from four distinct origins. Specifically, 100k examples are randomly allocated for training, with 5k each designated for validation and testing purposes. To generate candidate results, each instruction is fed into a set of N = 12 popular open-source language models (LLMs), such as Vicuna1, Open Assistant2, Llama3, and others (refer to Table 2). Evaluation To evaluate the efficacy of our framework, a range of automatic metrics are selected for the following experiments. BLEU (Post, 2018) and ROUGE (R-n) (Lin, 2004) evaluate the quality of a generated response in comparison to a reference by measuring n-gram overlap.4 We additionally adopt some neural network based metrics as a supplement, i.e. BERTScore (Zhang et al., 2019), BARTScore (Yuan et al., 2021) and BLEURT (Sellam et al., 2020) to measure the semantic similarity between the generated results and the references. These metrics leverage pre-trained language models to predict if the model response is semantically equivalent to the gold answer. Model Sizes We use two configurations for our URG method: (i) Flan-T5-large configuration consisting of 24 layers, 16 attention heads, and 1024 embedding dimensions, leading to 770M parameters, and (ii) a larger Flan-T5-xl configuration
1https://huggingface.co/lmsys/vicuna-13b-v1.5 2https://huggingface.co/OpenAssistant/
oasst-sft-4-pythia-12b-epoch-3.5 3https://huggingface.co/mosesjun0h/ llama-7b-hf-baize-lora-bf16 4R-L refers to the longest common subsequence.
consisting of 3B parameters. Compute Hardware We perform training on instances containing 8 A100 GPUs, each containing 80 GB RAM. Training Details When training with Flan-T5xl (3B), we perform training for 10 epochs using Adam with a batch size of 4, dropout value of 0.1, and peak learning rate of 8×10−5 with warmup and linear scheduling. Due to the smaller size of FlanT5-large, we train for 20 epochs with a batch size of 16. We save the model checkpoint every 1000 steps and perform model selection by evaluating it on the development set. To save memory, we use stage 2 and bfloat16 mixed precision in the deepspeed5 training framework to train the model. In the training phase, we select the top k ranked candidates, where k is set to 3 for the generation phase. When calculating the KL loss, we choose the top 32 scores for the computation, implying that m is set to 32. We set the value of the temperature hyperparameter (τ = 2) and weight coefficient (γ = 5, β = 2) using cross-validation.

heading: 4.2 Baselines
text: In this section, we introduce several baseline methods for ensembling LLMs. • SummaReranker (Ravaut et al., 2022): A
selection-based method concatenates the input and each candidate to learn ranking and utilizes binary cross-entropy (BCE) loss to differentiate the best candidate from the others. • SimCLS (Liu and Liu, 2021): A popular contrastive learning-based method encodes the input and each candidate, then computes their cosine similarity as the ranking score. • PAIRRANKER (Jiang et al., 2023): A pairwise comparison method focuses on learning to capture the differences between two candidates and prefers the ones of higher quality. • LLM-Blender (Jiang et al., 2023): A generationbased ensembling framework consisting of a PairRanker and a generation model. • SimCLS-Gen: A generation-based ensembling method first takes the top-3 ranked results from SimCLS and then inputs them into a generative model to generate a new fused result. • SummaReranker-Gen: A method similar to SimCLS-Gen, wherein the rank model is SummaReranker. The backbone of the ranking models in the base-
5https://github.com/microsoft/DeepSpeed
line is DeBERTa-large (418M) (He et al., 2020), and the generation models are all Flan-T5-XL (3B) (Chung et al., 2022b). We train the aforementioned baselines using the code and parameters provided by Jiang et al. (2023). All selectionbased methods involve sorting all candidates and outputting the top-1 candidate based on the scores as the ensemble output.

heading: 5 Experiments Results

heading: 5.1 Main Results
text: Table 2 shows the performance of N=12 opensource LLMs as well as other baseline models on the MixInstruct test set. The results of both the selection-based ensemble method and the generation-based method are obtained by integrating the outputs of these 12 open-source LLMs. Our proposed URG method demonstrates substantial performance improvements over the previous stateof-the-art (SOTA) model, LLM-Blender, across all evaluation metrics. Notably, even our smallersized URG (770M) model exhibits superior performance compared to the LLM-Blender approach, comprising a ranking model with 418M parameters and a generation model with 3B parameters, across most metrics. When compared to ChatGPT, which boasts 175 billion parameters, our URG (3B) model still achieves superior scores on several metrics. These results underscore the effectiveness and efficiency of our method through fine-grained ensembling. Additionally, we compare our ranking results with selection-based methods. Both the smaller and full versions of the URG-rank model outperform the previous SOTA method, PairRanker, across all metrics. This highlights the superiority of our method, which leverages cross-attention scores to simultaneously compare the superiority of all candidates, over previous pair-rank methods that compare candidates pairwise. Moreover, our method exhibits faster inference speeds, as we will analyze in §5.3.

heading: 5.2 Ablation Study
text: Since our proposed URG method works with two novel techniques introduced in ranking mechanism (§3.3) and enhancing robustness mechanism (ERM) (§3.4), we conduct an ablation study to validate the effectiveness of both. As Table 3 illustrates, the performance of URG substantially drops when either of the two mechanisms is removed, demonstrating the effectiveness of both mechanisms.

heading: 5.3 Analysis of Inference Speed
text: We conducted experiments to compare the inference speed of the ranking stage in the URG method with previous ranking models. The results of the experiments are presented in Table 4. Considering both the model size and speed, Our URG method models achieve the fastest inference speed. Furthermore, even though our URG (3B) model has a large number of parameters, it still has advantages over the previous SOTA method, PairRanker, in terms of both performance and inference speed. Aligning with the theoretical analysis in §3.5, even though the URG models have much larger model size (around 1.7 times and 6.7 times than the baselines), the URG models have less computational cost due to the optimized encoding and decoding strategy. Table 7 shows a comparison of the performance and inference speed of our model and the previous model under different batch sizes. We also calculated the corresponding inference speed size(M)×speed1B after normalizing the parameters to 1B, in order to explore the comparison between our method and other methods under the same parameter size. The results show that our method is faster than the previous method under different batch sizes, while also achieving higher BERTscore and Bartscore. This indicates that, compared to the state-of-the-art (SOTA) method PairRank, which previously compared candidates pairwise, our method achieves better performance by comparing all candidates together with a single decoder step, and it also has a faster inference speed.

heading: 5.4 Analysis of Model Robustness
text: To compare the ability of our method with other methods in resisting noise introduced in the ranking stage, we conduct a model robustness testing experiment. In this experiment, lower-ranked candidates, regarded as noise candidates, results are fed into the model for decoding. As shown in Table 5, the adverse effects of noise candidates have a significantly lesser impact (almost half) on the performance of the URG generator than that of the LLM-Blender generator. This suggests that our mechanism for enhancing model robustness (see §3.4) effectively bolsters the model’s resilience to noisy inputs.

heading: 5.5 Analysis of The Top k value
text: To assess the impact of different values of k on the fusion results, we conducted experiments to exam-
ine the variation of BERTScore as the number of candidate inputs k increases during the fusion stage. The BERTScore for each k value was obtained by training the model using that specific k value and selecting the best score on the development set.
As depicted in Figure 3, the BERTScores of the three models initially exhibit an upward trend with the increase of the k value. However, beyond k = 3, the growth rate notably decreases, and in
some cases, the scores cease to increase altogether. Consequently, to optimize model training and inference speed by minimizing input text, we fixed the settings of k = 3 for both training and testing in our model.

heading: 5.6 Human Evaluation
text: Three evaluators are invited to annotate 200 samples, which are randomly sampled on the test set of the MixInstruct dataset. Table 6 shows that compared with the LLM-Blender method, our URG method has a substantial increase in performance on both the appropriateness and informativeness metrics. This is in line with the evaluation conducted using machine learning and automated metrics, which consistently demonstrates that our method outperforms the previous SOTA methods,
regardless of the evaluation method employed. In addition, by integrating the outputs of 12 opensource large models, the results of our model’s output demonstrate competitive performance comparable to ChatGPT. This indicates that the integration method can partially compensate for the performance gap caused by the size of individual models.

heading: 5.7 Case Study
text: Table 8 illustrates an example showcasing the varied responses from our URG model, open-source LLMs, LLM-Blender, ChatGPT, and the reference human response, indicating that our responses are most similar to those of humans. More details are introduced in Appendix A.1.

heading: 6 Conclusion
text: In conclusion, we propose a novel framework, Unified Ranking and Generation Method (URG), which integrates the ranking and generation stages
within an end-to-end framework. To enhance the robustness of the URG framework against the adverse impact of low-quality LLM candidates, we introduce a KL-loss based mechanism to contrastively learn the features of positive and negative LLM candidates. Experiments on MixInstruct show that URG significantly reduces running time and achieves SOTA performance according to a variety of automatic metrics and human evaluations.

heading: 7 Limitations
text: The requirement for multiple high-performing models and a significant amount of accurately labeled data to train an ensemble model poses challenges in constructing datasets for other tasks. Despite the good performance of our approach in ensembling the outputs of LLMs in the instruction-following task, we haven’t explored whether our method is also effective in different tasks due to the difficulty of constructing other task datasets. Furthermore, considering that the computational complexity of model inference increases quadratically with the input length, even after narrowing down the candidate inputs from N to k through quality filtering, this stage remains sluggish. Therefore, the problem of extracting only a limited number of snippets from each candidate instead of the entire text requires resolution in the future.

heading: 8 Acknowledgements
text: This work is supported by the National Key Research and Development Program of China (Grant No. 2021ZD0112905), the Major Key Project of PCL (Grant No. PCL2023A09), the National Natural Science Foundation of China (Grant No. 62206140, 62076231), the China Postdoctoral Science Foundation (Grant No. 2022M711726), and the Fundamental Research Funds for the Central Universities (No. 2023SCU12094).

heading: A Appendix
text: A.1 Case Study Table 8 shows that to become a teacher, a bachelor’s degree and a teaching qualification are prerequisites, according to human responses. Additionally, they provide insights into different job-finding methods. The integration of LLM-Blender was influenced by the output of ChatGLM-6b. The answer emphasized methods of being an elementary school teacher, but the input question did not specify which stage of the teacher. Furthermore, it overlooked the important information of having a bachelor’s degree and a teaching qualification certificate. Our URG method filters out low-relevance candidates during the filtering stage and is able to better overcome the influence of noisy input during the fusion stage, ultimately producing outputs that are close to human responses.

heading: Future Work
text: Therefore, the problem of extracting only a limited number of snippets from each candidate instead of the entire text requires resolution in the future