TCRA-LLM: Token Compression Retrieval Augmented Large Language Model for Inference Cost Reduction
Authors: Junyi Liu
Liangzhi Li
Tong Xiang
Bowen Wang
Yiming Qian
Venue: EMNLP 2023 Findings (2023)
Keywords: Large language model;Retrieval augmented;Knowledge compression;Question Answering
TL;DR: In this paper, we propose multiple token reduction schemes that balance the trade-off between input length and performance.

Abstract: Since ChatGPT released its API for public use, the number of applications built on top of commercial large language models (LLMs) increase exponentially. One popular usage of such models is leveraging its in-context learning ability and generating responses given user queries leveraging knowledge obtained by retrieval augmentation. One problem of deploying commercial retrieval-augmented LLMs is the cost due to the additionally retrieved context that largely increases the input token size of the LLMs. To mitigate this, we propose a token compression scheme that includes two methods: summarization compression and semantic compression. The first method applies a T5-based model that is fine-tuned by datasets generated using self-instruct containing samples with varying lengths and reduce token size by doing summarization. The second method further compresses the token size by removing words with lower impact on the semantic. In order to adequately evaluate the effectiveness of the proposed methods, we propose and utilize a dataset called Food-Recommendation DB (FRDB) focusing on food recommendation for women around pregnancy period or infants. Our summarization compression can reduce 65\% of the retrieval token size with further 0.3\% improvement on the accuracy; semantic compression provides a more flexible way to trade-off the token size with performance, for which we can reduce the token size by 20\% with only 1.6\% of accuracy drop.

TCRA-LLM: Token Compression Retrieval Augmented Large Language Model for Inference Cost Reduction
Junyi Liu
Liangzhi Li
Tong Xiang
l Bowen Wang 0 P.
Yiming Qian 12
Meetyou AI Lab,
Xiamen Key Laboratory of Women's Internet Health Management,
Osaka University, Agency for Science, Technology and Research (A (A*STAR)
exiaooouzi... com, nowen.wangee ids. osaka-u. jp, qiany@ihpc.
(liujunyi, liliangzhi, qiang@tppc.a-strr.eee a-star. edu. sg
Abstract
in casual conversations with users, LLMs exhibit astonishing performance by leveraging strong context learning ability. However LLMs may pro- duce vague responses or incorrect answers in cer- tain specialized domains, owing to the absence of relevant knowledge or restricted scope of information acquired during the training stage, which might potentially result in untruthful answers and even cause physical damages users (Xiang et al., 2023). For QA in such domains, retrieval- augmented generation (RAG) (Lewis et al., 2020), where the system retrieves external knowledge be- forehand and then utilizes LLMs to generate an- swers leveraging retrieved knowledge, can greatly reduce the hallucinations generated (Shi et al., 2023; Shuster et al., 2021).
Since ChatGPT released its API for public use, the number of applications built on top commercial large language models (LLMs) increase exponentially. One popular usage of such models is leveraging its in-context learn- ing ability and generating responses given user queries leveraging knowledge obtained by re- trieval augmentation. One problem of deploy- ing commercial retrieval-ue -aggmmeted LLMs is the cost due to the additionally retrieved con- text that largely increases the input token size of the LLMs. To mitigate this, propose token compression scheme that includes two methods: summarization compression and se- mantic compression. The first method applies a T5-based model that fine-tuned by datasets generated using self-instruct containing sam- ples with varying lengths and reduce token size by doing summarization. The second method further compresses the token size by remov- ing words with lower impact on the semantic. order to adequately evaluate the effective- ness of the proposed methods, propose and utilize a dataset called Food- conddRecommenattion DB (FRDB) focusing on food recommendation for women around pregnancy period or infants. Our summarization compression can reduce 65% of the retrieval token size with further 0.3% improvement on the accuracy: semantic compression provides a more ficxible way to trade-off the token size with performance, for which we can reduce the token size by 20% with only 1.6% of accuracy drop.
Many current commercial LLMSs are black-box models, where the model architectures and the weight information are not disclosed. These LLMs own superior text comprehension abilities, yet in many cases they can only output desired answers through complicated prompt engincering, On the other hand, deploying open-source LLMs to local servers is resource-intensive,, in contrast deploy- ing smaller models such as T5 (Raffel al., 2020). Some commercial LLMs like GPT-3.5-turbo (Ope- nAI, 2023c) and GPT-4 offer access through API calls; however, these models charge users based on the size of input and output!. For individuals or companies looking to create their own services using LLMs through API calls, utilizing commer- cial ones can be resource-consuming if requests are made frequently. Therefore, it necessary mini- mize the number of input tokens while maintaining optimal performance during the API calls.
1 Introduction
With the increase in computing power and accumu- lation of enormous text data, large language mod- els (LLMs) such ChatGPT (OpenAI, 2023b) and GPT-4 (OpenAI, 2023a) have shown impressive performance dialogue-based question-answeriing (QA), allowing them to interact with users fluently. open-domain QA where the models are engaged
In this work, we propose token compres-
'As of May 5th, 2023, GPT4, capable of processing up to 8k tokens, charges $0.03 per thousand input tokens and $0.06 per thousand output tokens: GPT-4-32K which can process 32k tokens, charges $0.06 per thousand input tokens and 10.12 per thousand output tnkens. --rurb.-ttubbo charges $0.002 per thousand tokens.
X Corresponding author.

---

Large Language Model
Question
Answer
Question
Retriever
Context
Answer
LLM
Retrieval Augmented
Large Language Model
ledge database
Top-k Retriever
Question
NSP
Context
Answer
Compressor
LLM
Token Compression Retrieval Augmented Large Language Model
knowledge database
Figure I: Mustratiio of different ways to utilize LLMs for QA. Top: directly using L.M.. Middle: using a retreeval-uggmented LLM. Bottom: using auggeeted LLM with our proposed token compression methods.
sion scheme specifically designed for the retrieval- augmented LLMs (shown in Figure 1), namely, Token Compression Retrieval Augmented Large Language Model (TCRA-LLM). Our proposed scheme can reduce up 65% of the token size with additional 0.3% improvement on accuracy when doing QA on our proposed dataset called Food- Recommendation DB (FRDB). We propose two approaches to reduce the token size of the LLMS' input: summarization compression and semantic compression. For summarization compression, leverage self-instruct (Wang et al., 2022) scheme build multiple summarization datasets with vary- ing lengths to fine-tune the mT5 model (Xue al., 2020). The samples from the summarization datascts are generated by GPT-3.5-turbe (OpenAl, 2023c), which instructed to shorten the sum- mary of the input sentences in an iterative manner. The semantic compression approach is based on simple yet effective intuition, that removing seman- tically less important words in sentence won't 't drastically change its semantic. Here, deploy multi- multi-lingual sentence- -raeencccetranffmmerr (Reimers and Gurevych, 2020) to encode sentences into embed- dings where the distances between original and perturbed embeddings are used to measure the se- mantic deviation from the original meaning. Larger semantic deviation indicates that the correspond- ing word owns more important semantic in the sentence. We conduct iterative process that sures the semantic importance of each word in the sentence and remove less important words.
In conclusion, our work has the following three contributions:
We construct a food recommendation QA dataset (Section 3) which contains domain knowledge that general LLMs might not have.
This dataset serves the purpose of evaluating retrieval-augmented LLMs.
We propose a multi-level self-instruct scheme (Section 4.2) to build summarization datasets of different lengths.
3.
We propose two token compression methods (Section 4.2), both of which can reduce the number of input tokens during the API calls of aetrieval-augmenter commercial LLMs while maintaining optimal performance.
Related Work
LLMs such as GPT-3 (Brown et al., 2020). PALM (Chowdhery et al., 2022), OPT (Zhang et al., 2022), Bloom (Scao et al., 2022), and LLaMA (Touvron et al., 2023) are trained on mas- sive amounts of data and have demonstrated pow- erful comprehension capabilities. These models have been deployed in breadth of tasks and achieve promising results (Zhang et al., 2023; Ashok and Lipton, 2023; Lu al., 2023; Wang et al., 2023; Xiang al., 2023). One major bar- rier that prevents more people from participat- ing in commercial deployment the LLMs is

---

their training and hosting costs. A way re- duce such costs is through training smaller domain- specific models such as BioMedLM (Bolton et al., 2022), BloombergGPT (Wu et al., 2023), and LawGPT (Nguyen, 2023). Such domain-specific training enables smaller LLMs to be applied to cer- tain fields but still requires huge investment. For instance, BloombergGPT is trained on 512 40GB A100 GPUs with the total budget being approxi- mately $2.7 million (Sheikh, 2023).
Food type
Count
Entréc
31
Vegetables
31
Seafood
22
Sweets
22
Medicine/Health supplement
20
Fruit
17
Grains
16
Soft drink
13
Condiment
10
Alternatively, LLMs can be used without fine- tuming through retrieval augmentation leveraging external data sources, where the retrieved data is used supplementary information to help LLMs improve logical reasoning and language genera- tion (Thorne al., 2021; Izacard al., 2022). Pre- vious experiments (Ram et al., 2023) show that additional information can beneficial for LLMs across different model sizes. Retrieval augmenta- tion eliminates the cost of tuning an in-house LLM new data, and can be easily integrated with com- mercial LLM services such ChatGPT (OpenAl, 2023b) from OpenAl or Bard (Pichai, 2023) from Google. Many studies have shown, applying re- trieval augmentation the commercial LLMs such as ChatGPT allow the models to gain knowledge in specific domains such as natural science and medicine (Soong et al.. 2023; Inaba et al., 2023) which is not revealed during their training and re- trieval augmentation can be further improved by applying more sophisticated retrievers (Shi al., 2023). However, commercial LLMs all have a lim- itation on input lengths which put an upper ceiling on the amount of information that can fed into a LLM. Later models such as GPT-4 has looser restriction but the inference cost increases dras- tically in comparison with other models. Some previous work applies template-based prompt op- timization (Santra et al., 2023), which select re- trieved context (Mallen et al., 2022) in an adaptive manner, or uses cascading of LLMs with different sizes (Chen al., 2023) to reduce the inference costs. Our proposed method has no conflict with these methods and can used with them simulta- neously.
Meat/Fggs
10
Soybeans/Dried fruit
6
Dairy products
2
Total
200
Table I: Statistics of food types in FRDB.
parts: multiple-choice (MC) QA pairs and knowl- edge database. The QA pairs contain 1,000 sam- ples that cover 200 types of food. The categories of foods are shown in Table
The possible answers to the question falls into three choices based on the increasing degree of rec- ommendations ranging from 1 (avoid) to 3 (highly recommend). Each type of food has five recom- mendation rating corresponding to five groups: mre-pregnancy, pregnancy, postpartum, lactation, and infant. Additionally,
build knowledge database that contains 7,588 entries; details of the entries are shown Table The distribution of sentence length the knowledge database is shown in Figure 2.
Mean
Max
Min
Std.
# of words 88
248
12
27
Table 2: Statistics of the entries FRDB knowledge
database. Std. stands for standard deviation.
All the information has been verified by the health-domain professionals. During the verifica- tion, we remove the text that ambiguous the human annotators. Two samples knowledge are shown in Table 3. Sample questions are available in the Appendix A.
3 FRDB
4 Method
We build Food Recommendation Dataset in Chi- nese called FRDB, for recommending foods that are safe to consume for women before/duurng/aaterr their pregnancy as well as infants. contains two
Typically, a aerreval-auggeented LLM consists of three components (shown Figure 1), knowledge database, a retriever, and the LLM. The knowledge

---

High quality knowledge
Ambiguous knowledge
Consuming mushrooms after childbirth is benefi- cial for posipartum recovery, constipation relief. and promoting lactation due their rich B vi- tamins, protein, and amino acids. moderate amount of intake based on the recovery status recommended.
Postpartum mothers are safe to consume a moderate amount of cake, Cakes are easier to digest and absorb for postpartum mothers with weaker gastrointestinal systems. However, cakes have relatively smaller nutritional diversity and they should be consumed together with vegetables, fruits, and meats to make the nutrition more balanced.
Table 3: Examples from the knowledge database. The ambiguous ones are excluded from our dataset.
coders and then finds a query's nearest neighbors in the embedding space using a relevancy measure such as cosine similarity (Yu al., 2021). Sparse retrieval, on the other hand, maps queries and docu- ments into a high-dimensional space with methods such as TF-IDF (Sparck Jones, 1972; Jones, 1973) and the most relevant documents are returned to the user as the answer. Typical example of sparse retrieval is BM25 (Robertson et al., 1995).
0.1
0.08
Probabilty
0.06
0.04
0.02
Here we evaluate both dense and sparse retrieval methods. For dense retrieval, we follow a similar process from Huang et al. (2013): we first encode the text using the GPT-embedding (OpenAI, 2022) provided by OpenAl, then deploy vector database FAISS Index (Johnson et al., 2019) to store the embeddings, enabling faster manipulation on them. For sparse retrieval, we deploy BM25 (Robertson et al., 1995) which is considered the standard way.
50
100
150
200
250
Length of sentence
Figure 2: The distribution of sentence length in FRDB knowledge database.
database contains all available domain-specific knowledge. The retriever applies the question as a query to search for the relevant information from the knowledge database. The retrieved informa- tion is then formulated as the context packaged together with questions as a prompt for LLM to generate an answer. Our proposed methods are able to compress the retrieved information and formu- late shorter context but maintain the effectiveness of retrieval augmentation. In this section, we go through the pipeline of augmenttd LLM system for QA and introduce our proposed token compression methods.
4.1.1 Next Sentence Prediction
The retrieved top-k results D₁- [d db. d%) (dh are the i-th retrieved elements from the original set where 1 < < k) are deemed as the most relevant ranked by the retrieval method. Using only the top-1 result as the context is not always reliable, but using more retrieved results will consume more space in the input tokens, leading to higher costs; therefore, to improve reliability without incurring additional costs, propose to next-sentence prediction (NSP) second-stage ranking method. is based on an intuitive assumption that the re- trieved information is more likely to be predicted as the next sentence of the question if it is more related to the question. The implementation of this approach is based on the pre-trained NSP module from BERT (Devlin et al., 2018); the selected sen- tence with maximum probability from NSP is selected the best result (See Equation 1).
4.1 Information Retrieval
Generally, the first step for LLM's retrieval aug- mentation is knowledge retrieval. Given an user query x, the retriever extracts k pieces of in- formation from the knowledge database D - (d1, d2,... ... } that are most likely to be rele- vant z. There are two mainstream retrieval meth- ods: dense retrieval (Karpukhin al., 2020; Ni al., 2021) and sparse retrieval (Robertson al., 2009). Dense retrieval first encodes queries and documents into dense embeddings (Huang et al., 2013; Yi et al., 2019) using pre-trained neural en-
8
i = argmax (p(z, d(), .. pla, dbx (1)

---

We conduct experiments to evaluate the im- pact of including NSP into the retrieval-augmented LLM. Here we use OpenAl's GPT-3.5- as the base LLM and evaluate it on the FRDB dataset using the top-1 retrieval results as the context. The result is shown in Table 4. There is minor perfor- mance gain using NSP with both GPT-embedding and BM25 and thus we keep this NSP module in all our later experiments.
words with lower semantic impact on the sentence are removed.
4.2.1 Summarization Compression
Summarization models like the mTT5 model (Xue et al., 2020) have been widely used in many ap- plications to shorten the input text, but they could not output summary with arbitrary length due to the constraint of its training data. To solve this, we propose to build a summarization model that is able to output summary with various lengths.
Method
Acc. (%)
To build such a model, we leverage the power of self-instruct (Wang et al., 2022) where we use GPT- 3.5-turbo to generate training datasets. The proce- dure of the data generation shown in Figure 4. First, we start with text x from the dataset, then pack it with additional prompt instruction we illustrated in Figure 4 and send it to GPT-3.5-turbo to generate summary. If the length of the sum- mary meets requirements, the procedure is ended; otherwise, a follow-up prompt will instruct GPT- 3.5-turbo to further shorten the summary to the desired length. By doing this, we build a collection of training datasets with different summary length, We build three datasets that are 30%, 50%, and 70% of their original length. Each dataset is used to fine-tune one summary model independently. We randomly extract from FRDB and generate 400, 50, and 50 samples for training, validation, and testing respectively. Training on the generated datasets not only enables the model to produce summaries the desired length, but also familiarizes the model with domain-specific information by doing further domain adaptation (Gururangan et al., 2020).
Embedding
89.1
Embedding +NSP
90.2
BM25
83.4
BM25+NSP
84.9
Table Performance comparison retrieval methods with and without NSP. The retrieved sentences di- rectly used as context and fed into GPT-3.5-ture 5-tubb. Ace. stands for accuracy.
From the experiment, we also see that the com- bination of dense retrieval with the NSP approach obtain the highest accuracy. We tune the value of k by searching it from to 10 and perform cor- responding evaluation on the FRDB dataset. The experiment results are shown in Figure 3. We find that k = 3 is the optimal choice and we will adhere to this value in all our subsequent experiments.
90
Aocuracy
88
88
4.2.2 Semantic Compression
2
3
Top
We propose another compression method based on perturbations of the original sentence and rank- ing the impact of the semantic importance for each word in the sentence where words with less importance will be removed. We deploy a multi-lingual sentence-transsfomeer (Reimers and Gurevych, 2020) to encode a sentence into embed- ding Xo- Then we iteratively remove one word in the sentence and obtain an updated embedding Xi where i is the index of the word in the sen- tence and is the number of words in the sentence. We have new L that tracks the Euclidean distance between the original and perturbed em- bedding = (T.2(xo> X1),.... T,., Xn))- We denote Pj as the value of the j-th percentile in L. The L. is the new subset that removed the bottom
Figure 3: Evaluation of the retrieval performance using GPT-embedding and NSP with different choices of k.
4.2 Token Compression
The retrieval text is usually long and easily con- sume huge amount of space from the input tokens during the API calls while using commercial LLMs. In order to mitigate this, we propose two methods to compress the retrieved text. The first one is the summarization compression which produces shorten the original text by rephrasing. The second method is semantic compression which we perturb the original sentence and rank the impact of the se- mantic change from each word in the sentence. The

---

and 70% of their original length are generated. The GPT-3.5-turbo (OpenAI, 2023c) is used for pro- cessing prompts generated by our methods.
Original input
Summarize the following text, the length of the summary result is 30%550 70% of original text, keep the first sentence, and directly octpat
Prompt instruction
5.2
Token Compression Results
GPT-3.5-turb with summarization fine-turedd
GPT-3.5-turbo
GPT-3.5-turbo with semantic compression
GPT-S.5-turbo random deletion
GPT-3.5 @GPT-3.5-trrbo. with summarization no fine-tuned
GPT-3.5-turbe without retrieval
GPT-8.5-turbo with retrieval without token compression
Summary resalt
Loop input
90
The longth you
Accuracy
80
generated is long' please regenerate shorterlonger summmary:
70
the length of summmaty result meets the requirement!
No
Prompt instruction
40
50
60
70
80
Yes
Percentage of orignal length (%)
Figure 5: Performance comparison of token compres- sion methods. The horizontal dashed lines represent accuracy from the methods that do not output variable token lengths.
Final result
Figure 4: Self-instruct training data generation.
We conduct experiments FRDB which con- tains 1,000 maternity/infant food related domain- specific multiple-choic questions, each of which only contains one correct answer. Three sentence- compression methods are evaluated our experi- ments: 1) random deletion which randomly deletes words from a sentence, 2) summarization compres- sion, and 3) semantic compression.
j percentile elements:
C (w L pj}
(2)
The words corresponding to the elements in set
Lj are extracted as the context for the LLM.
5 Evaluation
5.1 Experiment Setup
The experiment results are shown in Figure 5, To construct the baseline, we evaluate the GPT-3.5- turbo performance two configurations: without retrieval and with retrieval but without token com- pression. We observe from the results that, once additional information is fed into the model con- text, the accuracy immediately improves, from 51 % to 90.2%. This shows the benefit that retrieving do- main information brings to the LLM.
We conduct studies on the FRDB datasct. The sum- marization compression is based on the pre-trained 105-mulilinguallXXSSnn (Xue et al., 2020). The maximum input and output length is set to 512, the learning rate is 2e-5, the number of epochs is set to 10, the batch size is 2, and the rest of the settings follow the default settings from the models. The training of the mT5 models are con- ducted on the server that contains an AMD EPYC 7763 CPU, 256GB RAM, and NVIDIA 4090 GPU with 24GB memory. Following the method de- scribed in section 4.2.1, three shortened versions of the summarization dataset with 30%, 50%, and 70% of its original length are generated. Each version is used fine-tune mT5 summariza- tion model. The pre-trained multi-lingual sentence- transformer (Reimers and Gurevych, 2020) used as an embedding encoder for semantic compres- sion. Three compressed sentences in 30%, 50%,
Next, we compare using the original pre-trained mT5 based model with using the version where the model is fine-tuned on the datasets generated by self self-instruction. With fine-tuning, the accuracy improves dramatically from 60% to 90.6%. The summarization model without fine-tuning output has average length of 15 words, compare with 88, 46, and 21 words for our 70%, 50%, and 30% length dataset fine-tuned model respectively. It shows that the summarization model might remove critical information by mistake if the input text

---

is on topic that the summarization model is not familiar with. A small fine-tuning set (400 sam- ples) is enough to help the summarization model adapting to the new domain.
5.4 Information Entropy Accuracy
In the next experiment, we investigate the impact the information entropy on the accuracy of different token compression methods. We measure the word frequency from the FRDB dataset and use it as a probabilistic model to calculate the information en- tropy of each word. The mean word information entropy is calculated on each sentence normal- the entropy. The results for the three token compression methods is shown in Figure 6. The random deletion method removes words randomly which leads to the average information entropy for different sentence lengths approximately the same. On the other hand, the semantic compression algo- rithm removes the words that have less semantic meaning. Our experiment shows that, the average information entropy goes lower sentences be- come shorter, indicating that the sentence becomes less compressible. Additionally, the average word information entropy is positively correlated with the accuracy when semantic compression is used, showing that higher information will benefit the model performance. On the contrary, the summa- rization compression shows distinct phenomenon. Instead of naively removing words, the summariza- tion compression compresses the original sentences into different lengths by rephrasing sentences. By doing this, the shortened sentences obtain lower average information entropy but the accuracy stays at similar level in comparison with the original sentences. The lower average information entropy indicates that sentences become more condensed but the semantic of the sentences stays approxi- mately the same.
The second compression method propose is semantic compression. It has better flexibility to generate variable lengths of tokens but delivers lower performance than the summarization com- pression method. The baseline for our experiment is random deletion where we randomly delete certain percentage of words. This random deletion consistently scores lower performance than both of proposed a algorithms.
5.3 Cost Reduction Comparison
Our proposed summarization compression method is tested server with one NVIDIA 4090 GPU (24GB), 32GB RAM and Intel i7 CPU. The run- time for such system is on average 0.383s per sam- plc. A similar server on the AWS, i.c., g5. g5.2xlarge, has an A10G GPU, 32GB RAM, and 8-core vCPU and the hourly price for such a system is $0,485. In one hour, such system can process approximately 9,400 summarizations, so the cost per summariza- tion is 6e-5, Assume the system utilization rate is only 50%, means that the cost per summariza- tion is $1.0319c-04,
The GPT-3.5-turbo itself does not have enough knowledge to precisely answer the question from dataset (5 % accuracy if without additional context in the prompt). Thus, both the common reurieedl-augmeented GPT-3.5-turbo and our sys- tem (retrieval-auu -ugmenetee GPT-3.5-turbe with addi- tional token compression) require dense retrieval to reach acceptable performance, and the retrieval cost, which is $0.0001 per 1,000 query tokens, is the same for both systems. Since the original ques-
95
tions are typically short, average length of them is about 128 tokens, which translates into $1.2500e-05 per question for the
assume that the
Accuracy
80
retrieval. Assuming at full size the input has 512 to-
S.PT-3.5-turbo with summarization line-tuned GPT-3 5-turbo with somantic compression GPT-3. 5-ttrb with random delation
kens and the output has 64 tokens, the total cost for the common retrievallaugmented GPT-3.5-turbo (for both retrieval and QA using API calls) is about $8.9050e-04 per question. In comparison, our algo- rithm compress the retrieved context of GPT- 3.5-turbo to 35% of its original length, which trans- lates into an averagely 50% of reduction during the APT calls. Thus, The cost using our token compres- sion system is around S.. 1869c-04 per question. reduces the overall costs by 30%.
5.4
5.6
5.8
6
6.2
6.4
6.6
6.8
Average information entropy
Figure 6: Impact of average information entropy on accuracy.
Next, investigate the impact of cosine similar- ity between original and compressed sentences on accuracy and we find a positive correlation between accuracy and cosine similarity value. It indicates

---

closer to the original semantic meaning would pro- duce better accuracy.
rate. The result is shown in Fig. 9. We find that, there positive correlation (0.31) between our selected input and compression rate with an RMSE of 1 1.4% and R --squared of 9.6%. This indicates the compression rate of each sentence can be estimated prior to the summarization process.
GPT 1-36-ttubb with summarization fine-tuned
88
-PT-b.6-uubo with sementic comprecsion
Accuracy
GPT-3.6-turo with random deletion
80
75
0.8
70
0.6
Compression
0.4
0.6
0.65
0.7
0.75
0.8
0.85
0.9
0.95
Average cosine similarity between orginal and compressed sentence
0.2
Figure 7: Impact of cosine similarity between original and compressed sentence on accuracy.
Adjusted data Fit: y- 06008828'''>
-0.2
cont. bounds
-04
8 Adjusted input features
7
5.5 Summarization Statistics
Our summarization model is built based on the pre-trained mT5 model and fine-tuned on our self- instruct generated dataset. We generate three datasets which are 30%, 50%, and 70% of the length compared to their original text. Three differ- ent models are fine-tuned independently. Figure 8 shows the distribution of sentence length from three fine-tuned models. At 70% compression, the summary text shifts from an average of 88 words to 46 words for 50% length and 21 words for 30% length.
Figure Visualization of linear regres- sion on predicting compression rate.
5.7 Ablation Study
From our experiments, we find the summarization compression method delivers the best performance. Here we compare different retrieval methods and investigate what the optimal settings are. Four con- figurations are evaluated: embedding only, BM25 only, embedding first then BM25, and BM25 first then embedding. The first two configurations are straightforward: in the third configuration, we ap- ply the embedding baased method to extract the top- q results, then apply BM25 to extract top-k results from the q results where g 2 k. In the fourth config- uration, we reverse the order where we first extract the top-q results using BM25 and then extract the top-k results from the & results using embedding- based methods. We set h 3 based previous experiments. The evaluation results are shown in Table We find the straightforward dense retrieval approach achieves the best performance.
0.3
Original sentences
0.25
Summarize 70% length
Summarize to 50% length
Summarize to 30% length
Probabilty
0.2
0.15
0.1
0.05
100
150
Length of sentence
Conclusion
Figure 8: Distribution of sentence lengths with different summarization lengths.
In this paper, we propose two methods that re- duce the token size for retrieval-augmmented LLM. Additionally, we propose a food recommenda- tion dataset contains domain-specific knowledge to benchmark the performance of retrieval-augmmente LLM performance on the GPT-3.5-turbo. We care- fully select subset that focuses on 200 types of food recommendations for maternity and infant people. Without retrieval augmentation, the com- mercial GPT-3.5-turke model is only able to get
5.6 Compression Rate vs. Entropy
We conduct a study on the compression rate of our mT5 model fine-tuned with the 30% length dataset. Our input consists of 1) the original length of the sentence, 2) average word entropy, and 3) accu- mulated word entropy. We deploy a simple linear regression algorithm to predict the compression

---

Method
Top-q
Acc. (%)
References
Embedding
n/a
90.9
Dhananjay Ashok and Zachary C Lipton. 2023. Prompt- NER: Prompting For Named Entity Recognition. arXiv preprint arXiv:2305. 15444.
Embedding+BBM25
10
89.7
BMb
100
88.0
BM25
n/a
74.7
E Bolton, D Hall, M Yasunaga, T Lee, C Manning, and P Liang. 2022. Stanford crfm introduces pubmedgpt 2.7 b.
BM25+Embedding
10
89.3
BM25+Embedding
100
89.8
Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33: 13:1877-1901.
Table Evaluation of different retrieval algorithm con- figurations. The top-q column indicates the value used the search. Ace. stands for accuracy.
51% % percent of the question right, compared to 90.2% with retrieved context. We use this 90.2% as our goal and compare the performance of different token compression algorithms. Our summarization- based approach achieves the best performance, reaching 90.5% of accuracy with 65% token re- duction on the retrieved context. It indicates that the summarized text maintains similar level of critical information but with significantly shorter length, showing promising ability of our proposed method. The semantic compression method can further remove the words that have lower seman- tic meaning and provides a more flexible way to trade-off the length of sentences with accuracy.
Lingjiao Chen, Matei Zaharia, and James Zou, 2023. FrugaIGPT: How to Use Large Language Models While Reducing Cost and Improving Performance. arXiv preprint 2rX5v:2305.05176
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.022 223..0231..
Jacob Devlin, Ming-Wes Weig Chang, Kenton Lee, and Kristina Toutanova. 2018, Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint 1xXiv:180.04400.
Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Beltagy, Doug Downey, and Noah Smith. 2020. Don't stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting the Association for Computational Linguistics, pages 1342-8366 Online. Association for Computational Linguistics.
Limitations
The goal of our model is to reduce the token size for retrieval-augmented LLMs. The compression rate is determined based on the methods' ability to condense sentences while preserving as much of their essential information as possible. the sen- tences are already short and compacted in meaning, the compression rate won't 't be able to be low if we want to maintain most of the critical information within the sentence. Our algorithm is designed for large commercial LLMs and smaller open-source LLMs may not experience similar levels of perfor- mance gain.
Po-Sen Huang. Xiaodong Hc, Jianfeng Gao, Li Deng, Alex Acero, and Larry Heck. 2013. Learning deep structured semantic models for web search using clickthrough data. In Proceedings of the 22nd ACM international conference on Information & Knowl- edge Munagement, pages 2333-2333
Tatsuro Inaba, Hirokazu Kiyomaru, Fei Cheng, and Sadao Kurohashi. 2023. MultiTool-CoT: GPT-3 Can Use Multiple External Tools with Chain of Thought Prompting. arXiv preprins 23V:22305.168%
Ethics Statement
Our proposed methods are designed solely for the goal of reducing cost while maintaining the perfor- mance using commercial LLMs through API calls. The algorithms are not designed for activities that are in violation of human rights or with military purposes. The data we collected for our proposed dataset does not contain any privacy information.
Gautier Izacard, Patrick Lewis, Maria Lomeli, Lu- cas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi- Yu, Armand Joulin, Sehastian Riedel, and Edouard Grave. 2022. Few-shot learning with trieval augmented language models. arXiv preprint arXiv:2208.03299,
Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2019. Billion-scale similarity search with gpus. IEEE Trunsactions on Big Data, 7(3):335-547.

---

Karen Sparck Jones, 1973. Index term weighting, In- formation storage and retrieval, 9(11):619-63 633.
Nils Reimers and Iryna Gurevych. 2020. Making Monolingual Sentence Embeddings Multilingual us- ing Knowledge Distillation, In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4512-4525, Online. Association for Computational Linguistics.
Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. arXiv preprint arXiv:2004.04900
Stephen Robertson, Hugo Zaragoza, et al. 2009. The probabilistic relevance framework: BM25 and be- yond. Foundations and Trends@ in Information Re- trieval, 344:3333-389
Patrick S. H. Lewis, Ethan Perez, Aleksandra Pik- tus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe generation for
Stephen E Robertson, Steve Walker, Susan Jones, Micheline M Bannockk-Beuulien Mike Gatford, et al. 1995. Okapi TREC-3. Nist Special Publication Sp, 109:109.
Kiela. 2020. retrieval-augmeene knowledge-intensivee NLP tasks. In Advances in Neu- ral Information Processing Systems 33: Annual Con- ference on Neurul Information P Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual.
Bishal Santra, Sakya Basak, Abhinandan De. Manish Gupta, and Pawan Goyal. 2023, Frugal prompting for Dialog Models, arXiv preprint arXiv:2305. 149.,
Guang Lu, Sylvia B Larcher, and Tu Tran. 2023. Hy- bridLong Long Document Summarization using C2F-FAR and ChatGPT: Practical Study. arXiv preprint arXiv:230.00166.
Teven Le Scao, Angela Fan, Christopher Akiki, El- lie Pavlick, Suzana Ilié, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022. Bloom: A 176b- parameter open-access multilingual language model. arXiv preprint arXiv:221
Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Hannanch Hajishirzi, and Daniel Khashabi. 2022. When Not to Trust Language Models: In- vestigating Effectiveness and Limitations of Paramet- ric and non-Parametric Memories. arXiv preprint arXiv:2212.10511.
Jamiel Sheikh. 2023. Bloomberg Uses Its Vast Data To Create New Finance AL.
Ha- Thanh Nguyen. 2023. A Brief Report on LPwGPT 1.0: A Virtual Legal Assistant Based on GPT-3, arXiv preprint arXiv:230..0572.
Weijia Shi, Sewon Min, Michihiro Yasunaga, Min- joon Sco, Rich James, Mike Lewis, Luke Zettle- moyer, and Wen-tau Yih. 2023. Replug: Retrieval- augmented black-box language models. arXiv preprint arXiv:2301. 12652.
Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gus- tavo Hernández Ábrego, Ji Ma, Vincent Y Zhao, Yi Luan, Keith B Hall, Ming-Wer Chang, et al. 2021. Large dual encoders are generalizable retriev- ers. arXiv preprint arXiv:2112.07899.
Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. 2021. Retrieval Augmentation Reduces Hallucination in Conversation. In Find- ings of the Association for Computational Linguistics: EMNLP 2021, Virtual Event / Punta Cana, Domini- can Republic, 16-20 November, 2021, pages 3784- 3803. Association for Computational Linguistics.
OpenAI. 2022. New and improved embedding model.
OpenAl. 2023a. GPT-4 Technical Report. CoRR, abs/2303.08774.
David Soong, Sriram Sridhar, Han Si, Jan-Samuel Wagner, Ana Caroline Costa Sá. Christina Y Yu, Kubra Karagoz, Meijian Guan. Hisham Hamadeh, and Brandon Higgs. 2023. Improving accu- racy of GPT-3/4 results on biomedical data using a augmentd-- language model. arXiv preprint arXiv:2305.1 17116.
OpenAl. 2023b. Introducing ChatGPT.
OpenAI. 2023c. Introducing ChatGPT and Whisper APIs.
Sundar Pichai. 2023, An important next step Al journey.
Karen Sparck Jones. 1972. A statistical interpretation of term specificity and its application in retrieval. Journal of documentation, 28(1):11-21.
Colin Raffel, Noam Shazeer, Adam Roberts, Kather- ine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67.
James Thorne, Majid Yazdani, Marzieh Saeidi, Fab- rizio Silvestri, Sebastian Riedel, and Alon Y. Levy. 2021. From Natural Language Processing to Neural Databases. Proc. VLDB Endow. 1436):0333
Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Brown and Yoav Shoham. 2023. In-context retrieval-augee auggmeted lan- guage models. arXiv preprini 2302: 00083.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro,

---

Faisal Azhar, al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint arXiv:2302. 13971.
Shuhe Wang, Xiaofei Sun, Xiaoya Li, Rongbin Ouyang, Fei Wu, Tianwei Zhang, Jiwei Li, and Guoyin Wang. 2023. Gpt-ner: Named entity recognition via large language models. arXiv preprint 2304. 10428.
Yizhong Wang. Yeganeh Kordi, Swaroop Mishra, Al- isa Liu, Noah A Smith. Daniel Khashabi, and Han- naneh Hajishirzi. 2022. Sclf-Instruct: Aligning Lan- guage Model with Self Generated Instructions. arXiv preprint X2v:22212.10560.
Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prabhanjan Kam- badur, David Rosenberg, and Gideon Mann. 2023. Bloomberggpt: A large language model for finance. arXiv preprint arXiv:2303. 17564.
Tong Xiang, Liangzhi Li, Wangyue Li, Mingbai Bai, Lu Wei, Bowen Wang. and Noa Garcia. 2023. CARE-MI: chinese benchmark for misinformation evaluation in maternity and infant care. CoRR,
abs/2307.01458.
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2020. mT5: A massively multilingual pre-trained text-to-text transformer. arXiv preprint arXiv:2010.I 1934.
Xinyang Yi, Ji Yang. Lichan Hong, Derek Zhiyuan Cheng, Lukasz Heldt, Aditee Kumthekar, Zhe Zhao, Li Wei, and Fd Chi. 2019. neural modeling for large corpus item recommenda- tions. In Proceedings of the 13th ACM Conference Recommender Systems, pages 269-277.
HongChien Yu, Chenyan Xiong, and Jamie Callan. 2021. Improving query representations for dense retrieval with pseudo relevance feedhack. In CIKM '1: The 30th ACM International Conference Information and Knowledge Management, Virtual Event, Queens- land, Australia, November 5, 2021, pages 3592- 3596. ACM.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher De- wan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. Opt: Open pre-trained transformer language models. arXiv preprint arxi::2005.0166.
Wenxuan Zhang, Yue Deng, Bing Liu, Sinno Jialin Pan, and Lidong Bing. 2023. Sentiment Analysis in the Era of Large Language Models: Reality Check. arXiv preprint 23X5:23355.5000

---

Appendix
A Example question for FRDB dataset
Is it suitable for an infant to consume/have hot chocolate? (1) Recommend, (2) Neutral, (3) Avoid
1. Food is beneficial to the current stage, and excessive consumption will not cause physical abnormalities.
2. Advocate healthy lifestyle and advise users to intake relatively limited amount of foods, including foods with excessive salt, oil, sugar, and so on.
3. There is evidence that it will cause harm after cating; Authoritative literature points out that eating is forbidden at a certain stage.
B Sentence compression examples
Examples are shown in Table 6.
C QA examples
Examples of QA are shown in Table 7.
D Knowledge examples
Examples of knowledge utilized for answering the questions are shown in Table

---

Dataset
Original
Compressed
In the first-instance procedure of public prosecution case, if the participants or observers disrupt the courtroom order, the presid- ing judge shall handle the situa- tion according to the law. Unau- thorized recording, videotaping, photographing, or disseminating trial proceedings through mail, blog, or microblog may lead the temporary seizure of storage media or relevant equipment.
Babies are strictly prohibited from consuming donkey meat buns. The reason is that don- key meat buns usually contain a large amount of spices and seasonings, with high salt con- tent, which will increase the metabolic burden the baby's kidneys. Furthermore, infants and young children are in critical period for developing taste preferences, and frequently consuming donkey meat buns may affect the formation their taste buds. It is not recom- mended to feed babies donkey meat buns.
FRDB
Table Token compression examples. Both original and compressed sentences are translated into English from Chinese.

---

Questions
Questions (English translation)
Answer
According to "Maternity should eat less foie gras. The reason is: moderate consump- tion of foie gras after childbirth can promote wound healing, make up for blood loss during childbirth, rebuild liver iron reserves, improve breast milk quality. It can also improve the immunity of mothers, and has an antioxidant and anti-aging effect. It is recommended that postpartum mothers can consume it in moder- ation." " Which of the following options is most suitable to describe the behavior of postpartum mothers consuming/eating foie gras? Options: (1) Recommend, (2) Neutral, (3) Avoid Regarding "Females preparing for pregnancy can eat curry. The reason is that curry is a mixed seasoning that can regulate intestinal movement, increase appetite, and its spiciness varies according to the ingredients. Curry that is too spicy can stimulate the stomach to some extent. Females preparing for pregnancy can choose curry with appropriate spiciness based their own taste preferences." Which of the following options is most suitable to describe the behavior of females preparing for preg- nancy consuming/cating curry? Options: (1) Recommend, (2) Neutral, (3) Avoid
#F OF
: M
# # FF
A
a. М.
11
Neutral
- # _
H. " 33
ド -M: (1) #
(2) x
F 4 E
MN 1 W
19
A 5) A :#
TE NT#
E. , E
Recommend
MUMTE " $
A U0N Y 13
/
#
-
: (1) # (2) $V. (3)
Regarding "Babies at the age of 6 months should consume less corn juice. The reason is that corn juice is rich in vitamins, minerals and carbohydrates, which can provide energy for babies, promote their growth and develop- ment, and has a high absorption rate. Babies over months old can consume it in modera- tion, but be mindful that freshly squeezed corn juice should not contain added sugar to pre- vent excessive intake of sugar, which could be detrimental to baby's 's oral health." Which of the following options is most suitable to describe the behavior of a 6-month-old baby consuning/drinking corn juice? Options: (1) Recommend, (2) Neutral, (3) Avoid
*T "6
K*II. -: 1 _m
E H *
* TH
A
Neutral
A $
A , 6 & H/
#
#
M. (1) # (2) #Z. (3)
Table QA examples. For all examples, the answers are chosen from 3 options, (1) Recommend, (2) Neutral, and (3) Avoid.

---

Knowledge
Knowledge (English translation)
Females preparing for pregnancy can eat mashed pota- toes. The reason is that it is safe for this group to con- sume mashed potatoes. However, mashed potatoes have high glycemic index, it is recommended not to eat too much at once.
RA iix -
Σ 13
Babies should not eat raw fish slices. The reason is that fish contains a large amount of unsaturated fatty acids (cspecially DHA), which can help promote the development of the baby's brain and vision. However, if raw fish slices are not properly processed, they can easily become contaminated with bacteria and parasites, posing health risks to babies. Therefore, it is not recommended to feed raw fish slices babies.
b. # M6 150
H DHA)
#
Pregnant women can eat tilapia. The reason that tilapia is rich in unsaturated fatty acids, protein and various amino acids, which are easily digested and absorbed by the human body. It can provide pregnant women with various nutrients to help promote fetal bone growth and development of the nervous system.
$
A
, #
KAM
Table 8: Examples of knowledge that are utilized for answering the questions.


Reviews:
[{"review_id": "0NEubRi8AI", "reviewer": "Reviewer_Z9sQ", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your comments. I would like to address the comments below.\n\n1.\tThe process of data collection and annotation for the proposed dataset (FRDB) lacks details;\n\nThis FRDB dataset is collected from a combination of frequently asked questions from social platforms and verified by our nutrition experts. We will add this additional detail in the paper. The full 1000 question answer FRDB dataset is available in the supplementary material. The retrieval knowledge database that contains 7,588 entries is subject to our internal approval. We will release it once it is approved.\n\n2.\tThe experiments are missing details, such as how the FRDB dataset was divided into train/dev/test splits;\nThe question-answer part of FRDB is included in the supplementary. We are doing retrieval augmented generation leveraging the power of ChatGPT. So, all the QA pairs are test sets. \n\n3.\tThe authors claim they use mT5 to compress context and spend less money on OpenAI API. However, what is the cost of summarizing context with mT5? A comparison is needed. The same applies to the embedding-based retriever, which uses again OpenAI embedding APIs;\nLet me do a cost study between renting an AWS server and using the GPT3.5 service. \n\nOur mT5 system was tested on a server with one Nvidia 4090 GPU (24GB), 32GB RAM and Intel i7 CPU. The runtime for such a system is on average 0.383s per sample.  On the AWS the similar g5.2xlarge server has A10G GPU, 32GB RAM, and 8-core vCPU. The hourly price for such a system is $0.485. In one hour, such a system can process about 9,400 summarizations, so the cost per summarization is $5.16e-5. Let’s assume the system utilization rate is only 50%, which means per summarization is $1.0319e-04.\n\nThe GPT3.5 does not have the knowledge to answer the question from our dataset (51% accuracy without supplying the context in the prompt). So, both our system and the gpt3.5 require embedding-based retrieval, so such cost is the same for both systems which is $0.0001/1k tokens. The questions are typically short and we assume the length is about 128 tokens, which translates into $1.2500e-05 per question.\n\nAssuming at full size the input is 512 tokens and the output is 64 tokens. We can calculate each total QA cost (embedding+QA) about 0.0015*0.5+0.002*0.064+1.2500e-05 =8.9050e-04 per question.\n\nOur algorithm can compress the context to 35% of its original length, which translates into an overall 50% of token savings on average. So, the new costs for answer-questions using our token compression system are 0.0015*0.25+0.002*0.064+1.0319e-04+1.2500e-05 = $6.1869e-04. It reduces the overall QA costs (embedding+QA) by 30%. \n\n\n4.\tThe generation hyper-parameters used for mT5 are missing, such as greedy search, beam search, ...\n\nDue to the page length limit, the hyper-parameters were not included in the main text. It is available in our supplementary material. We have attached a fully functioning training, and inference code with all the parameters. You are also welcome to try out the system. \n\n5.\tThe authors did not show which prompt they used to add context to OpenAI GPT, reducing reproducibility;\n\nWe have attached some prompt examples in the supplementary material Tables 6, and 7. Also, I noticed, that we missed the multiple-choice option part in the prompt sample. I will revise this in the supplementary material.\n\n6.\tThe evaluation is performed on a single and very small dataset in the health domain, thus reducing the possibility to apply the proposed techniques to other topics;\n\nThanks for your suggestion. Our goal is to test out a dataset that is never trained in GPT3.5’s system, due to the vast large training set it has, this kind of dataset is hard to find. So, we picked this domain specific dataset on food recommendation. We agree that more experiments should be running to verify other domains also can benefit from our algorithm. We will provide some additional experiments in the camera-ready version.\n\n7.\tIn Figure 6, is it not clear how a Average information entropy of ~6.5 can lead to very different accuracies at the same time.\n\nThis phenomenon appears in the random deletion of words in the context. From our experiment, when we randomly delete words, the average information entropy per word does not change much but the semantic meaning of the sentence changed significantly. This experiment shows semantic meaning is the leading factor in impact on the accuracy.\n\n8.\tSome claims are not explained. For example, on line 288, the authors say that \"LLM could only output fixed-length summary due to the fixed length of its training data.\". What about length penalties and beam search to select summaries that better fit the given requirements?\n\nOur goal is to study the impact of compressed tokens on the QA tasks. In our experiment, we tried three different compression rates which compressed the sentence to 30%, 50%, and 70% of its original length. To achieve such a control experiment, we utilized the power of self-instruct learning from GPT3.5 to construct a dataset to fine-tune the summarization model with sentences compressed to 35%, 43%, and 60% of the original length. Length penalties and beam search are alternatives for controlling the length, but our experiment shows the summarization model fine-tuned on a small set of self-instruct generated data significantly outperformed the version without fine-tuning. In our case, the fine-tuning process is merely a necessity, and after fine-tuning we obtained a shorter length of summary which is our goal. So we did not have a strong motivation to further control the output length of the fine-tuned model.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper provides an interesting study to add context to LLMs for better responses and methodologies to reduce input token length while maintaining a comparable level of performance;\n- The results show that context really helps in identifying the correct answer to a question, improving accuracy.", "reasons_to_reject": "- The process of data collection and annotation for the proposed dataset (FRDB) lack details;\n- The experiments are missing details, such as how the FRDB dataset was divided in train/dev/test splits;\n- The authors claim they use mT5 to compress context and spend less money on OpenAI API. However, what is the cost of summarizing context with mT5? A comparison is needed. The same applies to the embedding-based retriever, which uses again OpenAI embedding APIs;\n- The generation hyper-parameters used for mT5 are missing, such as greedy search, beam search, ...\n- The authors did not show which prompt they used to add context to OpenAI GPT, reducing reproducibility;\n- The evaluation is performed on a single and very small dataset in the health domain, thus reducing the possibility to apply the proposed techniques to other topics;\n- In Figure 6, is it not clear how a Average information entropy of ~6.5 can lead to very different accuracies at the same time.\n- Some claims are not explained. For example, on line 288, the authors say that \"LLM could only output fixed-length summary due to the fixed length of its training data.\". What about length penalties and beam search to select a summaries that better fit the given requirements?", "questions_to_authors": "", "paper_topic_and_main_contributions": "In this study, the researchers investigate the efficacy of Language Models (LLMs) in two scenarios: (i) when enhanced with supplementary context to improve responses, and (ii) when input context is shortened using various techniques, such as summarization and the removal of low informative tokens.\n\nThe results are promising and the described techniques provide important gains in accuracy. However, the paper lacks details about several \nexperiments and the performance are evaluated only with a small and domain specific dataset.\n\n**Update after rebuttal**\n\nThe authors addressed some of my concerns, however:\n- a lot of important details and hyper-parameters are still hidden in the supplementary materials, while they should be included in the text or appendix of the paper;\n- the comparison between the cost of using the proposed solution with compressed context and using only ChatGPT should be included at least in the appendix of the paper;\n- the evaluation is still performed on a single small dataset, reducing the possibility of applying the proposed technique to other domains;\n- the paper is still full of typos and hard to follow\n\nI've updated the scores accordingly, increasing the overall excitement because the topic and the solutions addresses by this paper are interesting. I kept a soundness of 3 for the reasons explained above.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Exploiting Context in LLMs:\nDi Liello, L., Garg, S., & Moschitti, A. (2023). Context-Aware Transformer Pre-Training for Answer Sentence Selection. (https://arxiv.org/abs/2305.15358)", "Typos_Grammar_Style_and_Presentation_Improvements": "- The paper is not well written and not easy to follow;\n- Some sentences are not complete, for example on line 175 or 508;\n- The paper contains many grammar typos."}, "extra_text": {}}, {"review_id": "Wk0ceMxgmm", "reviewer": "Reviewer_9Fyp", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1.\tConsidering that it’s a classification task, other performance metrics need to be studied. Increase in overall inference time needs to be studied. Overall cost and effort impact need to studied.\n\nThanks for your comments. We have conducted calculations on additional performance metrics.\n\nAcc: 0.909\nWeighted F1 Score: 0.908442161770658\nMacro F1 Score: 0.8865993509859474\nMicro F1 Score: 0.909\nMacro Recall Score: 0.8641033143866387\nMicro Recall Score: 0.909\nWeighted Recall Score: 0.909\nMacro Precision Score: 0.9195618277905991\nMicro Precision Score: 0.909\nWeighted Precision Score: 0.9130815891627097\n\nWe will add an additional cost study in the revised draft. The other reviewer also asked the study in terms of costs. In terms of inference time, the difference between the use of retrieval augment with and without the token compression is about 0.383s. The other part is the same. The average time for embedding creating using OpenAI API is 0.468s, and the average time for gpt3.5 is 0.813s. \n\nLet me do a cost study between renting an AWS server and using the GPT3.5 service. \n\nOur mT5 system was tested on a server with one Nvidia 4090 GPU (24GB), 32GB RAM and Intel i7 CPU. The runtime for such a system is on average 0.383s per sample.  On the AWS the similar g5.2xlarge server has A10G GPU, 32GB RAM, and 8-core vCPU. The hourly price for such a system is $0.485. In one hour, such a system can process about 9,400 summarizations, so the cost per summarization is $5.16e-5. Let’s assume the system utilization rate is only 50%, which means per summarization is $1.0319e-04.\n\nThe GPT3.5 does not have the knowledge to answer the question from our dataset (51% accuracy without supplying the context in the prompt). So, both our system and the gpt3.5 require embedding-based retrieval, so such cost is the same for both systems which is $0.0001/1k tokens. The questions are usually short and we assume the length is about 128 tokens, which translates into $1.2500e-05 per question.\n\nAssuming at full size the input is 512 tokens and the output is 64 tokens. We can calculate each total QA cost (embedding+QA) about 0.0015*0.5+0.002*0.064+1.2500e-05 =8.9050e-04 per question.\n\nOur algorithm can compress the context to 35% of its original length, which translates into an overall 50% token saving on average. So, the new costs for answer-question using our token compression system are 0.0015*0.25+0.002*0.064+1.0319e-04+1.2500e-05 = $6.1869e-04. It reduces the overall QA costs (embedding+QA) by 30%.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Relevant solution for today’s LLM applications.\n2. The solution is explained well.\n3. Good set of experiments were used to demonstrate the benefits of suggested approaches.\n4. Overall cost analysis, difference in before/after will be added in final version.\n5. Increase in overall inference time will be added in final version.\n6. Considering that it’s a classification task, other performance metrics will be added in final version.", "reasons_to_reject": "1. The increase in time and effort for data creation and summarization model training needs to studied.\n2. Generality for different domains and dataset sizes should be studied.", "questions_to_authors": "", "paper_topic_and_main_contributions": "API calls to commercial large language models are very costly. Data augmentation is used provide additional context to these calls. This increases the cost as it increases the number of tokens in the call.\nTo reduce the cost, author’s propose token compression scheme for such retrieval augmented large language model. They call it Token Compression Retrieval Augmented Large Language Model (TCRA-LLM).\nAuthor’s present 2 approaches to compress the additional relevant information retrieved from the knowledge database.\n1.\tSummarization – generate summaries of different lengths. Summarization model is trained using the data generated by self-instruct model.\n2.\tSemantic compression - words with less semantic impact  (such as filler words) are removed based on the Euclidian distance between the original and modified sentence embeddings.\nTo improve reliability of additional context from knowledge databases, author’s presented next-sentence prediction (NSP) to rank the retrieved information.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "PqgcVCg7vM", "reviewer": "Reviewer_VUd2", "summary": "", "questions": "QA: Could you please elaborate more on the generalization of the model? (refer to first concern on previous section)", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1.\tA weak novelty for a long paper. They just combined two already proposed methods for token compression, and the main improvement is comming from fine-tuning summarizer.The generalization of the approach to other domains is not straight-forward, as the modules should be fine-tuned on the new domain.\n\nThanks for your comments. We acknowledge our paper is an application paper that focuses on engineering a system to solve the problem that appears in the business deployment. During our daily business operation, we are experiencing high operating costs where every penny we save will translate into a penny of profit. During the construction of the system, we put a lot of engineering thought into it, where a higher system complexity will require higher server costs which may eat away the potential savings. By balancing the server cost and algorithm performance, we found an easy yet efficient method to reduce the cost by 30% which is significant for the survival of our business. Creating a generalizable solution for the business environment is always not easy, we need to balance the cost and performance of the system. From our experiment, a model fine-tuned for each domain is optimal to balance the costs and performance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. A novel combination of token compression methods, including summarization and semantic compression to reduce the input tokens of LLMs.\n2. Proposing a new dataset for food recommendation which includes the domain of recomending foods that are safe to consume for women before/during/after their pregnancy or infant.\n3. Achieving significant token compression while maintaining the performance.", "reasons_to_reject": "1. A weak novelty for a long paper. They just combined two already proposed methods for token compression, and the main improvement is comming from fine-tuning summarizer.\n2. The generalization of the approach to other domains is not straight-forward, as the modules should be fine-tuned on the new domain.", "questions_to_authors": "QA: Could you please elaborate more on the generalization of the model? (refer to first concern on previous section)", "paper_topic_and_main_contributions": "In this paper, they proposed two approaches (summarization and semantic compression) to compress the token size for retrieval augmented LLMs. They also proposed a food recommendation dataset which includes the domain of recomending foods that are safe to consume for women before/during/after their pregnancy or infant. By applying these two techniques, they reached a significant amount of token compression while maintaing the performance (e.g. 65% token reduction while havinf 0.3% performance improvement).", "excitement": 3, "reproducibility": 5, "ethical_concerns": "Yes", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Justification_for_Ethical_Concerns": "Further clarification about the collection process of FRDB dataset is required to check the ethical concern."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper addresses a task of cost reduction when creating application using LLM APIs (for example, from OpenAI). This is indeed a bottleneck for many applications: while LLMs achieve impressive results, encoding user-generated information as context can get costly. In this sense the study provides an important solution to today's problems (see reviewers 1 and 2). The solution itself is based on existing techniques (cf. reviewer 3), however, the application is novel and impactful. Altogether, this study might be more appropriate for the industrial/applications track.

The experiments are convincing, showing a good compression rate with virtually no performance loss.

However, there are several shortcomings:

1) a more detailed cost analysis (see reviewer 1). this is important as the cost is the main focus here. this issue has been addressed thoroughly in the rebuttal -- and has to be incorporated into the next version
2) presentation. a lot of information (e.g. hyperparameters etc) is presented in the supplementary material and should be upgraded to the appendix; the paper needs proofreading. this is a minor issue that can be fixed for the next version
3) the experiments are run on a single domain. it is not clear how well the proposed solution generalize to other domains/settings (after all, the main motivation behind this study is to reduce cost for user domains, so the paper should focus on diverse use cases). This is a major issue raised by all the reviewers.
Decision: Accept-Findings