LOCALRQA: From Generating Data to Locally Training, Testing, and Deploying Retrieval-Augmented QA Systems (2024)

Abstract: Retrieval-augmented question-answering systems combine retrieval techniques with large language models to provide answers that are more accurate and informative. Many existing toolkits allow users to quickly build such systems using off-the-shelf models, but they fall short in supporting researchers and developers to customize the model training, testing, and deployment process. We propose LOCALRQA1, an open-source toolkit that features a wide selection of model training algorithms, evaluation methods, and deployment tools curated from the latest research. As a showcase, we build QA systems using online documentation obtained from Databricks and Faire’s websites. We find 7B-models trained and deployed using LOCALRQA reach a similar performance compared to using OpenAI’s text-ada-002 and GPT-4-turbo.

heading: 1 Introduction
text: Retrieval-augmented question-answering (RQA) systems enhance large language models (LLMs) by enabling them to search through a large collection of documents before answering a user’s query. These systems have shown improved performance in providing more accurate, informative, and factually grounded answers compared to using LLMs alone (Guu et al., 2020; Izacard et al., 2022; Shi et al., 2023). Many existing toolkits, such as LlamaIndex (Liu, 2022) and LangChain (Chase, 2022), allow users to quickly build such an RQA system using off-the-shelf models such as text-ada-002 (OpenAI, 2022a) and GPT-4 (OpenAI, 2023). However, developers often find it costly to rely on these paid services, but also face difficulties to train/deploy smaller models with competitive performance. Researchers face even greater hurdles: they need to modify models/training algorithms, compare against prior work, and obtain
* denotes equal contribution. 1GitHub: https://github.com/jasonyux/LocalRQA,
YouTube: https://youtu.be/MEtFIcw7clY.
human evaluation for their RQA system — all of which are largely neglected by existing toolkits.
We introduce LOCALRQA, an open-source toolkit that enables researchers and developers to easily train, test, and deploy RQA systems using techniques from recent research. Given a collection of documents, users can use pre-built pipelines in our framework to quickly assemble an RQA system using the best off-the-shelf models. Alternatively, users can create their own training data, train open-source models using algorithms from latest research, and deploy a local RQA system that achieves similar performance compared to using paid services such as OpenAI’s models.
To our knowledge, LOCALRQA is the first toolkit that provides a wide range of training algorithms and automatic evaluation metrics curated from the latest research (see Table 1 and Appendix A). This not only helps researchers to develop new RQA approaches and compare with prior work, but also helps developers to train and deploy more cost-effective models. Specifically, we provide many training algorithms for retrievers such as: distilling from an encoder-decoder’s crossattention scores (Izacard and Grave, 2020a), distilling from a decoder’s language modeling probability (Shi et al., 2023), and using contrastive learning
136
approaches (Karpukhin et al., 2020; Wang et al., 2022, 2023). We also provide training algorithms for generative models such as: supervised finetuning using gold question-passage-answer pairs (Lewis et al., 2021), fine-tuning with a frozen retriever (Guu et al., 2020), and fusion-in-decoder training (Izacard and Grave, 2020b; Izacard et al., 2022). Then, to automatically evaluate the system’s performance, we implement metrics used in retrieval and question-answering domains, such as Recall@k, ROUGE (Lin, 2004), and GPT-4 Eval (Zheng et al., 2023a; Liu et al., 2023).
Furthermore, LOCALRQA provides two deployment methods to support researchers and developers to obtain human feedback for their RQA systems. First, we offer a static evaluation webpage where users can directly assess the system’s performance using a test dataset. This can be used to complement automatic evaluation. Next, we offer an interactive chat webpage where users can chat with the system and rate the helpfulness and correctness of each generated response. These ratings can be used to further improve models’ capability using techniques such as Reinforcement Learning from Human Feedback (RLHF, Ouyang et al. (2022)). To reduce latency and improve user experiences, our toolkit also integrates acceleration frameworks used to speed up document retrieval (Johnson et al., 2019) and LLM inference (Huggingface, 2023; Kwon et al., 2023; Zheng et al., 2023b). Together with our large collection of training algorithms and automatic metrics, LOCALRQA opens the possibility of future work to easily train, test, and deploy novel RQA approaches.

heading: 2 Background
text: RQA systems combine retrievers with powerful LLMs to provide answers that are more accurate and informative. Given a user query, a retriever first
selects k most relevant passages from a collection of documents. Then, a generative model produces an answer conditioned on the user’s query, selected passages, and a chat history. Popular methods to achieve this include concatenating all inputs into a single string and generating with decoder-only models (Chase, 2022; Ram et al., 2023), or processing the k passages in parallel and generating with fusion-in-decoder techniques (Izacard and Grave, 2020b; Izacard et al., 2022).

heading: 3 LOCALRQA
text: We introduce LOCALRQA, a Python-based toolkit designed to help users flexibly train, test, and deploy RQA systems. As shown in Figure 2, our toolkits employ a modular design to allow users to: generate and prepare RQA data (Section 3.1), train retrieval and generative models (Section 3.2 and Section 3.3), build an RQA system (Section 3.4), evaluate the system (Section 3.5), and finally deploy the system (Section 3.6).

heading: 3.1 Prepare Data
text: A prerequisite for training and evaluating RQA systems is a dataset of (question, answer, passage) pairs, denoted as ⟨q, a, p⟩. However, full ⟨q, a, p⟩ pairs may not always be available in practice. To cater to various scenarios, our toolkit provides: 1) scripts to generate ⟨q, a, p⟩ pairs from a collection of documents, and 2) scripts to convert existing QA datasets into ⟨q, a, p⟩ pairs. These scripts can be useful for researchers to create RQA datasets for new domains, or for developers to prepare training/testing data for specific applications.
Generate RQA Data Given a collection of documents, our scripts first use a sampling algorithm to select a set of gold (and hard negative) documents, and then use LLMs to generate questions and answers from each gold document (see Appendix C
for more details). These scripts can be used to create ⟨q, a, p⟩ pairs not only from a collection of documents, but also from a collection of ⟨q, p⟩ pairs (e.g., from information retrieval datasets).
Convert from Existing Datasets Many existing QA datasets include supporting passages for each gold question-answer pair. We provide scripts to download and reformat these datasets into ⟨q, a, p⟩ pairs compatible with the rest of our toolkit. This includes popular datasets such as Natural Questions (Kwiatkowski et al., 2019), TriviaQA (Joshi et al., 2017), and MS-Marco (Bajaj et al., 2018). These scripts allow researchers to easily compare against prior work that also uses these datasets.

heading: 3.2 Train Retrievers
text: Given a dataset of ⟨q, a, p⟩ pairs, users can train a retriever to select the most relevant passages for a given query. Prior work shows that using better retrievers often leads to more performant RQA systems (Karpukhin et al., 2020), and that fine-tuning them with task-specific data can greatly improve their performance (Izacard et al., 2021; Wang et al., 2022). To this end, LOCALRQA implements: 1) lexical-based and embedding-based methods, and 2) various trainers that finetune open-source embedding models to achieve a better performance.
Supported Models For lexical-based methods, we support BM25 (Trotman et al., 2014). For embedding-based methods, we support all huggingface (Wolf et al., 2020) encoder models such as Contriever (Izacard et al., 2021), E5 (Wang et al., 2022), and BGE (Xiao et al., 2023).
Trainers We implement trainers for encoders that distill from a down-stream LM, and trainers
that perform contrastive learning using a dataset of ⟨q, p⟩ pairs (and optionally hard negative examples). This includes trainers that: (1) distill from cross-attention scores of an encoder-decoder model (Izacard and Grave, 2020a); (2) distill from a decoder model’s LM probability (Shi et al., 2023); and (3) train using contrastive learning (Izacard et al., 2021; Wang et al., 2022, 2023)). We provide easy-to-use Python scripts for each trainer, where all training hyperparameters can be specified in a single command line.

heading: 3.3 Train Generative Models
text: Besides improving retrievers, using better generative models can more effectively incorporate retrieved passages. To this end, our toolkit provides: 1) direct support for many open-source generative models, and 2) various training algorithms to finetune these models to improve their task-specific performance.
Supported Models We support all huggingface (Wolf et al., 2020) decoder-only models such as LLaMA-2 (Touvron et al., 2023), and all T5 based encoder-decoder models such as FLAN-T5 (Chung et al., 2022). The former is compatible with our supervised trainers, and the latter is compatible with our fusion-in-decoder trainers.
Trainers We implement supervised fine-tuning trainers that concatenate input queries with groundtruth or retrieved passages, and fusion-in-decoder trainers that process retrieved passages in parallel. This includes trainers that: (1) supervised finetune a decoder using ground-truth ⟨q, a, p⟩ pairs (Lewis et al., 2021); (2) supervised finetune a decoder with a frozen retriever (Guu et al., 2020); and (3) train
an encoder-decoder with fusion-in-decoder training (Izacard and Grave, 2020b; Izacard et al., 2022). We provide easy-to-use Python scripts for each trainer, where all training hyperparameters can be specified in a single command line.

heading: 3.4 Assemble an RQA System
text: Given a retriever and a generative model, users can now assemble an end-to-end RQA system. Similar to frameworks such as LlamaIndex, LOCALRQA uses a modular design to support arbitrary combinations of retrievers, generative models, as well as user-defined modules (see ?? for more details), such as safety filters and decision planners (Kim et al., 2023; Peng et al., 2023). For a quick start, users can use ready-made RQA pipelines to assemble a system within five lines of code (Listing 1). These built-in pipelines support: retrievers available on huggingface, retrievers trained from Section 3.2, BM25 (Robertson and Zaragoza, 2009), and OpenAI embedding models (OpenAI, 2022a); generative models available on huggingface, models trained from Section 3.3, and OpenAI models such as ChatGPT (OpenAI, 2022b).
Alternatively, a user can also customize an RQA pipeline by implementing new/modifying existing modules. As an example, we provide an implementation of a (dummy) safety filter added to the SimpleRQA pipeline in Listing 2 in Appendix. In Listing 2, the DontKnowSafetyFilter module will ignore the answers generated by the previous components of the SimpleRQA modules, and always return “I don’t know.” as an answer.
In general, users can easily add new modules to an existing pipeline by: 1) implementing a class that inherits from Component, which requires defining a run method and run_input_keys, and 2) append the module to the components field. Alteratively, researchers can create a fully customized pipeline by inheriting from the RQAPipeline class. For more documentation and examples, please refer to our GitHub pages.

heading: 3.5 Evaluate an RQA System
text: Given an RQA system, LOCALRQA implements many automatic evaluation metrics to help users measure their system’s performance. This can be used by researchers to compare their system’s performance against prior work, or by developers to find the most cost-effective models/training methods suitable for their applications. We provide scripts to automatically evaluate the perfor-
Listing 1 Assembling an RQA system. 1 from local_rqa import ... 2 ### pre-built RQA Pipeline 3 rqa = SimpleRQA.from_scratch( 4 database_path="db_path/", 5 embedding_model_name_or_path="...", 6 qa_model_name_or_path="...", 7 ) 8 response = rqa.qa( 9 batch_questions=['What is ...?'],
10 batch_dialogue_session=[ 11 DialogueSession() 12 ], 13 )
mance of any RQA system that inherits from the RQAPipeline class. These scripts will also save the evaluation results in a JSONL file, which can be used to further obtain human evaluation using our serving methods (see Section 3.6). We describe the supported automatic metrics below.
Retrieval To test the performance of a retriever, we provide an evaluation script that measures: (1) Recall@k and nDCG@k score, and (2) runtime. Recall and nDCG scores are often used in information retrieval benchmarks such as BEIR (Thakur et al., 2021) and MTEB (Muennighoff et al., 2022). Runtime is important for real-world applications.
End-to-End To test the end-to-end performance of an RQA system, we provide an automatic evaluation script that measures: (1) retrieval performance such as Recall@k; (2) generation performance such as BLEU (Papineni et al., 2002), ROUGE (Lin, 2004) and GPT-4 Eval (Zheng et al., 2023a; Liu et al., 2023); and (3) end-to-end metrics such as runtime. BLEU and ROUGE scores are often used in open-ended generation tasks such as machine translation and summarization. GPT-4 Eval is a recent method using GPT-4 (OpenAI, 2023) to evaluate the quality of model-generated responses (Liu et al., 2023; Zheng et al., 2023a).

heading: 3.6 Deploy an RQA System
text: Finally, researchers and developers may want to showcase their RQA systems to the public, or to collect human feedback to further improve their systems using techniques such as RLHF (Ouyang et al., 2022). We provide: (1) support for efficient retrieval and LLM inference acceleration methods to reduce latency during interactive chats, and (2)

heading: 1 launch controller
text: implementations to easily launch an interactive chat webpage or a static evaluation webpage, given a user-built RQA system.
Acceleration Frameworks To speed up document retrieval, we support FAISS (Johnson et al., 2019), a library for efficient similarity search across billion-scale document datasets. To speed up LLM inference, we support Text Generation Inference (TGI, Huggingface (2023)), vLLM (Kwon et al., 2023), and SGLang (Zheng et al., 2023b). These inference acceleration frameworks support many decoder architectures such as LLaMA-2 and encoderdecoder architectures such as FLAN-T5.
Interactive UIs We provide (1) a static evaluation webpage where users directly evaluate the quality of pre-generated responses (e.g., computed from a test set); and (2) an interactive chat webpage where users can chat with a system and rate the correctness and helpfulness of each response. Both web interfaces can be easily launched with our toolkits, which not only support a variety of models (see Section 3.4) but also integrate with acceleration frameworks mentioned in the previous paragraph. See Figure A1 for an example of the human evaluation page, and Figure 3 for the interactive chat page.

heading: 4 Applications
text: To showcase our toolkit, we built two RQA systems using data scraped from Databricks and Faire’s online documentations (under consent). Databricks provides the world’s first data intelligence platform powered by generative AI, providing products that facilitate building, sharing, and maintaining data at scale. Faire is an online wholesale marketplace that connects independent retailers and brands around the world. Since the documents we obtained include many company/product-specific details, we believe this is an ideal use case for RQA systems.
First, we describe the documentation datasets we collected in Section 4.1. Then, we describe our model training, baselines, and evaluation procedures in Section 4.2, Section 4.3, and Section 4.4. Finally, we present our main results in Section 4.5.

heading: 4.1 Datasets
text: Databricks We use data provided by Databricks’ technical team, which includes documentations such as API references and technical tutorials from docs.databricks.com and kb.databricks.com. After applying our data processing scripts, we obtain a dataset of 11,136 passages with a maximum length of 400 tokens. See Appendix E for examples of preprocessed documents.
Faire We first crawled guides and FAQ documents from faire.com/support, and then processed the data to only keep raw texts (e.g., removing image hyperlinks). Similar to Databricks, we then apply the data processing scripts and obtain a dataset of 1,758 passages. See Appendix F for examples of preprocessed documents.
Since both datasets only contain document passages p, we use LOCALRQA to generate ⟨q, a, p⟩ pairs for training and testing. See Appendix H for more details.

heading: 4.2 Models and Training Algorithms
text: LOCALRQA supports a large variety of models and training algorithms. To demonstrate the flexibility of our toolkit, we experiment with all available trainers and the most capable open-source models.
Retrievers We consider the best open-source encoder models according to the MTEB benchmark (Muennighoff et al., 2022) as of Jan 24, 2024. However, as these models vary greatly in capability and size, for simplicity we use the best models of similar sizes. This includes E5-base (Wang et al., 2022), Contriever-base (Izacard et al., 2021), and BGE-base (Xiao et al., 2023). We also consider all trainers in our toolkit including: (1) distilling from cross-attention scores, denoted as DCA; (2) distilling LM probability, denoted as RPG; and (3) training with contrastive learning, denoted as CTL.
Generators We consider the best generator models according to the Chatbot Arena leaderboard (Zheng et al., 2023a) as of Jan 24, 2024. Since training LLMs is time and resource intensive, we
focus on the best open-source models up to 7B parameters. This includes encoder-decoder models such as FastChat-T5-3B (Zheng et al., 2023a; Chung et al., 2022), and decoder-only models such as: StableLM-3B (Tow et al., 2023), Vicuna-7B (Chiang et al., 2023), Starling-7B (Zhu et al., 2023), and Mistral-7B (Jiang et al., 2023). We also consider all trainers in our toolkit including: (1) supervised fine-tuning, denoted as SFT; (2) supervised fine-tuning with a frozen retriever, denoted as SwR; and (3) fusion-in-decoder training, denoted as FiD.

heading: 4.3 Baselines
text: Since LOCALRQA features developing new RQA systems locally, we compare against the most powerful models accessible remotely. This include using text-ada-002 (OpenAI, 2022a) as the retriever, and prompting GPT-3.5-turbo (ChatGPT) and GPT4-turbo as the generative models.

heading: 4.4 Metrics
text: We present a subset of automatic evaluation metrics from LOCALRQA, and also include human evaluations on the best-performing models using UIs from Section 3.6. To measure retrievers’ performance, we report Recall@1 and Recall@4 which are commonly used in information retrieval (Thakur et al., 2021; Muennighoff et al., 2022). To measure the final generation performance, we report ROUGE-L (Lin, 2004) and GPT-4 Eval (Zheng et al., 2023a; Liu et al., 2023), which are used in open-domain generation tasks (Zheng et al., 2023a). For the best models, we additionally perform human evaluation and report the accuracy of the generated answers.

heading: 4.5 Main Results
text: Table 2 presents our non-exhaustive combination of retrievers and generators trained and tested using LOCALRQA. First, we find contrastive learning (CTL) most effective for training retrievers. We believe this is because CTL was also used to pretrain all the encoders we investigated (Izacard et al., 2021; Wang et al., 2022; Xiao et al., 2023). We also find that simple supervised fine-tuning (SFT) with gold ⟨q, a, p⟩ pairs is suitable for generators, given the answers in the training data are generated only using the gold passage.
Next, we find using more powerful retriever models (BGE-base and E5-base) and generator models (Mistral-7B and Starling-7B) improves Recall@4 and GPT4-Acc score. This is understandable since these models have a better base performance. We also find that ROUGE-L does not correlate well with GPT4-Acc (or our human evaluation). This is consistent with Cohan and Goharian (2016); Nekvinda and Dušek (2021), since open-ended generations are inherently difficult to evaluate using automatic metrics.
Lastly, we use the best models from Table 2 according to GPT4-Acc scores, and further validate their performance with our human evaluation UI (see Section 3.6). In Table 3, we find the best local models reach a similar performance as the OpenAI’s baselines, despite being only 7B in size (see Appendix G for some examples). These results underscores the effectiveness of our toolkit in training and developing cost-effective RQA systems.

heading: 5 Conclusion
text: We present LOCALRQA, a Python-based toolkit designed to help users develop novel retrievalaugemented QA systems. Different from existing frameworks such as LlamaIndex and LangChain, our toolkit features a wide collection of training algorithms, evaluation metrics, and deploy-
ment methods to help users quickly develop costeffective RQA systems. Strong results using models and training algorithms from recent research pave the way for future work to explore RQA methods in both practical and academic settings.

heading: 6 Limitations and Future Work
text: Model Size We performed all of our experiments using a single A100 80G GPU, and investigated a large combination of model choices and training methods. Therefore, we considered the bestperforming models up to 7B parameters due to time and resource concerns. We believe experimenting with larger, more capable models could further improve the systems’ performance, and we leave this for future work.
More Training Algorithms Besides providing tools to help users easily build an RQA system, LOCALRQA features a collection of training algorithms and evaluation methods curated from latest research. However, this collection is nonexhaustive (Zhong et al., 2022; Asai et al., 2022; Min et al., 2023; Asai et al., 2023; Ram et al., 2023). We commit to add support for more models, training algorithms, and testing methods to reflect ongoing advancements in RQA research.
Compute Requirement LOCALRQA features methods to help users develop novel RQA systems locally. Compared with using paid services such as OpenAI’s text-ada-002 and GPT-4, this approach is less expensive but requires access to compute resources (e.g., GPUs). To make our toolkit more accessible, we not only support open-source models from huggingface of various sizes, but also support using “remote” models such as OpenAI’s ChatGPT and GPT-4.

heading: 7 Ethical Considerations
text: Our work describes a toolkit that can be used to help researchers develop new RQA systems. LOCALRQA offers a suite of tools, starting from data generation to locally training, testing, and serving an RQA system. While most toolkits are not designed for unethical usage, there is often potential for abuse in their applications. In our demo (Section 4), we apply our toolkit to train RQA systems based on documentations obtained from two companies’ website, Databricks and Faire. However, since our toolkit can be used with any kind of data, it is possible to use it for unethical tasks, such
as scamming and generating harmful responses (Gehman et al., 2020; Welbl et al., 2021). We do not condone the use of LOCALRQA for any unlawful or morally unjust purposes.

heading: 8 Acknowledgement
text: We thank Xiangrui Meng and Quinn Leng from Databricks, and Wenhao Liu from Faire for their valuable support and discussions.

heading: A Comparison against Existing Toolkits
text: Many existing toolkits, such as Haystack, LangChain, and LLamaIndex help users quickly build an RQA system (Pietsch et al., 2019; Chase, 2022; Liu, 2022). However, these frameworks provide very little support for researchers to train, test, and serve their RQA systems using recent advances in retrieval-augmented QA research. For instance, LlamaIndex only includes basic supervised finetuning methods to train “LLaMA-2 for better text-toSQL”, or finetune “GPT-3.5-turbo to distill GPT4”2. We provide three different retriever training algorithms (Section 3.2) and three different generator training algorithms (Section 3.3). We highlight our main contributions compared to other existing toolkits in Table 1.

heading: B Supported Data Formats
text: LOCALRQA support data coming from many different sources, by providing integration with frameworks such as LangChain (Chase, 2022) and LlamaIndex (Liu, 2022). This not only includes loading data of different formats (e.g., JSON, HTML, PDF files), but also data from different locations (e.g., Google Drive, S3 bucket, websites and more). For more details on data loading, please refer to our project website.

heading: C Details on Data Generation
text: LOCALRQA provides data generation scripts that can be used to create questions q from a set of documents p, and answers from a set of ⟨q, p⟩ pairs. These scripts can also be easily modified to use: 1) custom prompts to generate a question or answer, and 2) custom filtering functions to use a subset of the documents for question/answer generation.
Question Generation Given a set of documents, LOCALRQA first creates a set of gold passages by sampling. Since contrastive learning (Section 3.2) benefits from using hard negative passages (related passages but does not contain the answer), we also sample nearby passages as hard negatives. This is achieved by first organizing all passages according to their source si (e.g., URL or title):
{ps00 , ps01 , ..., ps0n , ps10 , ps11 ...} and then sample from {psj}j ̸=i as hard negatives for psi . Next, an LLM of choice (e.g., ChatGPT) is
2https://docs.llamaindex.ai/en/stable/ optimizing/fine-tuning/fine-tuning.html, visited on Feb 12, 2024.
prompted to generate k questions given a sampled gold passage. To filter duplicate questions, LOCALRQA uses ROUGE-L score (Lin, 2004) to remove questions with high word overlap with others.
Answer Generation Given a set of ⟨q, p⟩ pairs, LOCALRQA prompts an LLM of choice (e.g., GPT-4) to generate answers conditioned on the question q and the gold passage p.
See Appendix E for examples on how to customize the data generation scripts and Appendices E and F for examples commands.

heading: D More Details on Serving RQA Pipelines
text: LOCALRQA offers two serving methods: 1) an interactive chat page where users can chat with an RQA system while also providing ratings for each generated response, and 2) a static evaluation page where users directly evaluate the quality (e.g., accuracy, helpfulness, harmlesness) of the pre-generated response. The front-end UIs are created using Gradio (Abid et al., 2019), and the model back-end (for interactive chat) is modified from Zheng et al. (2023a). We provide an example of using each serving method in Figure 3 and Figure A1, respectively.

heading: E More Details on Databricks Demo
text: Collected Documents We use documents provided by Databrick’s technical team, which are already cleaned and parsed into markdown format. We present an example in Table A1.
QA Generation We generate questions and answers using the data generation scripts in LOCALRQA. We first customize the prompts and filtering functions in order to obtain high-quality questions based mostly on technical tutorials rather than version release notes3. This only requires:
1. creating a new python script with from scripts.data.doc_to_q import *
2. defining a custom prompt and filter function
3. assigning filter_fn=your_filter_fn and doc2q_prompt=YOUR_PROMPT in the imported main function
For a complete example, please refer to our scripts/data/doc_to_q_databricks.py. Finally, we run the above question generation script
3https://docs.databricks.com/en/release-notes/ runtime/index.html
Databricks Example Document Faire Example Document
content
<a id=“configure-access”></a>
## Step 3: Configure access to the ‘default.people10m‘ table Enable the user you created in [Step 1](#add-a-user) to access the ‘default.people10m‘ table you created in [Step 3](#createa-table). You can configure access using [Data Explorer](#data-explorer) or [SQL editor](#sql-editor).
### Data explorer - Click the <Data Icon> **Data** in the sidebar. - In the drop-down list at the top right ...
*Please note these settings are applicable to all products, if you’d like to change these settings on a product level, you can do so in the ‘Catalog Synchronization’ tab.
# API Token This is where you will enter the API token provided by Faire. Once entered, you will also receive a message to confirm that the connection is confirmed between Faire and Prestashop.
# Catalog Import for Wholesale This is ...
metadata "source": "https://docs.databricks.com/.../admin-set-up-user -to-query-table.html", "seq_num": 574, "description": "", "text": "..."
"source": "https://www.faire.com/support/articles/8726114634779", "seq_num": 426, "subtitle": "Prestashop Integration with Faire", "title": "Prestashop Integration with Faire"
Table A1: Example documents collected from Databrick’s documentation pages and Faire’s support pages. Omitted details are indicated as “...”.
Databricks Faire
Train 1,185 575 Validation 74 74 Test 76 76
Table A2: Number of ⟨q, a, p⟩ pairs used in training, validation, and testing. During testing, we use all available documents to measure an RQA system’s retrieval and generation performance.
followed by our answer generation script to obtain a collection of ⟨q, a, p⟩ pairs. We used ChatGPT (OpenAI, 2022b) and GPT-4-turbo (OpenAI, 2023) to generate questions and answers, respectively.
Model Training To show the flexibility of LOCALRQA training, we present at least one run of using trainer in our main experiments Table 2. All trainings are performed on a single A100 80G GPU. Please refer to our GitHub for more details on training hyperparameters and other command-line arguments used. We note that we did not rigorously hyperparameter-tune each model and trainer combination due to the large number of experiments to perform. We believe results in Table 2 may be further improved if a hyperparameter search is ran for each method.

heading: F More Details on Faire Demo
text: Collected Documents We contacted Faire’s Sales team and crawled documents from faire. com/support according to their suggestions. We only kept raw texts by removing all hyperlinks for images and other websites. We present an example in Table A1.
QA Generation Since document data from Faire include simpler guides and QAs compared to Databricks, we find using the default generation script in LOCALRQA sufficient to obtain highquality questions and answers. Therefore, we simply ran scripts/data/doc_to_q.py to generate questions, and scripts/data/docq_to_a.py to generate answers. Similar to Databricks, we used ChatGPT (OpenAI, 2022b) and GPT-4-turbo (OpenAI, 2023) to generate questions and answers, respectively.
Model Training Similar to the training process implemented in Databricks, we conduct experiments across various trainers and model choices. See Table 2 for an example. We note that we did not rigorously hyperparameter-tune each model and trainer combination due to the large number of experiments to perform. We believe results in Table 2 may be further improved if a hyperparameter search is ran for each method.

heading: G Example Model Answers
text: We provide example generated answers and retrieved documents in Table A3 and Table A4 for Databricks and Faire, respectively. In Table A3, the weaker retrievers fail to find the relevant document given the user’s query. As a result, the generative models also fail to provide the correct answer. This shows the benefit of training and using more accurate retrievers. In Table A4, despite all methods retrieved the correct document, the weaker generative models failed to incorporate all relevant information in its response. This underscores the importance to fine-tune capable generative models to improve the RQA system’s performance.

heading: H Dataset Statistics
text: We follow the procedure outlined in Appendix E and Appendix F to obtain ⟨q, a, p⟩ pairs from documentation data. For Databricks, we first use our sampling algorithm to obtain 600 passages for training, and 75 for validation and for testing. For Faire, we sampled 300 passages for training as Faire has fewer documents, and 75 for validation and for testing. We then use ChatGPT (OpenAI, 2022b) as the LLM to generate two questions for each sampled passage, and GPT-4-turbo (OpenAI, 2023) to generate an answer for each ⟨q, p⟩ pair (see Appendix C for more details). This results in a training, evaluation, and testing dataset with statistics reported in Table A2.
Databricks RQA Example
User: How many catalogs can a metastore have in Databricks?
text-ada-002+ GPT-3.5-turbo
In Databricks, a metastore can have multiple catalogs. There is no specific limit on the number of catalogs that can be associated with a metastore. Catalogs are used to organize [...] Users can see all catalogs on which they have been assigned the ‘USE CATALOG‘ data permission.
text-ada-002+ GPT-4-turbo
A metastore in Databricks can contain multiple catalogs. [...] There isn’t a specified limit to the number of catalogs a metastore can have mentioned in the documentation, which suggests that users can create multiple catalogs as needed to manage their data in an organized manner [...].
E5(CTL)+ Mistral-7B(SFT)
As of August 25, 2022 , a metastore can have up to 1000 catalogs .
BGE(CTL)+ Starling-7B(SFT)
As of August 25, 2022 , a metastore in Databricks can have up to 1000 catalogs .
reference document: # <UC> GA release note **August 25, 2022** <UC> is now generally available on <Databricks>. **This article describes <UC> as of the date of its GA release.** It focuses primarily on the features and updates [...] As of August 25, 2022
- Your <Databricks> account can have only one metastore per region - A metastore can have up to 1000 catalogs. - A catalog can have up to 10,000 schemas. - A schema can have up to 10,000 tables. For current <UC> quotas, see [...].
Table A3: Example answers generated by different RQA systems. Questions come from the test dataset for Databricks, which is described in Appendix H. Due to space limitations, we display the most relevant document amongst all documents retrieved. In this example, text-ada-002 failed to retrieve the most relevant documents, causing GPT-3.5-turbo and GPT-4-turbo to produce incorrect answers.
Faire RQA Example
User: How should I crop my feature image for it to be approved?
text-ada-002+ GPT-3.5-turbo
To ensure that your feature image is approved, you should crop it to a square shape. The image must fill a 1:1 ratio , meaning that the width and height should be equal. By cropping your image to a square, you can meet this requirement and increase the chances of approval.
text-ada-002+ GPT-4-turbo
To ensure your feature image gets approved, you should follow these cropping and content guidelines: 1. **Square Crop Ratio**: Crop your image to a perfect square. The aspect ratio should be 1:1 , which means [...].
2. **No Overlaid Text or Logos** : Avoid adding any text or logos [...]. 3. **Simplicity is Key**: Steer clear of using collages or images that are too busy. A simple, singular focus on the product [...].
4. **Visual Context**: Provide context by showing the product in use or in a setting where it is naturally found [...]. 5. **Avoid Direct Eye Contact**: When including models, aim to avoid direct eye contact with the camera. [...]
E5(CTL)+ Mistral-7B(SFT)
To have your feature image approved, you should ensure that it is cropped to fit a square aspect ratio, which must fill a 1:1 ratio . Additionally, it should not include any words or logos . The recommended approach is to choose a highresolution lifestyle image that showcases your best-selling product(s) in use, such as a cheese knife with cheese and grapes on a cutting board. Also, you should provide some visual context in [...]
BGE(CTL)+ Starling-7B(SFT)
To have your feature image approved, you should crop it to a square shape that fills a 1:1 ratio .
reference document: ## Feature image guidelines Crop your image to a square. It must fill a 1:1 ratio . Don’t include overlaid text or logos .
Avoid collages and keep it simple. Collages cause clutter in a space already crowded with many other brands. Choose a single image with a clear product focus. Simple, uncluttered often stand out more, as they provide rest to the shopper’s eye that may be weary from browsing many pages of content. Provide visual context (avoid plain white backgrounds). While it’s true that great packaging design attracts people all by itself, showing your product in use helps get people excited about your brand and helps tell a more visual story about what you have to offer. Avoid showing models with direct eye contact. [...]
Table A4: Example answers generated by different RQA systems. Questions come from the test dataset for Faire, which is described in Appendix H. Due to space limitations, we display the most relevant document amongst all documents retrieved. In this example, all methods retrieved the correct document, but GPT-3.5-turbo and Starling-7B failed to include details other than “fill a 1:1 ratio"" (c.f. Table 3).
> python local_rqa/serve/gradio_static_server.py \ file_path xxx/test-predictions.jsonl \ include_idx 1-50
web UI31 test prediction file
2 launch server retrieved documents a
chat historyb
evaluationc
evaluation results4
analysis
"question": "What ...", "gen_answer": "....", "retrieved_docs": [....], ... "question": "How ..?", "gen_answer": "....", "retrieved_docs": [....], ...
"session": {"history": ....}, "annotation": "{...}", "metadata": {...} "session": {"history": ....}, "annotation": "{...}", "metadata": {...} "session": {"history": ....}, "annotation": "{...}", "metadata": {...}
Figure A1: Researchers can launch a human evaluation page using LOCALRQA in a single command line. Given a prediction file (see Section 3.5), LOCALRQA launches a web server that allows other users to evaluate the quality of pre-generated responses. Evaluation results are automatically saved for researchers to conduct further analysis.
Listing 2 Adding a custom component to the SimpleRQA pipeline. In general, researchers can easily extend an existing pipeline by editing the .component field, or create new pipelines by inherting from the RQAPipeline class.
1 from local_rqa import ... 2 ### pre-built RQA pipeline 3 rqa = SimpleRQA.from_scratch( 4 database_path="db_path/", 5 embedding_model_name_or_path="...", 6 qa_model_name_or_path="...", 7 ) 8
9 ### custom module: safety filter 10 class DontKnowSafetyFilter(Component): 11 run_input_keys = [ 12 "batch_questions", 13 "batch_source_documents", 14 "batch_dialogue_session", 15 "batch_answers", 16 ] 17
18 def run(self, *args, **kwargs): 19 return RQAOutput( 20 batch_answers="I don't know.", 21 batch_source_documents=kwargs["batch_source_documents"], 22 batch_dialogue_session=kwargs["batch_dialogue_session"], 23 ) 24
25 rqa.components.append(DontKnowSafetyFilter()) 26
27 ### run QA! 28 rqa.qa(...) # output: "I don't know."

heading: Future Work
text: We believe experimenting with larger, more capable models could further improve the systems’ performance, and we leave this for future work. More Training Algorithms Besides providing tools to help users easily build an RQA system, LOCALRQA features a collection of training algorithms and evaluation methods curated from latest research. However, this collection is nonexhaustive (Zhong et al. , 2022; Asai et al. , 2022; Min et al. , 2023; Asai et al. , 2023; Ram et al. , 2023). We commit to add support for more models, training algorithms, and testing methods to reflect ongoing advancements in RQA research. Compute Requirement LOCALRQA features methods to help users develop novel RQA systems locally. Compared with using paid services such as OpenAI’s text-ada-002 and GPT-4, this approach is less expensive but requires access to compute resources (e. g. , GPUs). To make our toolkit more accessible, we not only support open-source models from huggingface of various sizes, but also support using “remote” models such as OpenAI’s ChatGPT and GPT-4