Unipa-GPT: a framework to assess open-source alternatives to Chat-GPT for Italian chat-bots
Authors: Irene Siragusa, Roberto Pirrone (Università di Palermo, IT University of Copenhagen, 2024)
Topic: Language Models

Abstract
This paper illustrates the implementation of Open Unipa-GPT, an open-source version of the Unipa-GPT chat-bot that leverages open-source Large Language Models for embeddings and text generation. The system relies on a Retrieval Augmented Generation approach, thus mitigating hallucination errors in the generation phase. A detailed comparison between different models is reported to illustrate their performance as regards embedding generation, retrieval, and text generation. In the last case, models were tested in a simple inference setup after a fine-tuning procedure. Experiments demonstrate that an open-source LLMs can be efficiently used for embedding generation, but none of the models does reach the performances obtained by closed models, such as \verb|gpt-3.5-turbo| in generating answers. Corpora and code are available on GitHub

Introduction
The increasing development of bigger and bigger Large Language Models (LLM), reaching 70B parameters as for Meta LLMs (Llama 2 AUTHOR and \verb|Llama 3| AUTHOR) and more as for OpenAI ones (GPT-3 AUTHOR and GPT-4 AUTHOR and gpt-3.4 respectively}), requires a significant computational resources for training, fine-tuning or inference. OpenAI models are accessible only upon payment via OpenAI API and cannot be downloaded in any way, while the open-source models by Meta are available also in the 8B and 13B parameters versions, and they can either be fine-tuned via Parameter-Efficient Fine-Tuning techniques (PEFT) AUTHOR such as LoRA AUTHOR, or they can make direct inference using a 8-bit quantization AUTHOR keeping the computational resources relatively small. The availability of open-source small-size LLMs is crucial for developing Natural Language Process (NLP) applications that leverage a fine-tuning procedure over a specific domain or language, as for \verb|Anita| AUTHOR, an Italian 8B adaptation of \verb|Llama 3|. Nevertheless, GPT and Llama models cannot be considered as truly open-source since their training data set is not available and, as for GPT models, and also their actual architecture is not accessible. \verb|Minerva| AUTHOR model, on the other side, is an Italian and English LLM whose architecture, weights, and training data are accessible, but it can be considered as an exception in the LLM landscape. Starting from this premises, in this paper we propose Open Unipa-GPT, an open-source-based version of Unipa-GPT AUTHOR, that is a virtual assistant that uses a Retrieval Augmented Generation (RAG) approach AUTHOR to answer university-related questions issued by secondary school students. Open Unipa-GPT has been developed upon the same architecture of Unipa-GPT, and uses open-source LLMs for embedding generation, retrieval, and text generation. Our models are small, compared to the ones used in our original version, namely \verb|text-embedding-ada-002| and \verb|gpt-3.5-turbo| from OpenAI. The paper is arranged as follows: related works are reported in Section sota, while the architecture of Open Unipa-GPT is described in Section architecture, and an overview of the data set is provided in Section data set. Experiments and related results are reported in Section experiments. Finally, concluding remarks are drawn in Section conclusion.

Conclusion
In this paper we presented Open Unipa-GPT, a virtual assistant, which is based solely on open-source LLMs, and uses a RAG approach to answer Italian university-related questions from secondary school students. The main intent of the presented research was setting up a sort of framework to test open-source small size LLMs, with either moderate or no fine-tuning at all, to be used for generating the embeddings and/or as text generation front-end in a RAG set up. Our study led us to devise \verb|E5-mistral-7b-instruct| as a valuable open-source alternative to OpenAI's embeddings, while none of the considered models attain a generation performance comparable to \verb|gpt-3.5-turbo|, even after a fine-tuning procedure. The most promising Generation LLM, when plunged in our architecture, appears to be \verb|Anita-8B|, but it still shows some issues related to both the tokenization and the grammatical correctness of the output. We are currently working to deep exploration of different fine-tuning approaches along with the use of huge size open-source LLMs for text generation. \appendix details