Package: primeqa
The Prime Repository for State-of-the-Art Multilingual Question Answering Research and Development.PrimeQA is a public open source repository that enables researchers and developers to train state-of-the-art models for question answering (QA). By using PrimeQA, a researcher can replicate the experiments outlined in a paper published in the latest NLP conference while also enjoying the capability to download pre-trained models (from an online repository) and run them on their own custom data. PrimeQA is built on top of theTransformerstoolkit and usesdatasetsandmodelsthat are directly downloadable.The models within PrimeQA supports End-to-end Question Answering. PrimeQA answers questions viaInformation Retrieval: Retrieving documents and passages using both traditional (e.g. BM25) and neural (e.g. ColBERT) modelsMultilingual Machine Reading Comprehension: Extract and/ or generate answers given the source document or passage.Multilingual Question Generation: Supports generation of questions for effective domain adaptation overtablesandmultilingual text.Retrieval Augmented Generation: Generate answers using the GPT-3/ChatGPT pretrained models, conditioned on retrieved passages.Some examples of models (applicable on benchmark datasets) supported are :Traditional IR with BM25PyseriniNeural IR with ColBERT, DPR(collaboration withStanford NLPIR led byChris Potts&Matei Zaharia).
Replicating the experiments thatDr. Decr(Li et. al, 2022) performed to reach the top of the XOR TyDI leaderboard.Machine Reading Comprehension with XLM-R: to replicate the experiments to get to the top of the TyDI leaderboard similar to the performance of the IBM GAAMA system. Coming soon: code to replicate GAAMA's performance on Natural Questions.🏅 Top of the LeaderboardPrimeQA is at the top of several leaderboards: XOR-TyDi, TyDiQA-main, OTT-QA and HybridQA.XOR-TyDiTyDiQA-mainOTT-QAHybridQA✔️ Getting StartedInstallationInstallation doc# cd to project root# If you want to run on GPU make sure to install torch appropriately# E.g. for torch 1.11 + CUDA 11.3:pipinstall'torch~=1.11.0'--extra-index-urlhttps://download.pytorch.org/whl/cu113# Install as editable (-e) or non-editable using pip, with extras (e.g. tests) as desired# Example installation commands:# Minimal install (non-editable)pipinstall.# GPU supportpipinstall.[gpu]# Full install (editable)pipinstall-e.[all]Please note that dependencies (specified insetup.py) are pinned to provide a stable experience.
When installing from source these can be modified, however this is not officially supported.Note:in many environments, conda-forge based faiss libraries perform substantially better than the default ones installed with pip. To install faiss libraries from conda-forge, use the following steps:Create and activate a conda environmentInstall faiss libraries, using a commandconda install -c conda-forge faiss=1.7.0 faiss-gpu=1.7.0Insetup.py, remove the faiss-related lines:"faiss-cpu~=1.7.2": ["install", "gpu"],
"faiss-gpu~=1.7.2": ["gpu"],Continue with thepip installcommands as desctibed above.JAVA requirementsJava 11 is required for BM25 retrieval. Install java as follows:condainstall-cconda-forgeopenjdk=11:speech_balloon: Blog PostsThere're several blog posts by members of the open source community on how they've been using PrimeQA for their needs. Read some of them:PrimeQA and GPT 3Enterprise search with PrimeQAA search engine for Trivia geeks🧪 Unit TestsTesting docTo run the unit tests you first need toinstall PrimeQA.
Make sure to install with the[tests]or[all]extras from pip.From there you can run the tests via pytest, for example:pytest--covPrimeQA--cov-config.coveragerctests/For more information, see:Ourtox.iniThepytestandtoxdocumentation🔭 Learn moreSectionDescription📒DocumentationFull API documentation and tutorials🏁Quick tour: Entry Points for PrimeQADifferent entry points for PrimeQA: Information Retrieval, Reading Comprehension, TableQA and Question Generation📓Tutorials: Jupyter NotebooksNotebooks to get started on QA tasks📓GPT-3/ChatGPT Reader NotebooksNotebooks to get started with the GPT-3/ChatGPT reader components💻Examples: Applying PrimeQA on various QA tasksExample scripts for fine-tuning PrimeQA models on a range of QA tasks🤗Model sharing and uploadingUpload and share your fine-tuned models with the community✅Pull RequestPrimeQA Pull Request📄Generate DocumentationHow Documentation works🛠Orchestrator Service REST MicroserviceProof-of-concept code for PrimeQA Orchestrator microservice📖Tooling UIDemo UI❤️ PrimeQA collaborators includeStanford NLPUniversity of IllinoisUniversity of StuttgartUniversity of Notre DameOhio State UniversityCarnegie Mellon UniversityUniversity of MassachusettsIBM Research