Package: codequestion
Semantic search for developerscodequestion is a semantic search application for developer questions.Developers typically have a web browser window open while they work and run web searches as questions arise. With codequestion, this can be done from a local context. This application executes similarity queries to find similar questions to the input query.The default model for codequestion is built off theStack Exchange Dumps on archive.org. Once a model is installed, codequestion runs locally, no network connection is required.codequestion is built with Python 3.7+ andtxtai.InstallationThe easiest way to install is via pip and PyPIpip install codequestionPython 3.7+ is supported. Using a Pythonvirtual environmentis recommended.codequestion can also be installed directly from GitHub to access the latest, unreleased features.pip install git+https://github.com/neuml/codequestionSeethis linkfor environment-specific troubleshooting.Download a modelOnce codequestion is installed, a model needs to be downloaded.python -m codequestion.downloadThe model will be stored in ~/.codequestion/The model can also be manually installed if the machine doesn't have direct internet access. The default model is pulled from theGitHub release pageunzip cqmodel.zip ~/.codequestionSearchStart up a codequestion shell to get started.codequestionA prompt will appear. Queries can be typed into the console. Typehelpto see all available commands.TopicsThe latest release integratestxtai 5.0, which has support for semantic graphs.Semantic graphs add support for topic modeling and path traversal. Topics organize questions into groups with similar concepts. Path traversal uses the semantic graph to show how two potentially disparate entries are connected. An example covering both topic and path traversal is shown below.VS CodeA codequestion prompt can be started within Visual Studio Code. This enables asking coding questions right from your IDE.RunCtrl+`to open a new terminal then typecodequestion.API servicecodequestion builds a standard txtai embeddings index. As such, it supports hosting the index via atxtai API service.Running the following:app.ymlpath:/home/user/.codequestion/models/stackexchange/embeddings:# Install API extra
pip install txtai[api]

# Start API
CONFIG=app.yml uvicorn "txtai.api:app"

# Test API
curl "http://127.0.0.1:8000/search?query=python+query+sqlite&limit=1"Outputs:[{"id":"616429","text":"How to fetch data from sqlite using python? stackoverflow python sqlite","score":0.8401689529418945}]Additional metadata fields can be pulled back with SQL statements.curl
    --get
    --data-urlencode "query=select id, date, tags, question, score from txtai where similar('python query sqlite')"
    --data-urlencode "limit=1"
    "http://127.0.0.1:8000/search"[{"id":"616429","date":"2022-05-23T10:45:40.397","tags":"python sqlite","question":"How to fetch data from sqlite using python?","score":0.8401689529418945}]Tech overviewThe following is an overview covering how this project works.Process the raw data dumpsThe raw 7z XML dumps from Stack Exchange are processed through a series of steps (seebuilding a model). Only highly scored questions with accepted answers are retrieved for storage in the model. Questions and answers are consolidated into a single SQLite file called questions.db. The schema for questions.db is below.questions.db schemaId INTEGER PRIMARY KEY
Source TEXT
SourceId INTEGER
Date DATETIME
Tags TEXT
Question TEXT
QuestionUser TEXT
Answer TEXT
AnswerUser TEXT
Reference TEXTIndexcodequestion builds a txtai embeddings index for questions.db. Each question in the questions.db schema is vectorized with a sentence-transformers model. Once questions.db is converted to a collection of sentence embeddings, the embeddings are normalized and stored in Faiss, which enables fast similarity searches.Querycodequestion tokenizes each query using the same method as during indexing. Those tokens are used to build a sentence embedding. That embedding is queried against the Faiss index to find the most similar questions.Build a modelThe following steps show how to build a codequestion model using Stack Exchange archives.This is not necessary if using the default model from theGitHub release page1.) Download files from Stack Exchange:https://archive.org/details/stackexchange2.) Place selected files into a directory structure like shown below (current process requires all these files).stackexchange/ai/ai.stackexchange.com.7zstackexchange/android/android.stackexchange.com.7zstackexchange/apple/apple.stackexchange.com.7zstackexchange/arduino/arduino.stackexchange.com.7zstackexchange/askubuntu/askubuntu.com.7zstackexchange/avp/avp.stackexchange.com.7zstackexchange/codereview/codereview.stackexchange.com.7zstackexchange/cs/cs.stackexchange.com.7zstackexchange/datascience/datascience.stackexchange.com.7zstackexchange/dba/dba.stackexchange.com.7zstackexchange/devops/devops.stackexchange.com.7zstackexchange/dsp/dsp.stackexchange.com.7zstackexchange/raspberrypi/raspberrypi.stackexchange.com.7zstackexchange/reverseengineering/reverseengineering.stackexchange.com.7zstackexchange/scicomp/scicomp.stackexchange.com.7zstackexchange/security/security.stackexchange.com.7zstackexchange/serverfault/serverfault.com.7zstackexchange/stackoverflow/stackoverflow.com-Posts.7zstackexchange/stats/stats.stackexchange.com.7zstackexchange/superuser/superuser.com.7zstackexchange/unix/unix.stackexchange.com.7zstackexchange/vi/vi.stackexchange.com.7zstackexchange/wordpress/wordpress.stackexchange.com.7z3.) Run the ETL processpython -m codequestion.etl.stackexchange.execute stackexchangeThis will create the file stackexchange/questions.db4.)OPTIONAL:Build word vectors - only necessary if using a word vectors model. If using word vector models, make sure to runpip install txtai[similarity]python -m codequestion.vectors stackexchange/questions.dbThis will create the file ~/.codequestion/vectors/stackexchange-300d.magnitude5.) Build embeddings indexpython -m codequestion.index index.yml stackexchange/questions.dbThedefault index.ymlfile is found on GitHub. Settings can be changed to customize how the index is built.After this step, the index is created and all necessary files are ready to query.Model accuracyThe following sections show test results for codequestion v2 and codequestion v1 using the latest Stack Exchange dumps. Version 2 uses a sentence-transformers model. Version 1 uses a word vectors model with BM25 weighting. BM25 and TF-IDF are shown to establish a baseline score.StackExchange QueryModels are scored usingMean Reciprocal Rank (MRR).ModelMRRall-MiniLM-L6-v285.0SE 300d - BM2577.1BM2567.7TF-IDF61.7STS BenchmarkModels are scored usingPearson Correlation. Note that the word vectors model is only trained on Stack Exchange data, so it isn't expected to generalize as well against the STS dataset.ModelSupervisionDevTestall-MiniLM-L6-v2Train87.082.7SE 300d - BM25Train74.067.4TestsTo reproduce the tests above, run the following. Substitute $TEST_PATH with any local path.mkdir -p $TEST_PATH
wget https://raw.githubusercontent.com/neuml/codequestion/master/test/stackexchange/query.txt -P $TEST_PATH/stackexchange
wget http://ixa2.si.ehu.es/stswiki/images/4/48/Stsbenchmark.tar.gz
tar -C $TEST_PATH -xvzf Stsbenchmark.tar.gz
python -m codequestion.evaluate -s test -p $TEST_PATHFurther readingFind answers with codequestion 2.0Building a sentence embedding index with fastText and BM25 (codequestion 1.0)