Package: semantic-code-search
Semantic Code SearchSearch your codebase with natural language. No data leaves your computer.🔍 Overview•🔧 Installation•💻 Usage•📖 Docs•🧠 How it worksOverviewsemis a command line application which allows you to search your git repository using natural language. For example you can query for:'Where are API requests authenticated?''Saving user objects to the database''Handling of webhook events''Where are jobs read from the queue?'You will get a (visualized) list of code snippets and theirfile:linelocations. You can usesemfor exploring large codebases or, if you are as forgetfull as I am, even small ones.Basic usage:sem'my query'This will present you with a list of code snippets that most closely match your search. You can select one and pressReturnto open it in your editor of choice.How does this work? In a nutshell, it uses a neural network to generate code embeddings. More infobelow.NB: All processing is done on your hardware and no data is transmitted to the Internet.InstallationYou can installsemantic-code-searchviapip.Pip (MacOS, Linux, Windows)pip3installsemantic-code-searchUsageTL;DR:cd/my/repo
sem'my query'Runsem --helpto seeall available options.Searching for codeInside your repo simply runsem'my query'(quotes can be omitted)Note that youneed tobe  inside a git repository or provide a path to a repo with the-pargument.Before you get yourfirstsearch results, two things need to happen:The app downloads itsmodel(~500 MB). This is done only once for the installation.The app generates 'embeddings' of your code. This will be cached in an.embeddingsfile at the root of the repo and is reused in subsequent searches.Depending on the project size, the above can take from a couple of seconds to minutes. Once this is complete, querying is very fast.Example output:foo@bar:~$cd/my/repo
foo@bar:~$sem'parsing command line args'Embeddingsnotfoundin/Users/kiril/src/semantic-code-search.Generatingembeddingsnow.
Embedding15functionsin1batches.Thisisdoneonceandcachedin.embeddings
Batches:100%|█████████████████████████████████████████████████████████|1/1[00:07<00:00,7.05s/it]Navigating search resultsBy default, a list of the top 5 matches are shown, containing :Similarity scoreFile pathLine numberCode snippetYou can navigate the list using the↑↓arrow keys orvimbindings. Pressingreturnwill open the relevant file at the line of the code snippet in your editor.NB: The editor used for opening can be set with the--editorargument.Example results:Command line flagsusage:sem[-h][-pPATH][-mMODEL][-d][-bBS][-xEXT][-nN][-e{vscode,vim}][-c][--cluster-max-distanceTHRESHOLD][--cluster-min-linesSIZE][--cluster-min-cluster-sizeSIZE][--cluster-ignore-identincal]...

Searchyourcodebaseusingnaturallanguage

positionalarguments:query_text

optionalarguments:-h,--helpshowthishelpmessageandexit-pPATH,--path-to-repoPATHPathtotherootofthegitrepotosearchorembed-mMODEL,--model-name-or-pathMODELNameorpathofthemodeltouse-d,--embed(Re)createtheembeddingsindexforcodebase-bBS,--batch-sizeBSBatchsizeforembeddingsgeneration-xEXT,--file-extensionEXTFileextensionfilter(e.g."py"willonlyreturnresultsfromPythonfiles)-nN,--n-resultsNNumberofresultstoreturn-e{vscode,vim},--editor{vscode,vim}Editortoopenselectedresultin-c,--clusterGenerateclustersofcodethatissemanticallysimilar.Youcanusethistospotnearduplicates,resultsaresimplyprintedtostdout--cluster-max-distanceTHRESHOLDHowclosefunctionsneedtobetooneanothertobeclustered.Distance0meansthatthecodeisidentical,smallervalues(e.g.0.2,0.3)arestricterandresultinfewermatches--cluster-min-linesSIZEIgnoreclusterswithcodesnippetssmallerthanthissize(linesofcode).Usethisifyouarenotinterestedinsmallerduplications(eg.oneliners)--cluster-min-cluster-sizeSIZEIgnoreclusterssmallerthanthissize.Usethisifyouwanttofindcodethatissimilarandrepeatedmanytimes(e.g.>5)--cluster-ignore-identincalIgnoreidenticalcode/exactduplicates(wheredistanceis0)How it worksIn a nutshell, this application uses atransformermachine learning model to generate embeddings of methods and functions in your codebase. Embeddings are information dense numerical representations of the semantics of the text/code they represent.Here is a great blog post by Jay Alammar which explains the concept really nicely:https://jalammar.github.io/illustrated-word2vec/When the app is ran with the--embedargument, function and method definitions are first extracted from the source files and then used for sentence embedding. To avoid doing this for every query, the results are compressed and saved in an.embeddingsfile.When a query is being processed, embeddings are generated from the query text. This is then used in a 'nearest neighbor' search to discover function or methods with similar embeddings. We are basically comparing thecosine similaritybetween vectors.ModelThe application usessentence transformermodel architecture to produce 'sentence' embeddings for functions and queries. The particular model iskrlvi/sentence-t5-base-nlpl-code_search_netwhich is based of aSentenceT5-Basecheckpoint with 110M parameters and a pooling layer.It has been further trained on thecode_search_netdataset of 'natural language' — 'programming language' pairs with aMultipleNegativesRankingloss function.You can experiment with your own sentence transformer models with the--modelparameter.Bugs and limitationsCurrently, the.embeddingsindex is not updated when repository files change. As a temporary workaround,sem embedcan be re-ran occasionally.Supported languages:{ 'python', 'javascript', 'typescript', 'ruby', 'go', 'rust', 'java', 'c', 'c++', 'kotlin' }Supported text editors for opening results in:{ 'vscode', 'vim' }LicenseSemantic Code Search is distributed underAGPL-3.0-only. For Apache-2.0 exceptions —kiril@codeball.ai