Saravanan666/pypi-org huggingface dataset:
Package: embayes
# embayes
Bayesian machine learning classifiers for embedded systems.
Train in Python, deploy on microcontroller.## Key featuresEmbedded-friendly ClassifierPortable C99 codeNo stdlib requiredNo dynamic allocationsInteger/fixed-point math onlySingle header file includeFast, sub-millisecond classificationConvenient TrainingAPI-compatible with [scikit-learn](http://scikit-learn.org)Implemented in Python 3C classifier accessible in Python using pybind11[MIT licensed](./LICENSE.md)## StatusMinimally usefulGaussian Naive Bayes classifier implementedTested running on ESP8266 and Linux.On ESP8266, 2 classes and 30 features classify in under 0.5ms## InstallingInstall from gitgit clonehttps://github.com/jonnor/embayespython3 setup.py install –user## UsageSee [examples/cancer.py](./examples/cancer.py) and [embayes.ino](./embayes.ino)## TODO0.2Make estimator a wrapper aroundsklearn.naivebayes.GaussianNBMake estimator work in sklearn pipelineMakepdfapproximation configurable as parameter1.0Support generating inline C code, not needing model coefficients in RAMSupport de/serializing coefficients at runtimeSupport training on microcontroller
Package: embedbase
EmbedbaseSeamless data integration for LLMsOpen-source API & SDK to integrate your data and easily hook them up to LLMsUsed byAVAand serving 100k request a dayTry the Hosted Version·Request Feature·Report BugCheck out thedocsfor more info.What is itEmbedbase is a dead-simple API to help you useVectorDBsandEmbeddings Modelswithout needing to host them!
You can use embedbase to customize LLM (like ChatGPT!) and automatically feed them the right information.Installationnpm i embedbase-js// this examples shows how you can use embedbase to automatically add context in a ChatGPT promptimport{createClient}from'embedbase-js'constquestion='What can I do with Embedbase API?'constembedbase=createClient('https://api.embedbase.xyz','api-key')constcontext=awaitembedbase.dataset('my-documentation').createContext(question);console.log(context)/* ["Embedbase API allows to store unstructured data...","Embedbase API has 3 main functions a) provides a plug and play solution to store embeddings b) makes it easy to connect to get the right data into llms c)..","Embedabase API is self-hostable...",] */constprompt=`Based on the following context:\n${context.join()}\nAnswer the user's question:${question}`// for await allows you to stream answersforawait(constresofembedbase.generate(prompt)){console.log(res)// You, can, use, ...}// answer:// You can use the Embedbase API to store unstructured data and then use the data to connect it to LLMsTable of ContentsGetting startedJavascript SDKDocs and supportIntegrationsContributingWhat are people buildingRecommendation Engines: AVA uses Embedbase to help their users find related notesChat with your data: Solpilot uses Embedbase to put smart contract integration on autopilotTalk to your docs: ChatGPT-powered search for markdown documentationThe fastest way to get started with Embedbase is signing up for free toEmbedbase Cloud.Supported IntegrationsConnectionsAny data with the sdk or apiPDFGithubVector DBsSupabasePostgresQdrantWeaviateRedisEmbedding ModelsOpenAI Embeddingssentence-transformersT5Docs and supportCheck out ourtutorialsfor step-by-step guides, how-to's, and best practices, our documentation is powered by GPT-4, so you can ask question directly.Ask a question in ourDiscord communityto get support.ContributingPlease readCONTRIBUTING.mdfor details on our code of conduct, and the process for submitting pull requests to us.Self-hostingNote: this render configuration works well for experimentation.
Package: embedbase-internet-search
embedbase-internet-searchembedbase-internet-search - internet search extension forEmbedbase⚠️ Status: Alpha release ⚠️The point of internet search in embedbase is to combine your private information with latest public information.Also remember that AIs like ChatGPT have limited knowledge to a certain date, for example try to ask ChatGPT about GPT4 or about Sam Altman talk with the senate (which happened few days ago), it will not know about it.Please checkexamplesfor usage.If you have any feedback or issues, please let us know by opening an issue or contacting us ondiscord.Regarding the SDK, please refer to thedocumentation.
Package: embedbase-qdrant
embedbase-qdrantEmbedbase+QdrantAdvanced and high-performant vector similarity search technology in your AI applications⚠️ Status: Alpha release ⚠️If you have any feedback or issues, please let us know by opening an issue or contacting us ondiscord.Please refer to thedocumentationGetting startedTo install the Embedbase Qdrant library, run the following command:pipinstallembedbase-qdrantQuick tourLet's try Embedbase + Qdrant with an OpenAIembedder:pipinstallopenaiuvicornimportosimportuvicornfromembedbaseimportget_appfromembedbase.embedding.openaiimportOpenaifromembedbase_qdrantimportQdrant# here we use openai to create embeddings and qdrant to store the dataapp=get_app().use_embedder(Openai(os.environ["OPENAI_API_KEY"])).use_db(Qdrant()).run()if__name__=="__main__":uvicorn.run(app)Start a local Qdrant:docker-composeup-dRun Embedbase:python3main.pyCheck out otherexamplesanddocumentationfor more details.
Package: embed-builder
embed_builderI got tired of manually writing dictionaries to send embeds via Discord webhooks, so I made this package to do it effortlessly.This package was developed on Python 3.10.x but has not been tested on earlier versions. If you happen to successfully use this package on an older version then do let me know.Installation$pipinstallembed-builderUsagefromembed_builderimportEmbedembed=Embed()embed.set_title("Hello")embed.set_description("How are you?")embed.set_color("#ff69b4")# New in 1.4.0 - Validating embed limits - https://discord.com/developers/docs/resources/channel#embed-object-embed-limitsifnotembed.is_valid():raiseException("woops! embed has exceeded Discord's limits")my_embed=embed.build()# Or via chaining...my_embed=Embed().set_title("Hello").set_description("How are you?").build()# Example usage with Discord webhooks and requests packagerequests.post("webhook url",json={"content":"here is an embed","embeds":[my_embed]})
Package: embedc
Python module for embedding C/C++ code within Python code. It is designed to allow coders to easily speed up slow code by replacing small portions of Python with compiled C/C++ that can transparently read and modify Python variables. Keeping all the code in the same source file simplifies development and improves code readability. All the compiling and linking is done dynamically by the module to further simplify development.* Embed C/C++ within Python source code* Automatically compile and dynamically link using GCC* Access and modify Python variables as C variables* Support arrays and strings* Use ctypes for maximum portabilitySimple Example:from embedc import Ca=[1,4,9]alen=len(a)C("""for (int i=0; i<alen; i++) {a[i] = a[i]*a[i];}""")print aOutput: [1,16,81]
Package: embedd
Embedd
简介
Embedd 是一个 Python 工具，用于处理和搜索文本嵌入。它支持从不同格式的文件中读取数据，生成文本嵌入，并提供了一种方法来搜索这些嵌入以找到与给定查询最相似的记录。功能
生成文本嵌入。
提供本地文档搜索功能。安装
您可以通过 pip 安装此包：
pip install embedd使用前的配置
在使用此包之前，您需要一个 OpenAI API 密钥。您可以通过以下几种方式提供 API 密钥：直接在创建 Embed 实例时传入。
通过配置文件传入。
从环境变量 OPENAI_API_KEY 中读取。基本使用
创建嵌入文件
embedder = Embed(api_key="您的API密钥")
embedder.create_embedding_file("path/to/input.csv", "path/to/output")从文件中搜索
python
Copy code
results = embedder.search_from_file("path/to/data.csv", "用户查询", top_n=3, to_print=True)功能说明
create_embedding_file: 从指定的文件中读取数据，并创建文本嵌入。支持的文件格式包括 CSV、XLS、XLSX 和 JSON。输出文件格式可以是 CSV 或 JSON。
search_from_file: 在提供的数据文件中搜索与用户查询最相似的文档。支持的文件格式同上。注意事项
某些方法（如 upload_embedding 和 search_docs_server）目前还在开发中，暂时不可用。
贡献
欢迎对此项目进行贡献。请确保在提交 Pull Request 之前测试您的代码。许可证
此项目根据 MIT 许可证发布。
Package: embedded-jubatus
embedded-jubatus-pythonembedded-jubatus-python is a Python bridge to callJubatus CoreC++ library.The interface of embedded-jubatus-python is the same as that of RPC client classes.
See theAPI Referencefor details.Installpip install embedded_jubatusRequirementsPython 2.7, 3.5, 3.6 or 3.7Jubatus (Python Client)1.0.2 or laternumpy 1.14-1.16 (1.17 or later don’t support Python 2.7)Jubatusneeds to be installed when build from source package or repositoryCythonneeds to be installed when build from repositoryLimitationsThe following methods are currently unavailable:get_status,get_proxy_status,do_mix,get_name,set_nameandget_client.savemethod saves the model file as/tmp/127.0.0.1_0_${type}_${id}.jubatus, where${type}is a name of the service (classifier,recommender, etc.) and${id}is the value specified as an argument to thesavemethod.
If you need to save the model into a different location, callsave_bytesmethod, which returns the model data as a binary, then save the returned bytes to the preferred location.
The same rule applies toload/load_bytesmethods.LicenseLGPL 2.1
Package: embedding4bert
Embedding4BERTTable of ContentsUser GuideInstallationUsageCitationReferencesThis is a Python library of extracting word embeddings from pre-trained language models.User GuideInstallationpipinstall--upgradeembedding4bertUsageExtract word embeddings of pretrained BERT models.Sum the representations of the last four layers.Take the mean of the representation of subword pieces as the word representations.Extract BERT word embeddings.fromembedding4bertimportEmbedding4BERTemb4bert=Embedding4BERT("bert-base-cased")# bert-base-uncasedtokens,embeddings=emb4bert.extract_word_embeddings('This is a python library for extracting word representations from BERT.')print(tokens)print(embeddings.shape)Expected output:14tokens:[CLS]ThisisapythonlibraryforextractingwordrepresentationsfromBERT.[SEP],19word-tokens:['[CLS]','This','is','a','p','##yt','##hon','library','for','extract','##ing','word','representations','from','B','##ER','##T','.','[SEP]']['[CLS]','This','is','a','python','library','for','extracting','word','representations','from','BERT','.','[SEP]'](14,768)Extract XLNet word embeddings.fromembedding4bertimportEmbedding4BERTemb4bert=Embedding4BERT("xlnet-base-cased")tokens,embeddings=emb4bert.extract_word_embeddings('This is a python library for extracting word representations from BERT.')print(tokens)print(embeddings.shape)Expected output:11tokens:ThisisapythonlibraryforextractingwordrepresentationsfromBERT.,16word-tokens:['▁This','▁is','▁a','▁','py','thon','▁library','▁for','▁extract','ing','▁word','▁representations','▁from','▁B','ERT','.']['▁This','▁is','▁a','▁python','▁library','▁for','▁extracting','▁word','▁representations','▁from','▁BERT.'](11,768)CitationFor attribution in academic contexts, please cite this work as:@misc{chai2020-embedding4bert,
  author = {Chai, Yekun},
  title = {embedding4bert: A python library for extracting word embeddings from pre-trained language models},
  year = {2020},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/cyk1337/embedding4bert}}
}ReferencesBERT: Pre-training of Deep Bidirectional Transformers for Language UnderstandingXLNet: Generalized Autoregressive Pretraining for Language Understanding
Package: embedding-adapter
Embedding Adapter 💬 📐Finetune embedding models in just 4 lines of code.Quick Start ⚡Installationpipinstallembedding_adapterUsagefromembedding_adapterimportEmbeddingAdapteradapter=EmbeddingAdapter()EmbeddingAdapter.fit(query_embeddings,document_embeddings,labels)EmbeddingAdapter.transform(new_embeddings)Once you've trained the adapter, you can use patch your pre-trained embedding model.patch=EmbeddingAdapter.patch()adapted_embeddings=patch(original_embedding_fn("SAMPLE_TEXT"))Synthetic Label Generation 🧪No user feedback to use as labels? 🤔 Create synthetic labels with theLabelGeneratorutilfromembedding_adapter.utilsimportLabelGeneratorgenerator=LabelGenerator()generator.run()Note:This requires an OpenAI API key saved as anOPENAI_API_KEYenv var.License 📄This project is licensed under the MIT License.
Package: embeddingcache
embeddingcacheRetrieve text embeddings, but cache them locally if we have already computed them.MotivationIf you are doing a handful of different NLP tasks, or have a single
NLP pipeline you keep tuning, you probably don't want to recompute
embeddings. Hence, we cache them.Quickstartpip install embeddingcachefrom embeddingcache.embeddingcache import get_embeddings
embeddings = get_embeddings(
            strs=["hi", "I love Berlin."],
            embedding_model="all-MiniLM-L6-v2",
            db_directory=Path("dbs/"),
            verbose=True,
        )Design assumptionsWe use SQLite3 to cache embeddings. [This could be adapted easily,
since we use SQLAlchemy.]We assume read-heavy loads, with one concurrent writer. (However,
we retry on write failures.)We shard SQLite3 into two databases:
hashstring.db: hashstring table. Each row is a (unique, primary
key) SHA512 hash to text (also unique). Both fields are indexed.[embedding_model_name].db: embedding table. Each row is a (unique,
primary key) SHA512 hash to a 1-dim numpy (float32) vector, which
we serialize to the table as bytes.Developer instructionspre-commit install
pip install -e .
pytestTODOUpdate pyproject.tomlAdd testsConsider other hash functions?float32 and float64 supportConsider adding optional joblib for caching?Different ways of computing embeddings (e.g. using an API) rather than locallyS3 backup and/orWALLiteStreamRetry on write errorsOther DB backendsBest practices: Give specific OpenAI version number.RocksDB / RocksDB-cloud?Include model name in DB for sanity check on slugify.Validate on numpy array size.Validate BLOB size for hashes.Add optional libraries like openai and sentence-transformersAlso consider other embedding providers, e.g. cohereAnd libs just for devsConsider the max_length of each text to embed, warn if we exceedpdoc3 and/or sphinxNormalize embeddings by default, but add optionOption to return torch tensorsConsider reusing the same DB connection instead of creating it
from scratch every time.Add batch_size parameter?Test check for collisionsUse logging not verbose output.Rewrite using classes.Fix dependabot.Don't keep re-using DB session, store it in the class or globalDRY.Suggest to use versioned OpenAI modelAdd device to sentence transformersAllow fast_sentence_transformersTest that things work if there are duplicate stringsRemove DBs after testDo we have to have nested embedding.embedding for all calls?codecov and code quality shields
Package: embeddingdb
This package provides a database schema and Python wrapper
for storing the embeddings generated through various representation
learning packages.Currently, this package focuses on using a SQL database with SQLAlchemy,
but might be extended to use a NoSQL database as an alternative.InstallationInstallembeddingdbdirectly from GitHub with:$pipinstallgit+https://github.com/cthoyt/embeddingdbSet the environment variableEMBEDDINGDB_CONNECTIONto a valid
SQLAlchemy connection string for a PostgreSQL instance, as this package uses
the PostgreSQL-specificARRAYtype.Command Line InterfaceThis package installs an entrypointembeddingdbthat can be used directly from
the shell.Uploading Entity EmbeddingsEntities can be embedded and stored from various types of representation learning,
including network representation learning, knowledge graph embedding, and textual
learning.Upload embeddings generated byword2vecby specifying the file path with:$embeddingdbupload--fmtword2vec--path~/path/to/file.txtUpload embeddings generated bypykeenby specifying the output directory
with:$embeddingdbupload--fmtkeen--path~/path/to/directory/Listing Entity EmbeddingsAfter uploading, the collections can be listed with:$embeddingdblsAnalyzing Entity Embeddings’ CorrelationsOne of the motivations for building this repository was to make a convenient way to
compare the embeddings for entities generated through orthogonal embedding tecnhiques.
For example, we wanted to know to what extent the embeddings for proteins generated from
their sequences withratveccontained the same information as the embeddings generated
from protein-protein interaction networks withpykeenornrl.The two positional arguments correspond to the collection identifiers in the database.$embeddingdbanalyze12Running with DockerAfter installing Docker, the entire web application can be instantiated with:$docker-composeupGet the endpoint/testto instantiate the database and add a test collection.
Package: embedding-evaluator
Embedding EvaluatorEmbeddingEvaluator is a tool to provide metrics for evaluating different embedding models.The current version only supports evaluation of evaluate only embeddings of the type:FastTextIt evaluates the embeddings based on the two following metrics:AnalogyOutlier DetectionInstallationThe EmbeddingEvaluator can be installed from PyPi:pipinstallembeddingevaluatorUsageAnalogy MetricsTo use the EmbeddingEvaluator to measure different embeddings basead on analogy metrics the user needs a file with the following configuration:Word 1Word 2Word 3Word 41st Pair 1st Word1st Pair 2nd Word2nd Pair 1st Word2nd Pair 2nd WordMenKingWomenQueenOuliter DetectionTo use the EmbeddingEvaluator to measure different embeddings basead on outlier detection metrics the user needs a file with the following configuration:Eight words which are semantically very similar and are all connected with each other by a clear well-known relation. (Cluster)Two words which are very similar to the ones in the cluster.Two words which are similar and related to the ones in the cluster.Two words which are related, but not similar to the ones in the cluster.Two words which are unrelated and not similar to the ones in the cluster.Initialize the EmbeddingEvaluatorThe EmbeddingEvaluator has three parameters as input:Input Metrics:A dictionary with a list of the paths for the input evaluation files.Example:input_metric={'analogy':['file_1','file_2'],'outlier':['file_1']}Input Models:
A dictionary with the model names and the paths to the models.Example:input_model={'model_1':'path_1','model_2':'path_2'}Initialize the class:emb_evaluator=EmbeddingMetrics(input_metric,input_model)Summary a model's metricsTo summarize the metrics of a model.emb_evaluator.summary_metrics('model_1')Compare models metricsTo compare the metrics of two or more models.emb_evaluator.compare_models(['model_1','model_2'])ReferencesLevy, O. and Goldberg, Y.: Linguistic Regularities in Sparse and Explicit Word Representations (2014)
Collados, J.C. and Navigli, R.: Find the word that does not belong: A Framework for an Intrinsic Evaluation of Word Vector Representations (2016)
Package: embeddinghub
Website|Docs|API Docs|Community forumEmbedding Hubis an open-source embedding management and storage solution.FeaturesSupported Operations: Run approximate nearest neighbor lookups, average multiple embeddings, partition tables (spaces), cache locally while training, and more.Storage: Store and index billions vectors embeddings from our storage layer.Versioning: Create, manage, and rollback different versions of your embeddings.Access Control: Encode different business logic and user management directly into the embedding hub.Monitoring: Keep track of how embeddings are being used, latency, throughput, and feature drift over time.Try it outQuickstartExplore our docsContributingTo contribute to Embedding Hub, please check outContribution docs.Welcome to our awesome community, please join ourSlack community.Report IssuesPlease help us byreporting any issuesyou may have while using Embedding Hub.LicenseMozilla Public License Version 2.0
Package: embeddings
Embeddings is a python package that provides pretrained word embeddings for natural language processing and machine learning.Instead of loading a large file to query for embeddings,embeddingsis backed by a database and fast to load and query:>>>%timeitGloveEmbedding('common_crawl_840',d_emb=300)100loops,bestof3:12.7msperloop>>>%timeitGloveEmbedding('common_crawl_840',d_emb=300).emb('canada')100loops,bestof3:12.9msperloop>>>g=GloveEmbedding('common_crawl_840',d_emb=300)>>>%timeit-n1g.emb('canada')1loop,bestof3:38.2µsperloopInstallationpipinstallembeddings# from pypipipinstallgit+https://github.com/vzhong/embeddings.git# from githubUsageUpon first use, the embeddings are first downloaded to disk in the form of a SQLite database.
This may take a long time for large embeddings such as GloVe.
Further usage of the embeddings are directly queried against the database.
Embedding databases are stored in the$EMBEDDINGS_ROOTdirectory (defaults to~/.embeddings). Note that this location is probablyundesirableif your home directory is on NFS, as it would slow down database queries significantly.fromembeddingsimportGloveEmbedding,FastTextEmbedding,KazumaCharEmbedding,ConcatEmbeddingg=GloveEmbedding('common_crawl_840',d_emb=300,show_progress=True)f=FastTextEmbedding()k=KazumaCharEmbedding()c=ConcatEmbedding([g,f,k])forwin['canada','vancouver','toronto']:print('embedding{}'.format(w))print(g.emb(w))print(f.emb(w))print(k.emb(w))print(c.emb(w))DockerIf you use Docker, an image prepopulated with the Common Crawl 840 GloVe embeddings and Kazuma Hashimoto’s character ngram embeddings is available atvzhong/embeddings.
To mount volumes from this container, set$EMBEDDINGS_ROOTin your container to/opt/embeddings.For example:dockerrun--volumes-fromvzhong/embeddings-eEMBEDDINGS_ROOT='/opt/embeddings'myimagepythontrain.pyContributionPull requests welcome!
Package: embeddings-for-trees
Embeddings for treesSet of PyTorch modules for developing and evaluating different algorithms for embedding trees.RequirementsYou can install the dependencies by using the requirement listpipinstall-rrequirements.txtAlthough it's better to installPyTorchandDGLmanually for correct CUDA support.Data preprocessingList of some useful tools for preprocessing source code into a dataset with ASTs:ASTMiner-
tool for mining raw ASTs and path-based representation of code using ANTLR, GumTree and other grammars.PSIMiner-
tool for processing PSI trees from IntelliJ IDEA and creating labeled dataset from them.Model zooList of supported algorithms of tree embeddings with links to wiki guides:TreeLSTMContributionSupporting different algorithms of encoding and decoding trees is the key area of improvement for this framework.
If you have any suggestions or questions, feel free to open issues and create pull requests.