Saravanan666/pypi-org huggingface dataset:
Package: embeddingsprep
EmbeddingsThis package is designed to provide easy-to-use python class and cli
interfaces to:clean corpuses in an efficient way in terms of computation timegenerate word2vec embeddings (based on gensim) and directly write them to a format that is compatible withTensorflow ProjectorThus, with two classes, or two commands, anyone should be able clean a corpus and generate embeddings that can be uploaded and visualized with Tensorflow Projector.Getting startedRequirementsThis packages requiresgensim,nltk, anddocoptto run. If
pip doesn't install this dependencies automatically, you can install it by
running :pipinstallnltkdocoptgensimInstallationTo install this package, simply run :pipinstallembeddingsprepFurther versions might include conda builds, but it's currently not the case.Main featuresPreprocessingFor Word2Vec, we want a soft yet important preprocessing. We want to denoise the text while keeping as much variety and information as possible. A detailed version of what is done during the preprocessing is availablehereUsage example :Creating and saving a loadable configuration:fromembeddingsprep.preprocessing.preprocessorimportPreprocessorConfig,Preprocessorconfig=PreprocessorConfig('/tmp/logdir')config.set_config(writing_dir='/tmp/outputs')config.save_config()prep=Preprocessor('/tmp/logdir')# Loads the config object in /tmp/logdir if it existsprep.fit('~/mydata/')# Fits the unigram & bigrams occurencesprep.filter()# Filters with all the config parametersprep.transform('~/mydata')# Transforms the texts with the filtered vocab.Word2VecFor the Word2Vec, we just wrote a simple wrapper that takes the
preprocessed files as an input, trains a Word2Vec model with gensim and writes the vocab, embeddings .tsv files that can be visualized with tensorflow projector (http://projector.tensorflow.org/)Usage example:fromembeddingsprep.models.word2vecimportWord2Vecmodel=Word2Vec(emb_size=300,window=5,epochs=3)model.train('./my-preprocessed-data/')model.save('./my-output-dir')ContributingAny github issue, contribution or suggestion is welcomed! You can open issues on thegithub repository.
Package: embeddings-prep
EmbeddingsEmbedding generation with text preprocessing.PreprocessorFor Word2Vec, we want a soft yet important preprocessing. We want to denoise the text while keeping as much variety and information as possible.Preprocesses the text/set of text in the following way :Detects and replaces numbers/float by a generic token 'FLOAT', 'INT'Add spaces in between punctuation so that tokenisation avoids adding 'word.' to the vocabulary instead of 'word', '.'Lowers wordsRecursive word phrases detection : with a simple probabilistic rule, gathers the tokens 'new', york' to a single token 'new_york'.Frequency Subsampling : discards unfrequent words with a probability depending on their frequency.Outputs a vocabulary file and the modified files.Usage example :frompreprocessing.preprocessorimportPreprocessorConfig,Preprocessorconfig=PreprocessorConfig('/tmp/logdir')config.set_config(writing_dir='/tmp/outputs')config.save_config()prep=Preprocessor('/tmp/logdir')prep.fit('~/mydata/')prep.filter()prep.transform('~/mydata')Word2VecFor the Word2Vec, we just wrote a simple cli wrapper that takes the
preprocessed files as an input, trains a Word2Vec model with gensim and writes the vocab, embeddings .tsv files that can be visualized with tensorflow projector (http://projector.tensorflow.org/)Usage example:pythontraining_word2vec.pyfile_dirwriting_dirDocumentationDocumentation is available here :https://sally14.github.io/embeddings/TODO :Clean code for CLI wrapperAlso write a python Word2Vec model class so that user doesn't have to switch from python to cliAlso write a cli wrapper for preprocessingMemory leak in preprocessor.transform
Package: embeddings-visualizer
Embeddings VisualizerEmbeddings Visualizeris a Python package that provides tools for visualizing text embeddings generated from the OpenAI API. The package uses Streamlit for creating interactive visualization dashboards, and can be executed within a local Python environment or deployed to a web server.InitializationInitialize the package using theinitcommand:ev initThis will guide you through the process of setting up your configuration. Please ensure you have your OpenAI API key available.UsageOnce initialized, you can start the Streamlit application using thestart-appcommand:ev start-appThis will start the Streamlit application where you can upload your dataset and interactively visualize the embeddings.To open the notebook, use theopen-notebookcommand:ev open-notebookThis will open theembeddings.ipynbJupyter notebook in your browser, where you can interactively experiment with generating and visualizing embeddings.RequirementsPython 3.9 or later is required to use this package. It also depends on several Python libraries, including Streamlit, Typer, numpy, pandas, OpenAI, python-dotenv, scikit-learn, plotly, matplotlib and langchain. These dependencies are automatically installed when you install theEmbeddings Visualizerpackage.For more information, please refer to thepyproject.tomlfile in the root directory of the project.
Package: embeddix
embeddixA small toolkit for processing word embeddings with numpy. You can useembeddixto convert .txt embeddings (such as glove files) to numpy and vice-versa.Installpipinstallembeddixor, after a git clone:python3setup.pyinstallUseExtract vocabulary from a txt embeddings fileembeddixextract--embeddings/absolute/path/to/embeddings.txtConvert from txt to numpyembeddixconvert--tonumpy--embeddings/absolute/path/to/embeddings.txtConvert from numpy to txtembeddixconvert--totxt--embeddings/absolute/path/to/embeddings.npyEvaluate DSM on intrinsic tasksEvaluate on lexical similarity (men, simlex, simverb) or concept categorization (essli, ap, battig)embeddixevaluate\--embeddings/absolute/path/to/embeddings.npy\--vocab/absolute/path/to/embeddings.vocab\--datasetinstrinsic_task_dataset_name
Package: embedeval
NLP Embeddings Evaluation ToolTheNLP Embeddings Evaluation Toolis a command line tool to evaluate Natural Language Processing Embeddings
using custom intrinsic and extrinsic tasks.Installationembedeval is available aspippackage:python-mpipinstallembedevalNOTE: it might not be installable as of today using pip with PyPI.
However, installing from source will work. Use.instead ofembedevalin the pip command.Getting startedRun theword-analogyTask on your Word Embedding:embedevalembedding.vec-tword-analogyRun theword-analogyandword-similarityTasks on your Word Embedding:embedevalembedding.vec-tword-analogy-tword-similarityDocumentationThe whole documentation of embedeval is available onRead The Docs.Supported platformsembedeval is supported on Windows, Mac and LinuxContributionYes, we are looking for some contributors and people who spread out a word about embedeval. Help us to improve these piece of software. You don't know what to do?
Just have a look at the Issues or create a new one.
Please have a look at theContributing Guidelines, too.Project Informationembedeval is released under the MIT license, its documentation lives atRead The Docs,
the code onGitHub,
and the latest release onPyPI.
It’s rigorously tested on Python 3.5+.If you'd like to contribute to embedeval you're most welcome and we've written alittle guideto get you started!This project is published underMIT.ATimo Furrerproject.- :tada: -
Package: embedin
Embedin - Empower AI with persistent memoryWhat is EmbedinEmbedin is a highly efficient software library designed to seamlessly convert widely-used databases, such as MySQL, PostgreSQL, and MS SQL Server, into a vector database with minimal effort. Its lightweight design enables quick and easy integration into your existing software stack, allowing you to leverage the benefits of a vector database without the need for extensive modifications to your current system.With fast indexing and retrieval, it's ideal for high-performance applications such as natural language processing, image recognition, and recommendation systems. Embedin's simple API and query language make it easy to use and integrate.InstallationpipinstallembedinQuick StartUsing memoryfromembedin.clientimportClientclient=Client(collection_name="test_collection",texts=["This is a test","Hello world!"])result=client.query("These are tests",top_k=1)print(result)Using Sqlitefromembedin.clientimportClienturl='sqlite:///test.db'client=Client(collection_name="test_collection",texts=["This is a test","Hello world!"])result=client.query("These are tests",top_k=1)Using PostgreSQLimportosfromembedin.clientimportClienturl=os.getenv('EMBEDIN_POSGRES_URL',"postgresql+psycopg2://embedin:embedin@localhost/embedin_db")client=Client(collection_name="test_collection",texts=["This is a test","Hello world!"])result=client.query("These are tests",top_k=1)Using MySQLimportosfromembedin.clientimportClienturl=os.getenv('EMBEDIN_MYSQL_URL',"mysql+pymysql://embedin:embedin@localhost/embedin_db")client=Client(collection_name="test_collection",texts=["This is a test","Hello world!"])result=client.query("These are tests",top_k=1)Using MS-SQLimportosfromembedin.clientimportClienturl=os.getenv('EMBEDIN_MSSQL_URL',"mssql+pymssql://sa:StrongPassword123@localhost/tempdb")client=Client(collection_name="test_collection",url=url)client.add_data(texts=["This is a test"],meta_data=[{"source":"abc4"}])result=client.query("These are tests",top_k=1)For developmentClone the repogitclonehttps://github.com/EmbedInAI/embedin.gitStart PostgreSQL DB (Only necessary for using PostgreSQL as the storage)cddocker
docker-composeupembedin-postgresStart MySQL DB (Only necessary for using MySQL as the storage)cddocker
docker-composeupembedin-mysqlStart MS-SQL DB (Only necessary for using MS-SQL as the storage)cddocker
docker-composeupembedin-mssqlRun unit test with coverage reportcoveragerun-munittestdiscover-stests-p'*.py'coveragereport
Package: embedisualization
Visualize embed of documentsInstall requirements and spacy modelYou could usecondaconda create --name embedisualization python=3.6source activate embedisualizationpip install -r requirements.txtpython -m spacy download en_vectors_web_lgThis spacy model (en_vectors_web_lg)[https://spacy.io/models/en#en_vectors_web_lg] is quite heavy (631MB)Install embedisualization libpip install embedisualizationExampleTo run exemplary visualisation go toexamplesdirectory and runpython sample_text_vis.pyIt will take minute or two to generate embeddings and create 2D vis. The new webpage with D3 visualisation will be presented.
Package: embedly_cards
Embedly-CardsEmbedly-cards is aPelicanplugin providing restucturedText directives to allow
easy embedding of external content usingEmbed.ly Cards.ReSTandmarkdownlive examples can also be viewed from a Pelican-built website.FeaturesEmbed content within a page or blog post easily, simply by specifying the URL of
the target page. Content is automatically recognised, extracted, and formatted as
a ‘card’; this may contain a short article preview, embedded video, picture etc.To preview a card, they can be generated online usingEmbed.ly.
Almost any site is compatible, including YouTube, Flickr, Google+, Maps, Wordpress etc.InstallationEmbedly-cards can be installed usingpip$pipinstallembedly-cardsor manually from the source code$pythonsetup.pyinstallOnce installed, simply add it to yourpelicanconf.pyconfiguration file:PLUGINS=[# ...'embedly_cards']If you are planning on embedding content in markdown.mdfiles,
you must also add it to theMD_EXTENSIONSoptions, like so:fromembedly_cardsimportEmbedlyCardExtensionMD_EXTENSIONS=['codehilite(css_class=highlight)','extra',# ...EmbedlyCardExtension()]ImportantIf creating theMD_EXTENSIONSvariable for the first time,
ensure that the Pelican'codehilite(css_class=highlight)'and'extra'markdown extensions are included in the list.UsageFor example, to embed a YouTube video in ReStructuredText:..embedly-card::https://www.youtube.com/watch?v=ZlfIVEy_YOAor in markdown:[!embedlycard](https://www.youtube.com/watch?v=ZlfIVEy_YOA)Or to embed an article/webpage in ReStructuredText:..embedly-card::http://physics.stackexchange.com/questions/5265/cooling-a-cup-of-coffee-with-help-of-a-spoonor in markdown:[!embedlycard](http://physics.stackexchange.com/questions/5265/cooling-a-cup-of-coffee-with-help-of-a-spoon)OptionsThecard-chrome(ReST) orchrome(markdown) option, if provided, specifies
whether or not to preserve the border around the card. By default, the border
will be removed automaticallyif Embed.ly supports it; however to force the
border to remain, you may pass:card-chrome:1(ReST),..embedly-card::https://www.youtube.com/watch?v=ZlfIVEy_YOA:card-chrome:1orchrome=1(markdown):[!embedlycard?chrome=1](https://www.youtube.com/watch?v=ZlfIVEy_YOA)
Package: embed-markdown
Embed-Markdown User GuideEmbed-Markdown embeds code snippets into Markdown files. It helps keep examples
up to date by syncing code referenced in Markdown files with your actual source
code. This also makes it easier to ensure samples are unit tested, etc. Snippets
can be extracted from any text file (not necessarily even a from a code file).TopicsInstallationQuick StartSource Code Referencesrelative_script_pathcmd1) byScope2) bySectionRemove Embedded Source CodeInstallationRunpip install embed-markdownMake sureembed-markdownis in the 'PATH'Runpip show embed-markdownto show where it's installedEdit~/.bash_profile(or equivalence of it) to include the pathRunsource ~/.bash_profileQuick StartTo use this tool, first add source code references to your Markdown files. Then
runembed-markdownto update all Markdown files inside the
current directory (and all subdirectories) with the latest source code
referenced.Source Code ReferencesSource code is embedded using a comment in your Markdown file like this:<!-- embed:relative_script_path:cmd:args -->- CAUTION: This should be placed on a line by itself to avoid losing data. Do- not modify embedded code directly because changes will be overwritten the- next time the script runs; instead, change the source code the embed- directive references.relative_script_pathThis is the relative path from the.mdfile to a source file, e.g.../src/api.js.cmdThis controls how a source code snippet is selected. There are two approaches:1) byScopeWhencmdis "scope", embed-md will extract the next code block (inside a pair
of "{" and "}"). In this mode,argsis a colon separated list of nested
scopes. The script scans through the target script and recursively narrows down
on each scope. When the innermost scope is found, the script extracts lines
until the next block scope ends (a complete pair of "{" and "}"). For example,<!-- embed:../src/api.js:scope:class SomeApi -->extractsclass SomeApi {
    // Some comments
    method (a) {
        if (a) {
            console.log(a)
        }
    }
}Whereas<!-- embed:../src/api.js:scope:class SomeApi:Some comments -->extracts// Some comments
method (a) {
    if (a) {
        console.log(a)
    }
}And<!-- embed:../src/api.js:scope:class SomeApi:Some comments:if (a) -->extractsif (a) {
    console.log(a)
}2) bySectionWhencmdis "section", embed-md will extract lines between the section start
and end patterns. In this mode,argsis two colon delimited strings with
patternstart_pattern:end_pattern. Code between lines with substringstart_patternandend_patternare extracted from the target script. For
example,<!-- embed:../example.js:section:Example 1 start:Example 1 end -->extractsconst a = 1from// Example 1 start
const a = 1
// Example 1 endRemove Embedded Source CodeRunembed-markdown --removeto remove all embedded code snippets, leaving only
the embed directive.
Package: embedmd
embedmd - embed Markdown into Markdown or HTMLembedmdis a command line tool to embed Markdown within
other documents. Right now you can embed Markdown within other
Markdown documents as well as embed markdown within HTML documents.This tool utilizes thePython-Markdownpackage.Installationpip install embedmdorpip install git+https://github.com/kylepollina/embedmdEmbed Markdown within HTMLPlace this text in your HTML file where you want to embed Markdown:<#INCLUDE filename.md>Wherefilename.mdis pointing to the Markdown file you wish to embed
in that spot of the HTML file. Then, run the tool from the command line.embedmdinput.htmlEmbed Markdown within other MarkdownPlace this text in your Markdown file where you want to embed markdown:[#INCLUDE filename.md]Wherefilename.mdis pointing to the Markdown file you wish to embed
in that spot of the HTML file. Then, run the tool from the command line.embedmdinput.mdExampleSay we have this HTML file...<!-- template.html --><!DOCTYPE html><html><head><metacharset="UTF-8"></head><body><divid="container"><#INCLUDE test1.md><#INCLUDE test2.md></div></body></html>... and we want to embed these Markdown documents within.# test1.mdHello world, this is brought to you using [embedmd](https://github.com/kylepollina/embedmd)# test2.md| date       | species
| :-----     | :-------
| 2020-07-21 | Red-winged blackbird
| 2020-07-21 | Eurasian Tree sparrow#include "test3.md"# test3.mdWort wort wortRunning theembedmdcommand:embedmd template.htmlwill print out<!-- template.html --><!DOCTYPE html><html><head><metacharset="UTF-8"></head><body><divid="container"><h1>test1.md</h1><p>Hello world, this is brought to you using<ahref="https://github.com/kylepollina/embedmd">embedmd</a></p><h1>test2.md</h1><table><thead><tr><thalign="left">date</th><thalign="left">species</th></tr></thead><tbody><tr><tdalign="left">2020-07-21</td><tdalign="left">Red-winged blackbird</td></tr><tr><tdalign="left">2020-07-21</td><tdalign="left">Eurasian Tree sparrow</td></tr></tbody></table><h1>test3.md</h1><p>Wort wort wort</p></div></body></html>LicenseMIT License
Copyright Kyle PollinaPermission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
Package: embedme
EmbedmeEmbedme is a python module that allows you to easily use embeddings from text fields with OpenAI's Embedding API and store them in a local folder.It's like a lazy version of pinecone - Numpy is actually pretty fast for embeddings stuff at smaller scale, why overthink stuff? We store the data and vectors as json and build the numpy array before you search (and store it until you add more)InstallationTo install Embedme, you can use pip:pipinstallembedmeSetupThe only thing youmustdo before you useembedmeis setup auth with OpenAI. We use it to embed your items and search queries, so it is required. I don't want to touchanyof that code - just sign in how they tell you to, either in the script via a file for the key, or an environment variable for your key.OpenAI Python Module (With Auth Instructions)UsageEmbedme provides a simple interface to use embeddings from text fields with OpenAI's Embedding API and store them in a local folder.Check out the example notebook for a better example, but useage is something like:importopenaiimportnltkfrommore_itertoolsimportchunkedfromembedmeimportEmbedmefromtqdmimporttqdm# Downloading the NLTK corpusnltk.download('gutenberg')# Creating an instance of the Embedme classembedme=Embedme(data_folder='.embedme',model="text-embedding-ada-002")# Getting the texttext=nltk.corpus.gutenberg.raw('melville-moby_dick.txt')# Splitting the text into sentencessentences=nltk.sent_tokenize(text)input("Hey this call will cost you money and take a minute. Like, a few cents probably, but wanted to warn you.")fori,chunkinenumerate(tqdm(chunked(sentences,20))):data={'name':f'moby_dick_chunk_{i}','text':' '.join(chunk)}embedme.add(data,save=False)embedme.save()And to search:embedme.search("lessons")You can do anything you would want to with.vectorsafter you call.prepare_search()(or... search something, it's automatic mostly), like plot clusters, etc.Follow UsSome friends and I are writing about large language model stuff atSensibleDefaults.io, honest to god free. Follow us (or star this repo!) if this helps you!NoteEmbedme uses OpenAI's Embedding API to get embeddings for text fields, so an API key is required to use it. You can get one fromhttps://beta.openai.com/signup/The token limit today is about 8k, so... you're probably fine
Package: embedml
embedMLpytorch like machine learning framework from scratch
I started the repo with building a simple compiler for embeded platfroms, but later decided to make a self contained abstract baby machine learning framework.Could be used mostly for the Educational purpose as well. Easy and full python implemtation using numpy.
Smaller sibling or framworks like pytorch or jax :) and useful to see how the behind the scene of those bigger platfrom.and it is only less than 200 lines of code!Instalpip install embedmlExample tarinig simiar to pytorch:y=model(x)loss=criterion(y,t)loss.backward()opt.step()opt.zero_grad()or like this implement the gradient decent:params=model.get_parameters()forparaminparams:param-=param.grad*self.lrExample of differentiation of a funtionform Jax example:fromjaximportgradimportjax.numpyasjnpdeftanh(x):# Define a functiony=jnp.exp(-2.0*x)return(1.0-y)/(1.0+y)grad_tanh=grad(tanh)# Obtain its gradient functionprint(grad_tanh(1.0))# Evaluate it at x = 1.0# prints 0.4199743and similar (backward gradient) with embedML:importnumpyasnpfromembedml.tensorimportTensordeftanh(x):# Define a functiony=(-2*x).exp()return(1-y).div((1+y))x=Tensor(np.array(1))y=tanh(x)y.backward()print(x.grad)