Saravanan666/pypi-org huggingface dataset:
Package: sema
semasema is a setting maker that lets user change the setting safely and with lots of information about what can be done.What is semaI have faced many setting files that all the available options for a setting or meaning of changes are hard to find
or the only way to understand the options is through indirect sites. the other issue is the certainty that
am i doing the changes the right way or will i break the system with the wrong syntax? In essence settings are painful to
deal with, but i am tring to make it easier through giving user a program to alter settings with and shippng informations
about settings with the setting it self in a seprate file, so user has some suggestions and informations about what can be
done.Future of semaThis is just a beta version, there are futures that are not done yet like: support for multivalues for an
option and much more, and there are bugs and also possibility of revisions.So basiclly there is much to be done, at this phase i just want people to see the idea and give feedbacks to me, i would
love to hear about ideas and know why you like or hate this idea. Feel free to ask or say anything.InstallationUsepip install semato installsemaor download sema repository from github and open terminal in root of the project then typepython setup.py buildafter that typepython setup.py installFor creating and managing settings for developerssemaandsema -cliare available in terminal for creating and managing settings for developers.Reading settings in your programTo extract settings in your program usefrom sema import extract, for now there is only 1 function in the library which
returns the value for given option nameget_value(file_name, option_name)Altering the settings for usersUse change_setting_gui.py and change_setting_cli.py files in change_setting folder to alter settings.To give the addresses of .setting file(s) you want this script to be responsible for, there are 2 ways:1. Enter the addresses in sema_files file line by line, in the same folder as the script2. Enter them indot_setting_listinside the script
Package: semantha-streamlit-compare
semantha-streamlit-compare: semantha building blocks for Streamlit appsThis project gives you an idea of how to use our component for building applications with streamlit. And all of this using semantha's native capabilities to process semantics in text.tl;dr: using Streamlit, you can employ semantha's semantic comparison with just three lines of code (see below).Which Components Are Involved?Streamlit.io offers easy GUI implementations. semantha.ai is a semantic processing platform which provides a REST/API for many use cases, end systems, etc.🚀 QuickstartYou can installsemantha-streamlit-comparefrom pip:pipinstallsemantha-streamlit-compareThen put the following example code in a file.fromsemantha_streamlit_compare.components.compareimportSemanticComparecompare=SemanticCompare()compare.build_input(sentences=("First sentence","Second sentence"))🎛 Setup, Secret, and API KeyTo use the component, you need to request a secrets.toml file. You can request that atsupport@thingsthinking.atlassian.netThe file is structured as follows:[semantha]
server_url="URL_TO_SERVER"
api_key="YOUR_API_KEY_ISSUED"
domain="USAGE_DOMAIN_PROVIDED_TO_YOU"
documenttype="document_with_contradiction_enabled"
Package: semantic
Semantic is a Python library for extracting semantic information from text, including dates, numbers, mathematical equations, and unit conversions.For each of these four semantic types, semantic provides a service module. Typical usage often looks like this:#!/usr/bin/env python
from semantic.dates import DateService

service = DateService()
date = service.extractDate("On March 3 at 12:15pm...")
...The full documentation can be foundhere, while the source code itself is also available onGitHub.InstallationInstalling semantic is simple:$ pip install semanticFeaturessemantic contains four main modules, each of which corresponds to a different semantic extractor.Dates (date.py)Useful for:Extracting relative (e.g., “a week from today”) and absolute (e.g., “December 11, 2013”) dates from text snippets.Converting date objects to human-ready phrasing.Numbers (number.py)Useful for:Extracting numbers (integers or floats) from text snippets.Converting numbers to human-readable strings.Example usage:#!/usr/bin/env python
from semantic.numbers import NumberService

service = NumberService()

print service.parse("Two hundred and six")
# 206

print service.parse("Five point one five")
# 5.15

print service.parse("Eleven and two thirds")
# 11.666666666666666

print service.parseMagnitude("7e-05")
# "seven to the negative five"Math (solver.py)Useful for performing mathematical operations expressed as words.Example usage:#!/usr/bin/env python
from semantic.solver import MathService

service = MathService()

print service.parseEquation("Log one hundred and ten")
# 4.70048Units (units.py)Useful for converting between units expressed as words.Example usage:#!/usr/bin/env python
from semantic.solver import ConversionService

service = ConversionService()

print service.convert("Seven and a half kilograms to pounds")
# (16.534, 'lbs')

print service.convert("Seven and a half pounds per square foot to kilograms per meter squared")
# (36.618, 'kg/m**2')TestingThe test suite (test.py) contains tons of examples and use-cases for each of the four modules.RequirementsThe Dates, Numbers, and Math modules can run in isolation (i.e., without any dependencies), while the Units module requiresquantitiesandNumpy.LicenseMIT ©Charles Marsh
Package: semantic3
Semantic is a Python library for extracting semantic information from text, including dates, numbers, mathematical equations, and unit conversions.For each of these four semantic types, semantic provides a service module. Typical usage often looks like this:#!/usr/bin/env python
from semantic.dates import DateService

service = DateService()
date = service.extractDate("On March 3 at 12:15pm...")
...The full documentation can be foundhere, while the source code itself is also available onGitHub.InstallationInstalling semantic is simple:$ pip install semanticFeaturessemantic contains four main modules, each of which corresponds to a different semantic extractor.Dates (date.py)Useful for:Extracting relative (e.g., “a week from today”) and absolute (e.g., “December 11, 2013”) dates from text snippets.Converting date objects to human-ready phrasing.Numbers (number.py)Useful for:Extracting numbers (integers or floats) from text snippets.Converting numbers to human-readable strings.Example usage:#!/usr/bin/env python
from semantic.numbers import NumberService

service = NumberService()

print service.parse("Two hundred and six")
# 206

print service.parse("Five point one five")
# 5.15

print service.parse("Eleven and two thirds")
# 11.666666666666666

print service.parseMagnitude("7e-05")
# "seven to the negative five"Math (solver.py)Useful for performing mathematical operations expressed as words.Example usage:#!/usr/bin/env python
from semantic.solver import MathService

service = MathService()

print service.parseEquation("Log one hundred and ten")
# 4.70048Units (units.py)Useful for converting between units expressed as words.Example usage:#!/usr/bin/env python
from semantic.units import ConversionService

service = ConversionService()

print service.convert("Seven and a half kilograms to pounds")
# (16.534, 'lbs')

print service.convert("Seven and a half pounds per square foot to kilograms per meter squared")
# (36.618, 'kg/m**2')TestingThe test suite (test.py) contains tons of examples and use-cases for each of the four modules.RequirementsThe Dates, Numbers, and Math modules can run in isolation (i.e., without any dependencies), while the Units module requiresquantitiesandNumpy.LicenseMIT ©Charles Marsh
Package: semantically
semanticallyIntroductionSemantically is a Python library designed to easily retrieve data from Semantic Scholar. All data is typed in dataclasses built around the Semantic Scholar API. Furthermore semantically supports methods to retrieve papers by their title in order.
For larger requests semantically offers methods to retrieve papers according to lists of keywords. In the background these requests are run asynchronously so that several requests can be sent in a short amount of time.RequirementsPython 3.6+Semantically relies mainly onLevenshteindaciteaiohttpInstallationSemantically can be installed via pip withpip install semanticallyExamplesUsing semantically to retrieve paper information is as easy asimportsemanticallysemantic_scholar=semantically.Semantically()paper_info=semantic_scholar.get_paper_by_title("Supporting the Exploration of Semantic Features in Academic Literature using Graph-based Visualizations")print(paper_info)# Paper(paperId='175ac9646f3a7b6491c6bad896c47c495633d54a', externalIds={'MAG': '3046741163', 'DBLP': 'conf/jcdl/BreitingerKMMG20', ...Getting an author by name:importsemanticallysemantic_scholar=semantically.Semantically()author_info=semantic_scholar.get_author_by_name("Albert Einstein")print(author_info[0])# DetailedAuthor(authorId='2059041927', externalIds={}, url='https://www.semanticscholar.org/author/2059041927',...This example queries the API for 7 keywords, each containing 10 results:importsemanticallykeywords=["Recommender System","Blockchain","Natural Language Processing","Plagiarism Detection","News Analysis","Information Retrieval","Machine Learning"]semantic_scholar=semantically.Semantically()paper_info=semantic_scholar.get_papers_by_keyword(keywords,limit=10)print(paper_info["Natural Language Processing"][4])# Paper(paperId='084c55d6432265785e3ff86a2e900a49d501c00a', externalIds=None, url=None, title='Foundations of statistical natural language processing'
Package: semantic-bibtool
Command Line Bib Generator via Semantic Scholar APIAutomatic generation of bibs from a single title or a list of titles withsemantic scholarAPI.This repository is in no way affiliated with Semantic Scholar.InstallationObtain an API key from Semantic Scholar:here.Place your API key (a string letters and numbers) into the environment
variableexport SEMANTIC_SCHOLAR_API_KEY="...".Install using pip:pip install semantic_bibtoolExample usage$semantic_bibtool"attention is all you need"prints@inproceedings{vaswani2017attention,
  author = {Vaswani, Ashish and M. Shazeer, Noam and Parmar, Niki and Uszkoreit, Jakob and Jones, Llion and N. Gomez, Aidan and Kaiser, Lukasz and Polosukhin, Illia},
  title = {{A}ttention is {A}ll you {N}eed},
  booktitle = {{N}{I}{P}{S}},
  year = {2017},
}Helpusage: semantic_bibtool [-h] [-o OUTPUT_FILE] [--add-url] input

A tool for converting paper titles (with optional author names as keywords) to formatted latex bib format using Semantic Scholar API.

NOT AFFILIATED WITH Semantic Scholar.

You need to obtain an API key for Semantic Scholar, which you can request here: https://www.semanticscholar.org/product/api#Partner-Form Export it in your environment as `export SEMANTIC_SCHOLAR_API_KEY="..."`

positional arguments:
  input                 input, either:
                                * a (quoted) string
                                * a .txt file with one title per line
                                * a zotero .csv export
                        This tool infers data type from file extension, make sure it matches!

options:
  -h, --help            show this help message and exit
  -o OUTPUT_FILE, --output-file OUTPUT_FILE
                        stdout by default
  --add-url             whether to add the url to the paper in the bib file

Example use:
        $ pip install .
        $ semantic_bibtool "attention is all you need"
        $ semantic_bibtool titles.txt -o references.bib
        $ semantic_bibtool zotero.csv
Package: semantic-cleaning
semantic-cleaningInstallpipinstallsemantic-cleaningHow to usefromdatasetsimportload_datasetimportnumpyasnpfromtransformersimportAutoTokenizer,AutoModelfromsemantic_cleaningimportpreprocess_data,compute_embeddings,deduplicate_embeddings,deduplicate_datasetProcessing a dataset to get a sentence for QA or comment and response
etc.data=load_dataset("0-hero/OIG-small-chip2")_=preprocess_data(data,schema=":{user}:{chip2}")data['train']['_merged'][0]2Compute the embadding fot the sentencestokenizer=AutoTokenizer.from_pretrained('sentence-transformers/all-mpnet-base-v2')model=AutoModel.from_pretrained('sentence-transformers/all-mpnet-base-v2').to('cuda')embedding=compute_embeddings(data=data,embedding_model=model,tokenizer=tokenizer,batch_size=64,num_workers=16,dataset_feature='_merged'):We can get the indicis of all the duplicated lines with the folowing
command:to_delete=deduplicate_embeddings(embedded=embeddeing,epsilon=1e-2,batch_size=20000)You could also find duplication between two datasets or splits like
this:to_delete=deduplicate_embeddings(embedded=embeddeing,embedded2=embeddeing2,epsilon=1e-2,batch_size=20000)The full process could be run like thisdeduplicated=deduplicate_dataset(dataset=data['train'],model=model,tokenizer=tokenizer,epsilon=1e-2,model_batch_size=64,deduplication_batch_size=20000,num_workers=16,dataset_feature='_merged')print(f"cleaned:{(1-len(deduplicated)/len(data['train']))*100:.2f}:%")And deduplicated can be pushed back to the hub or saved on local driveCommand-Line InterfaceThe semantic cleaning module also includes a command-line interface that
can be used to deduplicate datasets:pythonsemantic-cleaning.py\--model_path"sentence-transformers/all-mpnet-base-v2"\--tokenizer_path"sentence-transformers/all-mpnet-base-v2"\--dataset_path"0-hero/OIG-small-chip2"\--output_path"./deduplicated_imdb"The following arguments are available:–dataset_path: Path to the dataset to be deduplicated.–model_path: The model checkpoint for embeddings. Should be a path or
model id in HuggingFace model hub.–tokenizer_path: The tokenizer to be used.–epsilon: Threshold for cosine similarity to consider embeddings as
duplicates.–model_batch_size: Batch size for the model.–deduplication_batch_size: Batch size for the deduplication process.–num_workers: Number of worker processes for data loading.–dataset_feature: Feature in the dataset to be used for deduplication.–output_path: Path to save the deduplicated dataset. Can be a local
path or a HuggingFace dataset repository.–hub_repo: Repository on the Hugging Face hub to push the dataset.–hub_token: HuggingFace Hub token to push the dataset to the Hub.
Required when hub_repo is provided.–device: Device to use for computations (e.g., ‘cpu’, ‘cuda’,
‘cuda:1’). If not provided, it will use CUDA if available, otherwise
CPU.You can use the –help flag to get a description of all options:pythonsemantic-cleaning.py--help
Package: semantic-codebase-inspection
Semantic Codebase InspectionSCI helps developers familiarize themselves with a python project that has decent naming conventions.It requires the libraries:tensorflow-text
tensorflow-hub
inspectIt downloads an NLP model of about 300MB.https://tfhub.dev/google/universal-sentence-encoder-multilingual-large/3It uses a Natural Language Processing model to turn the names of functions, classes, and variables (variables that outside of functions) into vectors. Queries can be submitted against those names which don't need to be exact matches. Currently, the language model being used accommodates 16 languages, which means it accommodates synonyms too. This is accomplished with technology related to word2vec, doc2vec, and embeddings. If this seems somehow magic to you, I encourage you to read more about it so that you can be a magician too because it's not that complicated.>>importsemantic_codebase_inspection>>semantic_codebase_inspection.run_semantic_search_through_program()(printed)Importingmodulesandsemanticmodel.(printed)Whatmodulewouldyouliketoexplore?(input)requests(printed)Whatfunctioninthelibraryareyouinterestinginexploring?(input)get(printed)Howmanyobjectswouldyoulikeinyourresult?(integeranswersonly)(input)5*waitingforsearchtoexecute*Similarsemanticcandidatesinclude:get:getfunctionput:putfunction__build__:buildint__url__:urlstrpost:postfunctionExit with ctrl+C.
Package: semantic-commit
Semantic Commit (scommit)Semantic Commit (scommit) is a Python script that uses the ChatGPT API to auto-generate commit messages for your git commits. You can use it exactly likegit commit:scommitis equivalent to:gitcommit-m"auto-generated commit message"You can also use all of the args you normally would withgit commit, for example:scommit-ais equivalent to:gitcommit-a-m"auto-generated commit message"Think ofscommitas an alias forgit committhat appends-m "message"with an auto-generated message. That's it! I recommend using this when you don't actually care about commit messages, but want them to be slightly more informative than "sdflskdjafks". Plus, it uses more GPU's than regular commits, and we all know GPU usage is a proxy for agency.In addition, you can use the-mioption withscommitto run it locally using Ollama. For example:scommit-miThis will runscommitlocally using Ollama instead of making a request to the ChatGPT API.Installationpipinstallsemantic-commitYou should get command-line tool calledscommitthat you can use just like git commmit.You also need to set yourOpenAI API keyas an environment variable named OPENAI_API_KEY for scommit to work. Add this line in your.bashrcor.zshrc:exportOPENAI_API_KEY=your-api-keyLicenseThis project is licensed under the terms of the MIT license. See theLICENSEfile for details.
Package: semantic-counter
Wrapper around CountVectorizer to count phrases based on semantic similaritySemanticCountVectorizer extends CountVectorizer by counting the frequency of semantically similar tokens. This is useful when searching for frequency of similar but not exact terms.SemanticCountVectorizer uses sentence_transformers to compute embedding of n-grams and then clusters similar terms using Agglomerative Clustering using cosine similarity as the similarity metricInstallationpip install semantic_counterUsagefrom semantic_counter.semantic_vectorizer import SemanticCountVectorizer 
sentences = ['this is a test sentences','this is another test sentence']
svect = SemanticCountVectorizer(embedding_model_name='all-MiniLM-L6-v2', similarity_threshold=0.7, ngram_range=(3,3))
counts=svect.fit_transform(sentences)Argumentsembedding_model_name: Name of the sentence_transformer pretrained model to use. Refer topackage websitefor a list of available modelssimilarity_threshold: value between 0-1 which indicates minimum cosine similarity needed to group tokens.Arguments of sklearn's CountVectorizer as listedhere
Package: semantic-fscacher
fscacherCaching solution for functions that operate on the file system.Installationpip install <path to repository>UsageSimple usage:from fscacher import Cache

cache = Cache(cache_path)
fn = cache.memoize(expensive_fn)

result = fn("arg1", "arg2")Whenfnis called the first time, the functionexpensive_fnis evaluated
and its return value is serialized and stored atcache_path. Subsequent
calls tofndeserializes the stored result instead of re-evaluatingexpensive_fn.Optional arguments tomemoizeinclude:key: Function with arguments(func, args, kwargs)and return value of
typestr, used to create the function call signaturedump: Function with arguments(return_value, filename)and no return
value for serializing the result ofexpensive_fnload: Function with arguments(filename, )for de-serializing the
binary data on disk as a return valuedigest: Function with arguments(stream, )and return value of typestrfor digesting function call signature as well as the contents of serialized
filesprotocol: Use predefined functions forkey,dump,loadanddigest.
A list of known protocol schemes are presented below. If both protocol and
explicit functions are set, the explicit functions takes presedence.If any of the argumentskey,dump,loadordigestare set to"default", the functions defined by thecache.defaultsdict are used
instead.Default key function is constructed as follows:Arguments and keyword arguments (both keys and values) are converted to
string by thestrfunction. Numpy arrays are converted to lists before
conversion.Any arguments which are too long (> 22 chars) or contain invalid characters
(= \/:*?"<>|) are utf-8 encoded and converted to a 64-bit truncated
sha256 hash.Keyword arguments are joined as key-value pairs of the formk=v.If short enough, the key is the function name followed by the
space-separated argument list. If too long, the key is the function
name followed by the full sha256 hash of the space-separated argument
list.Default dump and load functions are from the python pickle module.Default digest function is sha256.Implemented protocols include:filename/<suffix>: Return value is interpreted as the name of a temporary
file, which should have the suffix<suffix>, including any leading dot.keyis default key, except that<suffix>is appended.dumpmoves the
file to the index location.loadreturns the file name as a string.filehash/<suffix>: Return value is interpreted as the name of a temporary
file, which should have the suffix<suffix>, including any leading dot.keyis default key with the suffix.hashappended.dumpcomputes the
hash of the temporary file and renames it to the hash value (unless already
present). Thereafter, it copies the file (as a hardlink if possible) to the
location specified bykey, except that.hashis replaced by<suffix>.
Finally, the hash is stored as a string (lowercase hex) at the location
specified bykey(the index location).loadreturns the contents of the
file at the index location and interprets it as apathlibpath. In the
end, this protocol works likefilename/<suffix>, except that multiple
function calls can be mapped to the same file if their return value has
equal contents.
Package: semantic-ink
INK: Instance Neighbouring using KnowledgeINK is a binary data structure for representing a knowledge graph (KG), taking into account nodes and their neighbourhoods till a certain, specified depth.This repository contains:The code to transform a KG into the binary INK matrix representationThe code INK Miner code: Semantic rule mining on the INK representation for both task-specific and task-agnostic use cases.Installationpip install semantic-inkor clone this repositoryUsageSee theexample notebookfor more information on how to use INK and the INK miner.PublicationsComing soon.ContributingPull requests are welcome.For major changes, please open an issue first to discuss what you would like to change.LicenseIMEC
Package: semanticjoshuadrc
Semantic NutritionThis repo demonstrates how our basic semantic nutrition API works. We submitted this process as a clinical abtract to MLHC 2020 (poster,abstract).This is a base version of what was presented at the MLHC, and operates on a limited dataset. The dataset we used for training and publication is not ours to share. If you're interested in using your own nutrition dataset with this API, you can designate a different (local or hosted) URL when you initialize the class. The USDA provides an excellentstarting pointto build a dataset.Installationpip install nutritiontoolsUsagefrom nutritiontools import Nutrition

n_obj = Nutrition() 

match = n_obj.semantic_search("I ate an apple") 

print(match)Connecting to mobile assistants, placing the API in an online server, and managing a databaseFor testing purposes, we usedSiri Shortcuts(iOS) andGoogle Assistant / IFTTT webhooks(Android) to send voice transcriptions toGoogle Firebase. Once received, the incoming data would trigger aFirebase Cloud Function. The Firebase Function relayed the voice transcription to ourSemantic Nutrition API(hosted online withFlaskandGoogle App Engine), which responded with estimates for nutrition data. The nutrition data was then logged in Google Firebase.While useful for testing and demonstrations, this method has been depecrated in favor of a native mobile application and private server / database. Please stay tuned for future developments, and feel free to use our API locally in the meantime!updated on 7/27/2020