Saravanan666/pypi-org huggingface dataset:
Package: semantic-kernel
About Semantic KernelSemantic Kernel (SK)is a lightweight SDK enabling integration of AI Large
Language Models (LLMs) with conventional programming languages. The SK
extensible programming model combines natural languagesemantic functions,
traditional codenative functions, andembeddings-based memoryunlocking
new potential and adding value to applications with AI.Semantic Kernel incorporates cutting-edge design patterns from the latest in AI
research. This enables developers to augment their applications with advanced
capabilities, such as prompt engineering, prompt chaining, retrieval-augmented
generation, contextual and long-term vectorized memory, embeddings,
summarization, zero or few-shot learning, semantic indexing, recursive
reasoning, intelligent planning, and access to external knowledge stores and
proprietary data.Getting Started ⚡Learn more at thedocumentation site.Join theDiscord community.Follow the team onSemantic Kernel blog.Check out theGitHub repositoryfor the latest updates.
Package: semantic-kit
Semantic Similarity and Relatedness ToolkitA toolkit to estimate semantic similarity and relatedness between two words/sentences.Installationpip install semantic-kitFunctionsLesk algorithm and improved versionSimilarity algorithms including WordNet , word2vec similarity, LDA, and googlenews-based methodsDistance algorithms like jaccard, soren, levenshtein, and their improved versionsUse Open Multilingual Wordnet to generate relevant keywords from multiple languageExamplesLesk Algorithmfromsemantickit.relatedness.leskimportleskfromsemantickit.relatedness.lesk_max_overlapimportlesk_max_overlapsent=['I','went','to','the','bank','to','deposit','money','.']m1,s1=lesk(sent,'bank','n')m2,s2=lesk_max_overlap(sent,'bank','n')print(m1,s1)print(m2,s2)WordNet-based Similarityfromsemantickit.similarity.wordnet_similarityimportwordnet_similarity_allprint(wordnet_similarity_all("dog.n.1","cat.n.1"))Corpus-based Similarityfromsemantickit.similarity.word2vec_similarityimportbuild_model,similarity_model# build similarity model based on text sourcebuild_model(data_path="text8",save_path="wiki_model")# estimate similarity between words using the built modelsim=similarity_model("wiki_model","france","spain")# print resultprint("word2vec similarity: ",sim)Pre-trained model-based Similarityfromsemantickit.similarity.googlenews_similarityimportgooglenews_similaritydata_path=r'GoogleNews-vectors-negative300.bin'sim=googlenews_similarity(data_path,'human','people')print(sim)Weighted Levenshteinfromsemantickit.distance.n_gram.train_ngramimportTrainNgramfromsemantickit.distance.weighted_levenshteinimportweighted_levenshtein,Build_TFIDF# train modeltrain_data_path='wlev/icd10_train.txt'wordict_path='wlev/word_dict.model'transdict_path='wlev/trans_dict.model'words_path="wlev/dict_words.txt"trainer=TrainNgram()trainer.train(train_data_path,wordict_path,transdict_path)# build words tf-idf fileBuild_TFIDF(train_data_path,words_path)# estimate weight lev distances0='颈结缔组织良性肿瘤's1='耳软骨良性肿瘤'result=weighted_levenshtein(s0,s1,word_dict_path=wordict_path,trans_dict_path=transdict_path,data_path=train_data_path,words_path=words_path)print(result)Get related words# use wordnet to generate multi-lang keywordsfromsemantickit.lang.wordnetimport*if__name__=="__main__":text="digitalization meets carbon neutrality, digital economy"nltk.download("wordnet")nltk.download('omw')dict_lang_all=get_all_related_word_from_text(text)print()forlangindict_lang_all:print(lang,','.join(dict_lang_all[lang]))Chinese WordVector Similarityfromsemantickit.similarity.chinese_word2vec_similarityimportChineseWord2Veccwv=ChineseWord2Vec(data_path="data/source",output_segment_path="data/segment",stop_words_path="data/stop_words.txt",user_dict_path="data/user_dict.txt",word2vec_model_path="models/word2Vec.model")cwv.train()sim=cwv.similarity('沙瑞金','易学习')print(sim)LicenseTheSemantic-Kitproject is provided byDonghua Chen.
Package: semanticlayertools
ReviewSemanticLayerToolsCollects tools to create semantic layers in the socio-epistemic networks framework. Source material can be any structured corpus with metadata of authors, time, and at least one text column.Documentation is available onReadTheDocs.Part of the code was reviewed byItay Zandbank, thank you.Installationtl;dr Use pippipinstallsemanticlayertoolsConsider using a clean virtual environment to keep your main packages separated.
Create a new virtual environment and install the packagepython3-mvenvenvsourceenv/bin/activate
pipinstallsemanticlayertoolsTo use some sentence embedding utility functions please install with theembeddmloptionpipinstallsemanticlayertools[embeddml]TestingTests can be run by installing thedevrequirements and runningtox.pipinstallsemanticlayertools[dev]toxBuilding documentationThe documentation is build usingsphinx. Install with thedevoption and runpipinstallsemanticlayertools[dev]tox-edocsFunding informationThe development is part of the research projectModelSENSocio-epistemic networks: Modelling Historical Knowledge Processes,in Department I of the Max Planck Institute for the History of Science
and funded by the Federal Ministry of Education and Research, Germany (Grant No. 01 UG2131).
Package: semantic-link
Semantic link is a feature that allows you to establish a connection betweenPower BI datasetsandSynapse Data Science in Microsoft Fabric.The primary goals of semantic link are to facilitate data connectivity, enable the propagation of semantic information, and seamlessly integrate with established tools used by data scientists, such as notebooks.Semantic link helps you to preserve domain knowledge about data semantics in a standardized way that can speed up data analysis and reduce errors.Package (PyPi)|API reference documentation|Product documentation|SamplesBy downloading, installing, using or accessing this distribution package for semantic link, you agree to theTerms of Service.This package has been tested with Microsoft Fabric.Getting startedPrerequisitesAMicrosoft Fabric subscription. Or sign up for a freeMicrosoft Fabric (Preview) trial.Sign in toMicrosoft Fabric.Createa new notebookor a newspark jobto use this package.Note that semantic link is supported only within Microsoft Fabric.About the semantic link packagesThe functionalities for semantic link are split into multiple packages to allow for a modular installation.
If you want to install only a subset of the semantic link functionalities, you can install the individual packages instead of thesemantic-linkmeta-package.
This can help solve dependency issues. The following are some of the available packages:semantic-link- The meta-package that depends on all the individual semantic link packages and serves as a convenient way to install all the semantic link packages at once.semantic-link-sempy- The package that contains the core semantic link functionality.semantic-link-functions-holidays- A package that contains semantic functions for holidays and dependence onholidays.semantic-link-functions-geopandas- A package that contains semantic functions for geospatial data and dependence ongeopandas....Install thesemantic-linkmeta packageTo install thesemantic-linkpackage in Microsoft Fabric, you have two options:Install theSemPyPython library in your notebook kernel by executing this code in a notebook cell:%pipinstallsemantic-linkAlternatively, you can add semantic link to your workspace libraries directly. For more information, seeInstall workspace libraries.Key conceptsSemPy offers the following capabilitites:Connectivity to Power BIConnectivity through Power BI Spark native connectorData augmentation with Power BI measuresSemantic propagation for pandas usersBuilt-in and custom semantic functionsNext stepsView ourSamples
Package: semantic-link-functions-geopandas
FabricDataFrames dynamically expose semantic functions based on logic defined by each function. For example, the is_holiday function shows up in the autocomplete suggestions when you're working on a FabricDataFrame containing both a datetime column and a country column.Each semantic function uses information about the data types, metadata (such as Power BI data categories), and the data in a FabricDataFrame or FabricSeries to determine its relevance to the particular data on which you're working.Semantic functions are automatically discovered when annotated with the @semantic_function decorator. You can think of semantic functions as being similar to C# extension methods applied to the popular DataFrame concept.fromsempy.fabricimportFabricDataFramedf=FabricDataFrame({"country":["US","AT"],"lat":[40.7128,47.8095],"long":[-74.0060,13.0550]},column_metadata={"lat":{"data_category":"Latitude"},"long":{"data_category":"Longitude"}},)df_geo=df.to_geopandas(lat_col="lat",long_col="long")
Package: semantic-link-functions-holidays
FabricDataFrames dynamically expose semantic functions based on logic defined by each function. For example, the is_holiday function shows up in the autocomplete suggestions when you're working on a FabricDataFrame containing both a datetime column and a country column.Each semantic function uses information about the data types, metadata (such as Power BI data categories), and the data in a FabricDataFrame or FabricSeries to determine its relevance to the particular data on which you're working.Semantic functions are automatically discovered when annotated with the @semantic_function decorator. You can think of semantic functions as being similar to C# extension methods applied to the popular DataFrame concept.fromsempy.fabricimportFabricDataFramedf=FabricDataFrame({"country":["US","AT"],"date":["2023-01-06","2023-01-06"]},column_metadata={"country":{"data_category":"Country"}},)df["date"]=pd.to_datetime(df["date"])holiday_series=df.is_holiday(date_col="date",country_col="country")
Package: semantic-link-functions-meteostat
FabricDataFrames dynamically expose semantic functions based on logic defined by each function. For example, the is_holiday function shows up in the autocomplete suggestions when you're working on a FabricDataFrame containing both a datetime column and a country column.Each semantic function uses information about the data types, metadata (such as Power BI data categories), and the data in a FabricDataFrame or FabricSeries to determine its relevance to the particular data on which you're working.Semantic functions are automatically discovered when annotated with the @semantic_function decorator. You can think of semantic functions as being similar to C# extension methods applied to the popular DataFrame concept.fromsempy.fabricimportFabricDataFramedf=FabricDataFrame({"lat":[40.7128,47.8095,47.8095,47.8095],"long":[-74.0060,13.0550,13.0550,13.0550],"date":["2023-01-06","2023-01-06","2023-01-07","2023-01-10"]},column_metadata={"lat":{"data_category":"Latitude"},"long":{"data_category":"Longitude"}},)df["date"]=pd.to_datetime(df["date"])df_weather=df.add_weather_meteostat("lat","long","date")
Package: semantic-link-functions-phonenumbers
FabricDataFrames dynamically expose semantic functions based on logic defined by each function. For example, the is_holiday function shows up in the autocomplete suggestions when you're working on a FabricDataFrame containing both a datetime column and a country column.Each semantic function uses information about the data types, metadata (such as Power BI data categories), and the data in a FabricDataFrame or FabricSeries to determine its relevance to the particular data on which you're working.Semantic functions are automatically discovered when annotated with the @semantic_function decorator. You can think of semantic functions as being similar to C# extension methods applied to the popular DataFrame concept.fromsempy.fabricimportFabricDataFramedf=FabricDataFrame({"num":["+442083661177","ABC000-0000","abc"],"country":["US","AT","Unknown"]},column_metadata={"country":{"data_category":"Country"}},)is_phone_series=df.is_phonenumber(col_num="num",col_country="country")
Package: semantic-link-functions-validators
FabricDataFrames dynamically expose semantic functions based on logic defined by each function. For example, the is_holiday function shows up in the autocomplete suggestions when you're working on a FabricDataFrame containing both a datetime column and a country column.Each semantic function uses information about the data types, metadata (such as Power BI data categories), and the data in a FabricDataFrame or FabricSeries to determine its relevance to the particular data on which you're working.Semantic functions are automatically discovered when annotated with the @semantic_function decorator. You can think of semantic functions as being similar to C# extension methods applied to the popular DataFrame concept.fromsempy.fabricimportFabricDataFramedf=FabricDataFrame({"contact_email":["a@b.com","d.com"],"amex":["378282246310005","4242424242424242"],"iban":["DE29100500001061045672","123456"],"es_nie":["X0095892M","X0095892X"]})df["contact_email"].validators.is_email()
Package: semantic-link-sempy
Semantic link is a feature that allows you to establish a connection betweenPower BI datasetsandSynapse Data Science in Microsoft Fabric.The primary goals of semantic link are to facilitate data connectivity, enable the propagation of semantic information, and seamlessly integrate with established tools used by data scientists, such as notebooks.Semantic link helps you to preserve domain knowledge about data semantics in a standardized way that can speed up data analysis and reduce errors.Package (PyPi)|API reference documentation|Product documentation|SamplesBy downloading, installing, using or accessing this distribution package for semantic link, you agree to theTerms of Service.This package has been tested with Microsoft Fabric.Getting startedPrerequisitesAMicrosoft Fabric subscription. Or sign up for a freeMicrosoft Fabric (Preview) trial.Sign in toMicrosoft Fabric.Createa new notebookor a newspark jobto use this package.Note that semantic link is supported only within Microsoft Fabric.About the semantic link packagesThe functionalities for semantic link are split into multiple packages to allow for a modular installation.
If you want to install only a subset of the semantic link functionalities, you can install the individual packages instead of thesemantic-linkmeta-package.
This can help solve dependency issues. The following are some of the available packages:semantic-link- The meta-package that depends on all the individual semantic link packages and serves as a convenient way to install all the semantic link packages at once.semantic-link-sempy- The package that contains the core semantic link functionality.semantic-link-functions-holidays- A package that contains semantic functions for holidays and dependence onholidays.semantic-link-functions-geopandas- A package that contains semantic functions for geospatial data and dependence ongeopandas....Install thesemantic-linkmeta packageTo install thesemantic-linkpackage in Microsoft Fabric, you have two options:Install theSemPyPython library in your notebook kernel by executing this code in a notebook cell:%pipinstallsemantic-linkAlternatively, you can add semantic link to your workspace libraries directly. For more information, seeInstall workspace libraries.Key conceptsSemPy offers the following capabilitites:Connectivity to Power BIConnectivity through Power BI Spark native connectorData augmentation with Power BI measuresSemantic propagation for pandas usersBuilt-in and custom semantic functionsNext stepsView ourSamples
Package: semantic-locators
Semantic Locators in Python WebDriverSemantic locators can be used with Selenium WebDriver in a similar way toByXPathorByCssSelector. Currently only available for Python 3.6+.Install from PyPi:python -m pip install semantic-locatorsOnce installed, use Semantic Locators as follows:fromsemantic_locatorsimport(find_element_by_semantic_locator,find_elements_by_semantic_locator,closest_precise_locator_for,)...search_button=find_element_by_semantic_locator(driver,"{button 'Google search'}")all_buttons=find_elements_by_semantic_locator(driver,"{button}")generated=closest_precise_locator_for(search_button);# {button 'Google search'}General Semantic Locator documentation can be found onGitHub.
Package: semanticmatcher
semantic-matcherThis library is built to handle anything related to semantic matching.In its current state, it has two main uses:Find the closest matches of a user query to a text corpus, using sentence transformer encodings and FAISS for optimization.Measure the semantic similarity between two tables and determine common columns. Useful for detecting duplicates and determining which columns to join on.1. InstallationThis package can be installed after cloning by running "make install". Alternatively it can be installed using pip:pip install semanticmatcher2. Usage:This package can be used to simply search for a query within a text corpus.fromsemanticmatcher.searchimportsemantic_searchquery=["boy jumping"]corpus=["There was a young man running around town","The mayor is looking for a new house","I had pasta for dinner"]scores,indices=semantic_search(query,corpus)Additionally, the package can also be used to determine the similarity between two tables. It returns a matrix that compares each column.df1=pd.DataFrame({"col1":["hello","world"],"col2":["how","are"],"col3":["you","doing"]})df2=pd.DataFrame({"col1":["hola","mundo"],"col2":["como","estas"],"col3":["tu","haciendo"],})similarity_matrix_df=similarity_matrix(df1,df2)By default, the following sentence transformer model is used to encode the text: "all-MiniLM-L6-v2".3. Next StepsAdd functionality to allow users to join two tables on a column, depending on the similarity match between the columns.
Package: semantic-model
Semantic-modelAll data about data in one model.
The single source of truth about data in one model.Installpipinstallsemantic-modelCreate initial sm modelpython3-msemantic_model-modeone_to_one-inputexample.json>sm_model.jsonRun sm modelpython3-msemantic_model-inputexample.json-smsm_model.jsonDisplay sm modelsudodockerrun-iattumm/dsm_png:latest<sm_model.json>output.pngExtendAdd to the functionality by creating your own custom formatters.
And adding them into semantic-model.fromsemantic_modelimportPOSTFORMATdefstrip(item,**kwargs):returnitem.strip()POSTFORMAT["strip"]=strip
Package: semantic-navigator
semantic-navigatorAn active learning approach to query and search through large archival datasets.InstallationStable Release:pip install semantic-navigatorDevelopment Head:pip install git+https://github.com/CouncilDataProject/semantic-navigator.gitDocumentationFor full package documentation please visitCouncilDataProject.github.io/semantic-navigator.DevelopmentSeeCONTRIBUTING.mdfor information related to developing the code.MPLv2 License
Package: semantic-nutrition
Semantic NutritionThis repo demonstrates how our basic semantic nutrition API works. We submitted this process as a clinical abtract to MLHC 2020 (poster,abstract).This is a base version of what was presented at the MLHC, and operates on a limited dataset. The dataset we used for training and publication is not ours to share. If you're interested in using your own nutrition dataset with this API, you can designate a different (local or hosted) URL when you initialize the class. The USDA provides an excellentstarting pointto build a dataset.Installationpip install nutritiontoolsUsagefrom nutritiontools import Nutrition

n_obj = Nutrition() 

match = n_obj.semantic_search("I ate an apple") 

print(match)Connecting to mobile assistants, placing the API in an online server, and managing a databaseFor testing purposes, we usedSiri Shortcuts(iOS) andGoogle Assistant / IFTTT webhooks(Android) to send voice transcriptions toGoogle Firebase. Once received, the incoming data would trigger aFirebase Cloud Function. The Firebase Function relayed the voice transcription to ourSemantic Nutrition API(hosted online withFlaskandGoogle App Engine), which responded with estimates for nutrition data. The nutrition data was then logged in Google Firebase.While useful for testing and demonstrations, this method has been depecrated in favor of a native mobile application and private server / database. Please stay tuned for future developments, and feel free to use our API locally in the meantime!updated on 7/27/2020
Package: semantics2021-toy-languages
semantics2021_toy_languagesA Python implementation of the toy languages used inPart IB "Semantics of Programming Languages" (2020 - 2021)Source hosted at:https://github.com/MitalAshok/semantics2021_toy_languagesPackage hosted at:https://pypi.org/project/semantics2021-toy-languages/Use as a module, like:$python3-msemantics2021_toy_languages--helpYou should use a file likeexample.L1:l -> 1234, l0 -> -134, l1 -> 123
!l + !l0 + !l1Where the first line is the initial mappings, and any subsequent lines an L1 expression.See examples/ for more examples.
Package: semantic-search-faiss
Semantic search with FAISSThe idea of this project is to build a semantic search engine which can search across multiple research papers related to covid and return the response. This can pretty much help people who want to know about ongoing research with respect to covid'19I have used -retrieval-ranking method with faiss indexapproach for the faster retrieval of data for the given query.Installationpip install semantic-search-faissInference examplefromsemanticsearchimportsearch,utils,configfromsemanticsearch.pretrainedimportget_modelfromsentence_transformersimportCrossEncoderbi_encoder,index,documents=get_model(config.BI_ENCODER,config.INDEX,config.DATA)cross_encoder=CrossEncoder(config.CROSS_ENCODER)query='death rates of covid'results=search.search(query,index,bi_encoder,cross_encoder,documents)Training pipeline1. Synthetic query generation using T5
2. Finetuning Bi-encoder using the synthetic query
3. Indexing the data with FAISS using finetuned BI-encoder
4. Bi-encoder + Cross encoder with FAISS searchTry out the code on google colab.KaggleDetailed walk through of the solution can be found in the below kaggle notebookAcknowledgementsI would like to thank Kaggle community as a whole for providing an avenue to learn and discuss latest data science/machine learning advancements.Vladimir Iglovikov for his wonderful article"I trained a model. What is next?"Xhululufor the dataset.