Saravanan666/pypi-org huggingface dataset:
Package: databrowser
databrowserA easy file browser to view data files.Currently supports parquet,json and csv with the Pandas libraryBased on the example code_browser from Textual.installThe package is published on pypihttps://pypi.org/project/databrowser/runpip install databrowser(pip install databrowser --upgradeto get latest version)Then just executedatabrowserBuildExecutepoetry installto install the package and the dependenciesRunpython3 src/databrowser/data_browser.pyor runpoetry run databrowserto execute in virtual envUsagepython3 data_browser.py [optional path]S3 Supportwith the help of S3Path and S3fs the browser now supports s3.usedatabrowser s3://to start browsing bucketsordatabrowser s3://bucket/path/subdir/to browse a specific directory.it uses the default aws credentials in the environmentSelect a data file to viewpress F to hide the filebrowserpress D to show the dtypespress S to save a screenshot in svgpress Q to quitexamples
Package: data-buckaroo
Data BuckarooLightweight AWS Data Wrangler, for Athena queries only. Also has a lightweight "DataFrame" implementation if you prefer not to use Pandas.Installpipinstalldata-buckaroo# Optional if you would like to use Pandaspipinstalldata-buckaroo[pandas]Usagefromdata_buckarooimportAthenaQueryfromlightframeimportLightFrameaq=AthenaQuery(workgroup="ATHENA_WORKGROUP",database="ATHENA_DATABASE")lf:LightFrame=aq.read_sql_query("SELECT * FROM TABLE_NAME")print(lf)id  string_object  string  float  int        date            timestamp   bool  par0  par1
0   1            foo     foo    1.0    1  2020-01-01  2020-01-01 00:00:00   True     1     a
1   2                                                                                1     b
2   3            boo     boo    2.0    2  2020-01-02  2020-01-02 00:00:01  False     2     b
Package: databudgie
DatabudgieDatabudgie is a CLI & library for database performing targeted backup and
restore of database tables or arbitrary queries against database tables.UsageA minimal config file might look like:# databudgie.yml or config.databudgie.ymlbackup:url:postgresql://postgres:postgres@localhost:5432/postgrestables:-name:public.productquery:"select*from{table}wherestore_id>4"location:s3://my-s3-bucket/databudgie/public.productrestore:url:postgresql://postgres:postgres@localhost:5432/postgrestables:-name:public.productlocation:s3://my-s3-bucket/databudgie/public.productWith that config in place, backing up the defined tables (using the specified
config) is as simple asdatabudgie backup; and restoredatabudgie restore.Installationpipinstalldatabudgie
Package: databuild
makefor your data.An automation tool for data manipulation.Inspired by Open Refine.The general principles inDatabuildare:Low entry barrierEasy to installEasy to graspExtensibleDatabuildcan be useful for scenarios such as:Documenting data transformations in your infoviz projectAutomate data processing in a declarative wayInstallationInstall databuild:$ pip install databuildQuickstartFor more details, see theExtended Documentation.$ data-build.py buildfile.jsonbuildfile.yamlcontains a list of operations to be performed on
data. Think of it as a script for a spreadsheet.An example of build file could be:-operation:sheets.import_datadescription:Importing data from csv fileparams:sheet:dataset1format:csvfilename:dataset1.csvskip_last_lines:1-operation:columns.add_columndescription:Calculate the gender ratioparams:sheet:dataset1name:Gender Ratioexpression:language:pythoncontent:"returnfloat(row['TotaleMaschi'])/float(row['TotaleFemmine'])"-operation:sheets.export_datadescription:save the dataparams:sheet:dataset1format:csvfilename:dataset2.csvJSON buildfiles are also supported.databuildwill guess the type
based on the extension.LicenseLicensed under BSD 3-clauses.
Package: databuilder
DataBuilderHave you ever needed some dummy data to demonstrate some basic data analysis / machine learning topics?DataBuilder can save you time by creating customized dummy data sets within minutes.Installationpip install databuilderQuick Exampleimportdatabuilderasdb# make a dummy dataset about "our employees"config={'fields':{'empID':db.ID(),'first_name':db.Name(first_only=True),'last_name':db.Name(last_only=True),'department':db.Group(["Sales","Acct","Mktg","IT"]),'salary':db.NormalDist(50000,10000),'hire_date':db.Date("1990-01-01","2020-12-31")}}# create a Pandas DataFrame with# the fields defined in `config`df=db.create_df(config,n=200)print(df.head(2))##   Example output:#         empID first_name last_name department  salary  hire_date#      0      1      Frank      Ward         IT   69210 2004-05-05#      1      2    Barbara    George       Mktg   46744 2019-05-20Complete Usage GuideDetailed docs on how to use DataBuilder can be found in thedocs/folder of this repo (orclick here)
Package: databuild_lua
InstallationIn order to be able to use Lua, you’ll need to have the Lua environment
installed before you installdatabuild.LinuxRequirements:LuaorLuaJITOn Ubuntu, these are the packages you need:$ sudo apt-get install liblua5.1-0-dev liblua50-dev liblualib50-devThen you can installdatabuild:$ pip install https://github.com/fcurella/databuild/archive/master.tar.gzOS XRequirements:Lua(LuaJITis not supported ATM)Temporarily unlinkLuaJITif you have it installed:$ brew unlink luajitInstallLuawithbrew:$ brew install luaInstalldatabuildwith theWITHLUAenvironment variable set:$ WITHLUA=True pip install https://github.com/fcurella/databuild/archive/master.tar.gzAfterdatabuildis installed, you can re-linkLuaJITif you need:$ brew link luajitConfigurationAdd‘lua’: ‘databuild_lua.lua.LuaEnvironment’to yourLANGUAGESsettings:LANGUAGES = {
    'python': 'databuild.environments.python.PythonEnvironment',
    'lua': 'databuild_lua.lua.LuaEnvironment',
}
Package: databunnystoragedriver
Databunnystoragedriver contains code forked from iXsystem cinder driver code under BSD license.https://github.com/iXsystems/cinderThe purpose of this repository is provide a python package that can be easily installed seperate from Openstack cinder volume driver packageTested on Freenas 11.3, Truenas core 13, Truenas scale 22https://github.com/openstack/cinder
Package: databuri
Data·BuriAll-in-one Python package for Thailand's public APIsAPIs ImplementedNameDescriptionGovSpendingThai government procurement projectsAir4ThaiAir quality measurements in ThailandAPIs to be includedhttps://api-portal.sec.or.th/apideschttp://opend.openservice.in.th/opend/http://dolwms.dol.go.th/tvwebp/https://data.moc.go.thข้อมูลนิติบุคคล จาก กรมการค้าภายในDevelopmentReleaseBump version$ bumpversion [major|minor|patch]
$ git push && git push --tagsPublish to pypi$ ./scripts/publish.sh <tag>Related Links:Project IdeaProject Proposal
Package: data-butler
DataButlerA python-based software for automatic data profiling and catalogingIt takes a lot of man-hours and energy to transform data dumps into forms that are more understandable and suitable for databases. The objective of DataButler is to model a framework that performs data profiling and cataloging, thereby providing more efficient search/discover functionalities.Process FlowInstallation$pipinstall--index-urlhttps://test.pypi.org/simple/db_stDirectionsimportdata_butler

data_butler.db('file directory/filename.csv')AuthorsKeerthi Pullela, Rahul Madhu, Rukmini Sunil, Sagar Kurada, Xema PathakAcknowledgmentsProfessor Matthew Lanham,
Krannert School of Management,
Purdue UniversityMike Lutz, Caleb Keller,
Samtec Inc.
Package: databutton
databutton-cliThe CLI for building and deploying databutton projectsGetting StartedUsage:databutton[OPTIONS]COMMAND[ARGS]...

Options:-v,--verboseEnableverboselogging--helpShowthismessageandexit.

Commands:buildBuildtheproject,builtcomponentswillbefoundin.databuttoncreateCreateaDatabuttonprojectintheprovidedproject-directorydeployDeployyourprojecttoDatabuttondocsLauncheshttps://docs.databutton.cominitCreatesanewprojectinDatabuttonandwritestodatabutton.jsonloginLogintoDatabuttonlogoutRemovesallDatabuttonlogininfoserveStartsawebserverforproduction.startRuntheDatabuttondevelopmentserverversionGetthelibraryversion.whoamiShowstheloggedinuserDevelopingPrerequisitesThis project uses poetry, so if you haven't already;pip install poetryInstall dependenciespoetry installTestpoetry run pytest -sLint`make lint``All these are being run in a github action on pull requests and the main branch.Test locally in another packageTo test in another package, you can simplypip install -e .assuming you're in this folder. If not, replace the.with the path to thedatabutton-clifolder.AuthorsDatabutton-Initial work-githubLicense: Copyright (c) DatabuttonAll rights reserved.
Package: datacache
DataCacheHelpers for transparently downloading datasetsAPIfetch_file(downloadurl, filename = _None, decompress =False, subdir =None)fetch_and_transform(transformedfilename, transformer, loader,
source_filename, source_url, subdir = _None)fetch_fasta_dict(downloadurl, filename = _None, subdir =None)fetch_fasta_db(tablename, download_url, fasta_filename = _None,
key*column = *'id', value_column ='seq', subdir = _None)fetch_csv_db(tablename, download_url, csv_filename = _None, subdir =None,
**pandas_kwargs)
Package: datacamprojects
Thedatacamprojectspython packageSkip the boilerplate of scikit-learn machine learning examples.InstallationpipinstalldatacamprojectsUsageIn a shell environment, you can rundatacamprojectswith no arguments to perform a Logistic Regression
on thedigitsdataset.This will produce a 10 x 10 confusion matrix
with the Accuracy Score at the top.You can also pass arguments to datacamprojects at the command line.For example,datacamprojects-datasetdiabetes-modellinear_model.Lasso# Ordatacamprojects-ddiabetes-mlinear_model.Lassowill run a linear regression with lasso regularization (L1)
on thediabetesdataset.Thedatasetargument can be any of
the following built-in scikit-learn datasets:RegressionbostondiabetesClassificationdigitsiriswinebreast_cancerThemodelargument refers to the model type and name from scikit-learn.
The first part is the submodule, e.g.linear_modelnaive_bayesensemblesvmwhile the second is what is actually imported, e.g.LinearRegressionGaussianNBRandomForestRegressorSVCSimplify code to a single function call per step:fromsklearn.metricsimportconfusion_matrix,accuracy_scoreimportdatacamprojectsasdcpdataset=dcp.get_data('digits')x_train,x_test,y_train,y_test=dcp.split_data(dataset)model=dcp.get_model(model_type='ensemble',model_name='RandomForestClassifier')fit=model.fit(x_train,y_train)dcp.pickle_model(filename='digits_rf.pickle',model=fit)predictions=fit.predict(x_test)confmat=confusion_matrix(y_true=y_test,y_pred=predictions)accuracy=accuracy_score(y_true=y_test,y_pred=predictions)dcp.confusion_matrix_plot(cm=confmat,acc=accuracy,filename='digits_rf.png')Or run a whole pipeline with one function:importdatacamprojectsasdcpdcp.classification(dataset='digits',model_type='ensemble',model_name='RandomForestClassifier',pickle_name='digits_rf.pickle',plot_name='digits_rf.png')For inspiration, look at the example pipeline in thepipeline folderof thedatacamprojects repo.
Package: datacards
DatacardThis repo aims to find and update the missing model cards for Hugging face datasets.If you find this a worth while pursute, feel free to reach out and let's try to make the Hugging face datasets complete :wink:Setup# install poetrygitclone--recurse-submodules--remote-submodulesgit@github.com:Hugging-Face-Supporter/datacards.gitcddatacards
gitsubmoduleupdate

poetryinstallRunpoetryshell
pythondatacards/main.pyWIPLook into how to provide multiple answers in model card (ex. Glue dataset)Find the datasets that are missing information by parsing the READMEFind ways to know what categories are valid answersCreate method to filter for missing datasetsCreatetool to annotate the datasetsToggle between datasets to annotate.Save modified files to the README againOnce done, find ways to create automatic PR to Hugging face datasets
Package: datacatalog
Python package implementing core Data Catalog business logic*Software projects using this package:https://github.com/SD2E/pipeline-jobs-managerhttps://github.com/SD2E/pipelines-managerhttps://github.com/SD2E/demo-jobs-enabled-reactorhttps://github.com/SD2E/demo-jobs-enabled-reactorUsage_More information is forthcoming._
Package: datacatalog-custom-model-manager
datacatalog-custom-model-managerA Python package intended to load user-specified metadata models into Google Cloud Data Catalog,
comprisingCustom Entries, Tag
Templates, andTags.It is powered bydatacatalog-custom-entries-
manager,datacatalog-
tag-template-processor, anddatacatalog-tag-manager, leveraging
their features in the format of a single CLI.Load Custom ModelFrom CSV filesSCHEMASLoading custom models from CSV requires a folder (--files-folderargument in the below commands)
containing three files:entries.csv,tag_templates.csv, andtags.csv. Please refer to the
following external resources for further details on their layouts:FILELAYOUT DOCUMENTATIONentries.csvdatacatalog-custom-entries-manager READMEtag_templates.csvdatacatalog-tag-template-processor READMEtags.csvdatacatalog-tag-manager READMESAMPLE INPUTsample-input/egeria-business-glossaryfor reference;Sample metadata for datacatalog-custom-model-manager(Google Sheets) might help to create/export the mandatory CSV files.COMMANDSPython + virtualenvpipinstalldatacatalog-custom-model-manager

datacatalog-custom-modelload\--files-folder<CSV-FILES-PATH>\--project-id<YOUR-PROJECT-ID>--location-id<YOUR-LOCATION-ID>\[--delete-existing-tag-templates]Dockerdockerbuild--rm--tagdatacatalog-custom-model-manager.

dockerrun--rm--tty\--volume<CREDENTIALS-FILE-FOLDER>:/credentials--volume<CSV-FILE-FOLDER>:/data\datacatalog-custom-model-managerload\--files-folder<CSV-FILES-PATH>\--project-id<YOUR-PROJECT-ID>--location-id<YOUR-LOCATION-ID>\[--delete-existing-tag-templates]
Package: datacatalog-object-storage-processor
datacatalog-object-storage-processorA package for performing Data Catalog operations on object storage solutions.Table of Contents1. Environment setup1.1. Get the code1.2. Auth credentials1.2.1. Create a service account and grant it below roles1.2.2. Download a JSON key and save it as1.3. Virtualenv1.3.1. Install Python 3.6+1.3.2. Create and activate an isolated Python environment1.3.3. Install the dependencies1.3.4. Set environment variables1.4. Docker2. Create DataCatalog entries based on object storage files2.1. python main.py3 Delete up object storage entries on entry groupDisclaimers1. Environment setup1.1. Get the codegitclonehttps://github.com/mesmacosta/datacatalog-object-storage-processorcddatacatalog-object-storage-processor1.2. Auth credentials1.2.1. Create a service account and grant it below rolesData Catalog AdminStorage Admin or Custom Role with storage.buckets.list acl1.2.2. Download a JSON key and save it as./credentials/datacatalog-object-storage-processor-sa.json1.3. VirtualenvUsingvirtualenvis optional, but strongly recommended unless you useDocker.1.3.1. Install Python 3.6+1.3.2. Create and activate an isolated Python environmentpipinstall--upgradevirtualenv
python3-mvirtualenv--pythonpython3envsource./env/bin/activate1.3.3. Install the dependenciespipinstall--upgrade--editable.1.3.4. Set environment variablesexportGOOGLE_APPLICATION_CREDENTIALS=./credentials/datacatalog-object-storage-processor-sa.json1.4. DockerDocker may be used as an alternative to run all the scripts. In this case, please disregard theVirtualenvinstall instructions.2. Create DataCatalog entries based on object storage files2.1. python main.pypythondatacatalog-object-storage-processor\object-storagecreate-entries--typecloud-storage\--project-idmy_project\--entry-group-namemy_entry_group_name\--bucket-prefixmy_bucketdockerdockerbuild--rm--tagdatacatalog-object-storage-processor.
dockerrun--rm--tty-vyour_credentials_folder:/datadatacatalog-object-storage-processor\--typecloud-storage\--project-idmy_project\--entry-group-namemy_entry_group_name\--bucket-prefixmy_bucket3 Delete up object storage entries on entry groupDelete entries for given entry groupdatacatalog-object-storage-processor\object-storagedelete-entries--typecloud-storage\--project-idmy_project\--entry-group-namemy_entry_group_nameDisclaimersThis is not an officially supported Google product.History0.1.0 (2020-05-01)First release on PyPI.
Package: datacatalog-tag-template-exporter
Datacatalog Tag Template ExporterA Python package to manage Google Cloud Data Catalog Tag Template export scripts.Disclaimer: This is not an officially supported Google product.Table of ContentsExecuting in Cloud Shell1. Environment setup1.1. Python + virtualenv1.1.1. Install Python 3.6+1.1.2. Get the source code1.1.3. Create and activate an isolated Python environment1.1.4. Install the package1.2. Docker1.3. Auth credentials1.3.1. Create a service account and grant it below roles1.3.2. Download a JSON key and save it as1.3.3. Set the environment variables2. Export Templates to CSV file2.1. A CSV file representing the Templates will be created2.2. Run the datacatalog-tag-template-exporter scriptExecuting in Cloud Shell# Set your SERVICE ACCOUNT, for instructions go to 1.3. Auth credentials# This name is just a suggestion, feel free to name it following your naming conventionsexportGOOGLE_APPLICATION_CREDENTIALS=~/datacatalog-tag-template-exporter-sa.json# Install datacatalog-tag-template-exporterpip3installdatacatalog-tag-template-exporter--user# Add to your PATHexportPATH=~/.local/bin:$PATH# Look for available commandsdatacatalog-tag-template-exporter--help1. Environment setup1.1. Python + virtualenvUsingvirtualenvis optional, but strongly recommended unless you useDocker.1.1.1. Install Python 3.6+1.1.2. Get the source codegitclonehttps://github.com/mesmacosta/datacatalog-tag-template-exportercd./datacatalog-tag-template-exporterAll paths starting with./in the next steps are relative to thedatacatalog-tag-template-exporterfolder.1.1.3. Create and activate an isolated Python environmentpipinstall--upgradevirtualenv
python3-mvirtualenv--pythonpython3envsource./env/bin/activate1.1.4. Install the packagepipinstall--upgrade.1.2. DockerDocker may be used as an alternative to run the script. In this case, please disregard theVirtualenvsetup instructions.1.3. Auth credentials1.3.1. Create a service account and grant it below rolesData Catalog Admin1.3.2. Download a JSON key and save it asThis name is just a suggestion, feel free to name it following your naming conventions./credentials/datacatalog-tag-template-exporter-sa.json1.3.3. Set the environment variablesThis step may be skipped if you're usingDocker.exportGOOGLE_APPLICATION_CREDENTIALS=~/credentials/datacatalog-tag-template-exporter-sa.json2. Export Templates to CSV file2.1. A CSV file representing the Templates will be createdTemplates are composed of as many lines as required to represent all of their fields. The columns are
described as follows:ColumnDescriptiontemplate_nameResource name of the Tag Template for the Tag.display_nameResource name of the Tag Template for the Tag.field_idId of the Tag Template field.field_display_nameDisplay name of the Tag Template field.field_typeType of the Tag Template field.enum_valuesValues for the Enum field.2.2. Run the datacatalog-tag-template-exporter scriptPython + virtualenvdatacatalog-tag-template-exportertag-templatesexport--project-idsmy-project--file-pathCSV_FILE_PATHHistory0.1.0 (2020-04-15)First release on PyPI.
Package: datacatalogtordf
datacatalogtordfA small Python library for mapping a data catalog to rdfThe library contains helper classes for the following dcat classes:CatalogDatasetDistributionData ServiceOther relevant classes are also supported, such as:Contactvcard:KindThe library will map tothe Norwegian Application Profileofthe DCAT standard.UsageInstall%pipinstalldatacatalogtordfGetting startedfromdatacatalogtordfimportCatalog,Dataset# Create catalog objectcatalog=Catalog()catalog.identifier="http://example.com/catalogs/1"catalog.title={"en":"A dataset catalog"}catalog.publisher="https://example.com/publishers/1"# Create a dataset:dataset=Dataset()dataset.identifier="http://example.com/datasets/1"dataset.title={"nb":"inntektsAPI","en":"incomeAPI"}## Add dataset to catalog:catalog.datasets.append(dataset)# get rdf representation in turtle (default)rdf=catalog.to_rdf(format="turtle")print(rdf.decode())Will produce the following output:@prefixdcat:<http://www.w3.org/ns/dcat#>.
@prefixdct:<http://purl.org/dc/terms/>.

<http://example.com/catalogs/1>adcat:Catalog;dct:publisher<https://example.com/publishers/1>;dct:title"A dataset catalog"@en;dcat:dataset<http://example.com/datasets/1>.

<http://example.com/datasets/1>adcat:Dataset;dct:title"incomeAPI"@en,"inntekstAPI"@nb.DevelopmentRequirementspyenv(recommended)python3pipx(recommended)poetrynox%pipxinstallpoetry==1.1.13
%pipxinstallnox==2022.1.7
%pipxinjectnoxnox-poetry==0.9.0Install developer tools%gitclonehttps://github.com/Informasjonsforvaltning/datacatalogtordf.git
%cddatacatalogtordf
%pyenvinstall3.8.12
%pyenvinstall3.9.10
%pyenvlocal3.8.123.9.10%poetryinstallRun all sessions%noxRun all tests with coverage reporting%nox-rstestsDebuggingYou can enter intoPdbby passing--pdbto pytest:nox-rstests----pdbYou can set breakpoints directly in code by using the functionbreakpoint().
Package: datacenter
==============
DataCenter-SDK面向DataCenter-Service#XARC的sdk和cli工具SDK 功能特性数据工具SMILES-转-SVGSMILES-转-SDF生化数据分子摘要、分子详情、分子搜索靶标摘要、靶标详情、靶标搜索应用自有空间：数据上传（APP-TOKEN-ONLY）数据列表（APP-TOKEN-ONLY）数据下载 (PUBLIC)数据集：见详细文档CLI 功能特性暂无详细文档方法一：自行编译获取代码拷贝之后，在终端执行make docs方法二：内网访问http://192.168.1.182/documents/datacenter-sdk-doc/========
发版历史0.0.1 (2020-10-28)First release on PyPI.0.0.2 (2020-12-25)第一次测试版本SDK:数据工具SMILES-转-SVGSMILES-转-SDF生化数据分子摘要、分子详情、分子搜索靶标摘要、靶标详情、靶标搜索应用自有空间：数据上传（APP-TOKEN-ONLY）数据列表（APP-TOKEN-ONLY）数据下载 (PUBLIC)数据集：见详细文档