Saravanan666/pypi-org huggingface dataset:
Package: data-wrangling-components
data-wrangling-componentsThis project provides a collection of utilities for doing lightweight data wrangling.There are two goals of the project:Create a shareable client/server schema for serialized wrangling instructionsMaintain an implementation of a basic wrangling engine (based on Arquero) and in the case of python implemented in PandasBuildingYou need to installpoetrypython package manager.Run:poetry installUsageThis project is inteded to be used as a library for lightweight data wrangling. In the examples folder there is aNotebookwhich provides several examples of how to create data wrangling pipelines and how to read json specifications that can be generated by the js implementation.Example of joining two tables:fromdata_wrangling_components.pipelineimportPipelineimportdata_wrangling_components.typesastypesimportpandasaspd# id   name# 1    bob# 2    joe# 3    janeparents=pd.DataFrame({"id":[1,2,3],"name":['bob','joe','jane']})# id   kid# 1    billy# 1    jill# 2    kaden# 2    kyle# 3    moekids=pd.DataFrame({"id":[1,1,2,2,3],"kid":['billy','jill','kaden','kyle','moe']})pipeline=Pipeline()pipeline.add_dataset('parents',parents)pipeline.add_dataset('kids',kids)pipeline.add(Step(verb=Verb.join,input="parents",output="output",args={"other":"kids","on":["id"]}))# id   name    kid# 1    bob     billy# 1    bob     jill# 2    joe     kaden# 2    joe     kyle# 3    jane    moeresult=pipeline.run()ContributingThis project welcomes contributions and suggestions. Most contributions require you to agree to a
Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us
the rights to use your contribution. For details, visithttps://cla.opensource.microsoft.com.When you submit a pull request, a CLA bot will automatically determine whether you need to provide
a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions
provided by the bot. You will only need to do this once across all repos using our CLA.This project has adopted theMicrosoft Open Source Code of Conduct.
For more information see theCode of Conduct FAQor
contactopencode@microsoft.comwith any additional questions or comments.TrademarksThis project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft
trademarks or logos is subject to and must followMicrosoft's Trademark & Brand Guidelines.
Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship.
Any use of third-party trademarks or logos are subject to those third-party's policies.
Package: datawrapper
datawrapperA light-weight python wrapper for the Datawrapper API (v3). While it is not developed by Datawrapper officially, you can use it with your API credentials from datawrapper.de🚀 FeaturesRetrieve your account information (including folders).Add data to charts, tables or maps.Create charts, tables or maps - and add data from apandas.DataFramein one call!Update chart descriptions.Publish charts, tables or maps.Retrieve chart properties, update its metadata, and other information.Display a chart (as output of notebook cell - it gets weird because interactivity ¯\(ツ)/¯ )Retrieve a chart, table or map's iframe code to embed.Export chart as png (still working on the svg and pdf parts).Move charts across folders and organizations.Delete charts.Get a list of all your charts.Installationpipinstall-Udatawrapperorpipenvpipenvinstalldatawrapperor install withPoetrypoetryadddatawrapperContributingClone the repository. Move into the directory on your terminal.Install dependencies for development.pipenvinstall--devInstall pre-commit to run a battery of automatic quick fixes against your work.pipenvrunpre-commitinstall📈 ReleasesYou can see the list of available releases on theGitHub Releasespage.We followSemantic Versionsspecification. When you're ready to make a new release, visit the releases page and create a new entry. Set the tags and press publish. That will trigger a GitHub Action that automatically deploys the code to the Python Package Index.🛡 LicenseThis project is licensed under the terms of theMITlicense. SeeLICENSEfor more details.📃 Citation@misc{datawrapper,
  author = {chekos},
  title = {A light-weight python wrapper for the Datawrapper API (v3). While it is not developed by Datawrapper officially, you can use it with your API credentials from datawrapper.de},
  year = {2021},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/chekos/datawrapper}}
}CreditsThis project was generated withpython-package-template.
Package: data-writers
DescriptionThis project is useful in any scenario where you need to write some data in a readable manner quickly and easily.It can be used to write to:csvsqlitemysqlredismongopostgresaerospikeLook athttps://pypi.python.org/pypi/data-readers/to write to them. This library was written with the intention to be useful withoutdata-readersbut they work very well together.SetupInstallpip install data-writersUsageImport DataWriterfrom datawriters.datawriter import DataWriterCreate a list of dictionaries for your data:data = [{“column1”:”row1-item1”, “column2”:”row1-item2”},
{“column1”:”row2-item1”, “column2”:”row2-item2”},
{“column1”:”row3-item1”, “column2”:”row3-item2”}]csvDataWriter(writer=’csv’, database=’data.csv’).save(data)sqliteDataWriter(writer=’sqlite’, database=’data.sqlite’, table=’DataTable’).save(data)mysqlDataWriter(writer=’mysql’, database=’data’, user=’root’, table=’DataTable’).save(data)redisDataWriter(writer=’redis’, database=’1’).save(data)mongoDataWriter(writer=’mongo’, database=’data’, table=’DataTable’).save(data)postgresDataWriter(writer=’postgres’, database=’data’, table=’DataTable’).save(data)aerospikeDataWriter(writer=’aerospike’, namespace=’data’, set=’DataTable’).save(data)
Package: datax
DataXplorerRequires Python 3.7Install withpip install datax, then run withdx [OPTIONS] [FILES]....Options:-v, --verbose      Print verbose messages.
  -i, --interactive  Enter into an interactive loop to query data.
  -q, --query TEXT   The query string.
  --help             Show this message and exit.Example:dx -q "search n>5" file.txtwherefile.txtlooks like{"event": 1, "n": "2", "ip": 7, "type": "geoip"}
{"event": 2, "n": "6", "ip": 10, "type": "geoip"}
{"event": 3, "n": "8", "ip": 15, "type": "geoip"}-q TEXTis not required if using interactive mode (-i). When in interactive mode, you will enter an input loop, allowing you to repeatedly query data without having to execute the command again. This is faster because the data is kept in memory and doesn't have to be loaded from disk each query. Typeexit,quit, or enterCTRL+Cto quit. Typesearch <query-string>to search the data. If the-qand-ioptions are used together, the query will be executed, results will be printed, then you will enter into an input loop.DataXplorer also accepts data from a pipe. Example:cat file.txt | dx -q "search n>5".DataXplorer usesDXQLto search through the data. See the DXQL readme to learn how to search.
Package: data_xray
Data-Xray provides a Pythonic interface to organization and analysis of
data, acquired primarily with Nanonis controllers, manufactured by Specs GmbH.The primary emphasis at the moment is on scanning probe microscopy measurements,
but most of the analysis applies to other domains of hyperspectral data.Data organization is achieved through the use of xarray structure - which is
an ingenious extension of Pandas phenomenology onto arbitrary dimensions.Data-Xray draws upon many truly excellent python libraries, for which we as developers
are phenomenally greatful.Installation / SetupUse pip to install the library:pip install data-xrayTestingThere are a few ways to run the unit tests.One option is to use the shell script in the root of the repository
calledtest.example.sh. Copy it usingcp test.example.sh test.sh.
Edittest.shto include your Cicero API username and password. Then, run
the tests using./test.sh.Another option is to edit thetest/tests.pyfile directly, adding your
Cicero API credentials where indicated. Doing so will allow you to execute
tests usingnosetests(if you have the nose package installed), or
usingpython setup.py test, or invoking thetests.pyfile itself.DocumentationHelp!Licensedata-xray is licensed under the Apache 2.0 license. SeeLICENSE.txtfor
more details.ContributeSee a bug? Want to improve the docs or provide more examples? Thank you!
Please open a pull-request with your improvements and we’ll work to respond
to it in a timely manner.
Package: datayoga
DataYoga CLIDevelopmentTo set up environment in development mode:Set Up Virtual Environmentpython-mvenvvenvsourcevenv/bin/activateUpgradepipto Latest VersionPipversion 22 and up is needed for editable install.python-mpipinstall--upgradepipInstall Dependenciescdcore
python-mpipinstall-e.Run CLI in Development Modepython./cli/src/datayoga/__main__.pyRunning testsCorecd core
pip install .
pip install .[test]
PYTHONPATH=src python -m pytest -s --log-cli-level=DEBUG
Package: datazets
datazetsdatazetsis Python packageStar this repo if you like it! ⭐️ContentsInstallationContributeCitationMaintainersLicenseInstallationInstall datazets from PyPI (recommended). datazets is compatible with Python 3.6+ and runs on Linux, MacOS X and Windows.A new environment can be created as following:condacreate-nenv_datazetspython=3.8
condaactivateenv_datazetspipinstalldatazets# normal installpipinstall--upgradedatazets# or update if neededAlternatively, you can install from the GitHub source:# Directly install from github sourcepipinstall-egit://github.com/erdogant/datazets.git@0.1.0#egg=master
pipinstallgit+https://github.com/erdogant/datazets#egg=master
pipinstallgit+https://github.com/erdogant/datazets# By cloninggitclonehttps://github.com/erdogant/datazets.gitcddatazets
pipinstall-U.Import datazets# Import libraryimportdatazetsasdz# Import data setdf=dz.get('titanic')Example:# Import from urlurl='https://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.data'df=dz.get(url=url,sep=',')CitationPlease cite in your publications if this is useful for your research (see citation).MaintainersErdogan Taskesen, github:erdogantContributeAll kinds of contributions are welcome!If you wish to buy me aCoffeefor this work, it is very appreciated :)LicenceSeeLICENSEfor details.
Package: datazimmer
datazimmerTo create a new projectmake sure thatpythonpoints topython>=3.8and you havepipandgitthenpip install datazimmerrundz init project-namepullsproject-templateadd a remoteboth to git and dvc (can rundz build-metato see available dvc remotes)git remote can be given withdz initcreate, register and document steps in a pipeline you will run in differentenvironmentsbuild metadata to exportable and serialized format withdz build-metaif you defined importable data from other artifacts in the config, you can import them withload-external-dataensure that you import envs that are served from sources you have access tobuild and run pipeline steps by runningdz runvalidate that the data matches thedatascriptdescription withdz validateSchedulinga project as a whole has a cron expression inzimmer.yamlto determine the schedule of rerunsadditionally, aswan projects within the dz project can have different cron expressions for scheduling new runs of the aswan projectsTest projectsTODO: document dogshow and everything else much better hereLookaheadoverlapping names conventionresolve naming confusion with colassigner, colaccessor and table feature / composite type / index base classesabstract composite type + subclass of entity classimport ACT, inherit from it and specifyimporting composite type is impossible now if it contains foreign key :(add option to infer data type of assigned featurecan be problematic b/c pandas int/float/nan issuecreate similar sets of features in a dry wayoverlapping in entitiesdetect / signal the same type of entityexports: postgres, postgis , supersetW3C compliancy plantest suite for compliance:https://w3c.github.io/csvw/publishing-snapshots/PR-earl/earl.htmlhttps://github.com/w3c/csvwhttps://www.w3.org/TR/2015/REC-tabular-data-model-20151217/https://www.w3.org/TR/tabular-metadata/@article{tennison2015model,
  title={Model for tabular data and metadata on the web},
  author={Tennison, Jeni and Kellogg, Gregg and Herman, Ivan},
  year={2015}
}@article{pollock2015metadata,
  title={Metadata vocabulary for tabular data},
  author={Pollock, Rufus and Tennison, Jeni and Kellogg, Gregg and Herman, Ivan},
  journal={W3C Recommendation},
  volume={17},
  year={2015}
}
Package: datazone-northstar
Datazone Northstar Client Library 👋InstallationYou can install package via pippipinstalldatazone-northstarConfiguration & LoginRun the following command in your terminaldatazoneauthconfigureWhen prompted, enter the following information:Host[app.datazone.co]:<DATAZONE_ADDRESS>
Username:<USERNAME>
Password:<PASSWORD>
Repeatforconfirmation:<PASSWORD_CONFIRM>Then you can run following command to logindatazoneauthloginIt will keep your tokens until refresh token expired. (Default 30min). Then you can re-run and login again.Create first repositorydatazonerepositorycreatehello-worldcdhello-worldIt creates a new folder in your current directory. Thıs folder has two file initially.config.yamlandfirst_pipeline.py.
After you modify your script you can deploy project via following command. (It takes about one minute).datazonerepositorydeployAfter the deployment is complete, you can check your repository status via following command.datazonerepositorysummaryIf repository has deployed, you can run your first execution with following command.datazoneexecutionrun<pipeline_id>CommandsAlso you can check all command with help sub command.datazone auth --helpordatazone repository summary --help# Authdatazoneauthconfigure
datazoneauthlogin# Repositorydatazonerepositorylist
datazonerepositorycreate
datazonerepositorydeploy
datazonerepositorysummary<file_name>
datazonerepositoryclone<repository_id>
datazonerepositorypull# Datasetdatazonedatasetlist
datazonedatasetshow<dataset_id>[<branch_name>][<size>][<transaction_id>]datazonedatasettransactions<dataset_id># Viewdatazonedatasetviewcreate
datazonedatasetviewlist<dataset_id>
datazonedatasetviewdelete<view_id># Sourcedatazonesourcecreate
datazonesourcelist
datazonesourceupdate<source_id>
datazonesourcedelete<source_id># Extractdatazoneextractcreate
datazoneextractlist
datazoneextractupdate<extract_id>
datazoneextractdelete<extract_id>
datazoneextractexecute<extract_id># Scheduledatazoneschedulecreate
datazoneschedulelist
datazonescheduledelete<source_id># Executiondatazoneexecutionrun[--extract-id][--pipeline-id][<execution_type>][<transform_selection>]datazoneexecutionlist[--extract-id][--pipeline-id]datazoneexecutionlog<execution_id># Pipelinedatazonepipelinecreate# Commondatazoneversion
datazoneinfo
Package: datazoo
Data zooThis repository provides unified access to multiple datasets.UsageFirst of all, you have to import data_provider from datazoo package:from datazoo import data_providerThen, you can select dataset from the list and get iterable:# Dataset object
fashionmnist = data_provider(
	dataset='fashionmnist',	data_dir='data/fashionmnist/', split='test',
	download=True, columns=['index', 'image', 'class']
)

print('Dataset length:', len(fashionmnist))

# Iterate over samples
for i in fashionmnist:
    print(i)ClassificationSingle-label datasetsDatasetName in data providerNumber of classesNumber of samplesSourceAuto downloadingMNISTmnist1060 000 / 10 000torchvisionYesFashion MNISTfashionmnist1060 000 / 10 000torchvisionYesCIFAR-10cifar101050 000 / 10 000torchvisionYesCIFAR-100cifar10010050 000 / 10 000torchvisionYesIndoor Scene Recognitionindoor_scene_recon6715620--YesThe Street View House Numbers (SVHN)svhn_cropped1073257 digits for training, 26032 digits for testing, and 531131 additional--YesLinnaeus5linnaeus55 classes: berry, bird, dog, flower, other (negative set)1200 training images, 400 test images per class--YesCOIL-100coil100100 (100 objects)7200 images--YesLicenseThis software is covered by MIT License.
Package: dat-build
Welcome to DAT !Accelerate the development of your C/C++ project.Hit the star for support. New features will be added.Generate your package:Use it:DAT is setting up the following for you:a standard repository structuretheConanpackage flowuse any build tool you wantthe starter package generated with dat new contains a basic Bazel build with GTest integratedrelease the package inArtifactorydocumentation withMkdocsthrough extensions, testing and running your package (check the extensions folder)Create your own templates and generate your own packages using DAT.Although a work in progress, DAT provides already the basics to start using it and benefit from increased development efficiency.Write your own templates and use them:dat new -n <package_name> -t <your_template_file> -d <destination>Checkhttps://github.com/demorgan104/dat-templates.gitfor samples.Getting startedInstall DAT in your python environmentClone DAT:
    git clone https://github.com/demorgan104/dat
    
Make sure your Python version is 3.9.0

Create a python virtual environment:
    python -m venv ./venv

Activate the virtual environment
    Windows:
      venv/Scripts/activate
    Macos:
      source venv/bin/activate

Install DAT:
    python setup.py install

Test the setup:
    dat -hUsing DATTypical usage:Generate a package:dat new -n dat_helloChange the directory to your package:cd dat_helloBuild your package:This step requires the installation of Bazel:https://docs.bazel.build/versions/4.1.0/install.htmldat buildTest the package:dat testThis test is for both testing that the package can be used by other packages and executing the unit tests.Check the documentation:dat documentRun the application (this works for desktop apps only of course):dat runReleasing to artifactory:For this you need to setup an artifactory server. You can use the following repo for the demo:https://github.com/demorgan104/dat-infrastructureClone this repo and navigate to artifactory directory. There you can run:docker-compose upDAT is for the moment configured to work with that and there is no portable configuration available. Feel free to open a pull request for that or modify the source code :)Then you can use:dat releaseDAT CommandsNAMEDESCRIPTIONdat newCreate a new ready to use packagedat buildBuild a packagedat releaseRelease a packagedat testTest your packagedat runRun your executabledat documentCheck your documentationDAT RepositoriesNAMEDESCRIPTIONLINKdatDat main repositoryhttps://github.com/demorgan104/datdat-infrastructureDat package infrastructure repohttps://github.com/demorgan104/dat-infrastructuredat-templatesTemplates for DAThttps://github.com/demorgan104/dat-templates.git
Package: datcat
DatCatPlease note this is an alpha version and still in active development. Naturally all feedback is welcome.Datcat is a simple and lightweight data catalogue api for big query.
Datcat loads your .json schema files to memory for use with either your own synchronisation service orcatasyn- it's sibling application.
Look into the example_catalogue directory orhereto find out how to define your bigquery schemas.
Here's a quick snippet if you are as lazy as I am:[{"description":"Unique Identifier","mode":"REQUIRED","name":"MY_UNIQUE_ID","type":"INT64"},{"description":"Favourite Colour","mode":"REQUIRED","name":"MY_FAVOURITE_COLOUR","type":"STRING"}]Currently, datcat supports partition generation and pii identification via tagging the relevant column's description with{"partition": true}and/or{"pii": true}.[{"description":"{\"pii\": true}","mode":"REQUIRED","name":"col_4","type":"STRING"},{"description":"{\"partition\": true}","mode":"REQUIRED","name":"date","type":"DATE"}]In addition to serving schema definitions via its api, it  creates a basic mapping between a schema - topic - subscriber that is later used to create the relevant infrastructure[1]from the schema definition.
After the schemas are defined runpython -m datcat.service_layer.mappingsto create those mappings. The naming conventions are basic, with each topic containing all versions of an event and each topic having only one subscriber for the purposes of data lake ingestion alone.//schema_topic_subscription.json{"login_v1":{"schema_class_name":"login","topic_name":"login_topic","subscription_name":"login_subscription"}}CI/CD is your gig but if you fancy seeing datcat in action in your local docker run./docker-docker-build.shand go to:http://0.0.0.0:50000Footnote 1IAM and general permissions are out of scope in this project. It's up to you to ensure your service account has all the necessary roles/permissions to create bigquery tables and topics/subscribers. Checkthisfor a reminder.
Package: datch
DatChThe DatCh data checker checks data files for any inconsistancies.DatCh currently checks for:nan valueswhite space in stringsCAPS errorsdata typesUsage:# check a pandas DataFrameimportpandasaspdimportdatchdf=pd.DataFrame({'id':[0,1,'x',3,4,5,'y',7,None,9],'name':['Rose','Lily','Tulip','Orchid','Carnation','Freesia',3.14159265,'Gladiolus','Anemone','Daffodil'],})datch.errors.nan(df['id'])>>DataError(error_type='nan error',errors=1,values=10,error_ratio=0.10)datch.errors.dtype(df['id'])>>DataError(error_type='data type error',errors=3,values=10,error_ratio=0.30)datch.errors.dtype(df['name'])>>DataError(error_type='data type error',errors=1,values=10,error_ratio=0.10)datch.check('data/data.xlsx')>>DatChisdone!Theoutputfileissavedas/data/data_datched.xlsxWish list as optional checks:zero values (or maybe not)outliers (np.abs(float value) > mean + 3 * np.std)