Saravanan666/pypi-org huggingface dataset:
Package: datapackage-pipelines-aws
# AWS Extensions for datapackage-pipelines[![Build Status](https://travis-ci.org/frictionlessdata/datapackage-pipelines-aws.svg?branch=master)](https://travis-ci.org/frictionlessdata/datapackage-pipelines-aws)## Install```# clone the repo and install it wit pipgit clone https://github.com/frictionlessdata/datapackage-pipelines-aws.gitpip install -e .```## UsageYou can use datapackage-pipelines-aws as a plugin for [dpp](https://github.com/frictionlessdata/datapackage-pipelines#datapackage-pipelines). In pipeline-spec.yaml it will look like this```yaml...- run: aws.dump.to_s3```You will need AWS credentials to be set up. See [the guide to set up the credentials](http://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/credentials.html)### dump.to_s3Saves the DataPackage to AWS S3._Parameters:_* `bucket` - Name of the bucket where DataPackage will be stored (should already be created!)* `acl` - ACL to provide the uploaded files. Default is 'public-read' (see [boto3 docs](http://boto3.readthedocs.io/en/latest/reference/services/s3.html#S3.Client.put_object) for more info).* `path` - Path (key/prefix) to the DataPackage. May contain format string available for `datapackage.json` Eg: `my/example/path/{owner}/{name}/{version}`* `content_type` - content type to use when storing files in S3. Defaults to text/plain (usual S3 default is binary/octet-stream but we prefer text/plain).* `endpoint_url` - api endpoint to allow using S3 compatible services (e.g. 'https://ams3.digitaloceanspaces.com')_Example:_```yamldatahub:title: datahub-to-s3pipeline:-run: load_metadataparameters:url: http://example.com/my-datapackage/datapackage.json-run: load_resourceparameters:url: http://example.com/my-datapackage/datapackage.jsonresource: my-resource-run: aws.dump.to_s3parameters:bucket: my.bucket.namepath: path/{owner}/{name}/{version}-run: aws.dump.to_s3parameters:bucket: my.another.bucketpath: another/path/{version}acl: private```Executing pipeline above will save DataPackage in the following directories on S3:* my.bucket.name/path/my-name/py-package-name/latest/...* my.bucket.name/another/path/latest/...### change_aclChanges ACL of object in given Bucket with given path aka prefix._Parameters:_* `bucket` - Name of the bucket where objects are stored* `acl` - Available options `'private'|'public-read'|'public-read-write'|'authenticated-read'|'aws-exec-read'|'bucket-owner-read'|'bucket-owner-full-control'`* `path` - Path (key/prefix) to the DataPackage._Example:_```yamldatahub:title: datahub-to-s3pipeline:-run: load_metadataparameters:url: http://example.com/my-datapackage/datapackage.json-run: load_resourceparameters:url: http://example.com/my-datapackage/datapackage.jsonresource: my-resource-run: aws.dump.to_s3parameters:bucket: my.bucket.namepath: path/{owner}/{name}/{version}-run: aws.change_aclparameters:bucket: my.another.bucketpath: path/acl: private```Executing pipeline above will save DataPackage on S3 and change ACL to private to all keys prefixed `path`:**Note:** If path parameter is not set this will change ACL for all object in given bucket
Package: datapackage-pipelines-datahub
# DataHub Extensions for datapackage-pipelines## Install```pip install datapackage-pipelines-datahub```## UsageYou will need [DataHub Command Line tool](http://docs.datahub.io/publishers/cli/#installation) to be installed on you machine.You can use datapackage-pipelines-datahub as a plugin for [dpp](https://github.com/frictionlessdata/datapackage-pipelines#datapackage-pipelines). In pipeline-spec.yaml it will look like this```yaml...- run: datahub.dump.to_datahub```*Note: For pushing datasets to testing server set `DATAHUB_ENV=testing`*### dump.to_datahubpublishes DataSet to [DataHub.io](http://next.datahub.io/)Parameters:* `config` - full path to the `config.json` file. Default: `~/.config/datahub/config.json`* Alternatively you can just set `DATAHUB_JSON` environt variable to be equal to the path to the config file* `findability` - Dataset visibility on the DataHub.io. One of `public` (default), `private`, `unlisted`.* other `data push` related options. Eg: `schedule`, `name` etc... see `data push -h` for more.Example:```datahub:title: my-datasetpipeline:-run: load_metadataparameters:url: http://example.com/my-datapackage/datapackage.json-run: load_resourceparameters:url: http://example.com/my-datapackage/datapackage.jsonresource: my-resource-run: datahub.dump.to_datahubparameters:findability: privateschedule: every 2dconfig: config/config.json.datahq```
Package: datapackage-pipelines-elasticsearch
ElasticSearch Extensions for datapackage-pipelinesInstall::# use pip install

pip install datapackage-pipelines-elasticsearch

# OR clone the repo and install it with pip

git clone https://github.com/frictionlessdata/datapackage-pipelines-elasticsearch.git
pip install -e .UsageYou can use datapackage-pipelines-elasticsearch as a plugin for
(dpp)[https://github.com/frictionlessdata/datapackage-pipelines#datapackage-pipelines\ ].
In pipeline-spec.yaml it will look like this.. code:: yaml...
  - run: elasticseach.dump.to_indexdump.to_indexSaves the datapackage to an ElasticSearch instance.

*Parameters*:

-  ``engine`` - Connection string for connecting to the ElasticSearch
   instance (URL syntax)
    Also supports ``env://<environment-variable>``, which indicates that
   the connection string should be fetched from the indicated
   environment variable.
    If not specified, assumes a default of ``env://DPP_ELASTICSEARCH``
    Environment variable should take the form of 'host:port' or a
   fully-qualified url (e.g.
   '`https://user:pass@host:port <https://user:pass@host:port>`__\ ' or
   '`https://host:port <https://host:port>`__\ ' etc.)
-  ``indexes`` - Mapping between resources and indexes. Keys are index
   names, value is a list of objects with the following attributes:
-  ``resource-name`` - name of the resource that should be dumped to the
   table
-  ``doc-type`` - The document type to use when indexing docuemtns
Package: datapackage-pipelines-github
# datapackage-pipelines-githubExtension for `datapackage-pipelines` for pulling stuff from GitHub.- **Pipelines and Processor Code** from a specific branch will be pulled- **Issues** will be presented, using their title and description, as 'unable to start' pipelines.- **Pull Requests** will be shown and executed, based on a predefined policies defining:- Should pipeline specs be pulled from PRs- Should processor code be pulled from PRs- Which processors are allowed to run, when executing PRsYou can define separate policies for local PRs (from the same fork) or remote PRs (from another fork.)note: When pulling code from PRs, we will only bring processors referenced from thepipeline spec and residing in the same directory as the `pipeline-spec.yaml` file.## Source specPlace files named `github.source-spec.yaml` in your pipelines directory.Each one should be of the form:```yaml<pipeline-id-prefix>:repository: <owner/repo>base-path: <where to look for pipelines and code. default is 'pipelines/'>code: <if not present, won't fetch code. set to {} to use the defaults>ref: <which branch/commit to use to get code from. default is 'master>issues: <if not present, won't fetch issues. set to {} to use the defaults>closed: <boolean, should fetch closed issues? default is no>pipeline-id-format: <string, see below>pull-requests: <if not present, won't fetch prs. set to {} to use the defaults>local: <policy for local PRs>specs: <boolean, should pull pipeline specs? default is yes>code: <boolean, should pull code from PRs? default is yes>disallow-processors: <which processors are not allowed to run in PRs? default is ["dump\..*"]>- <regular-expression for processor name>remote: <policy for remote PRs, same as local>specs: <boolean, should pull pipeline specs? default is yes>code: <boolean, should pull code from PRs? default is no>disallow-processors: <which processors are not allowed to run in PRs? default is ["dump\..*"]>- <regular-expression for processor name>pipeline-id-format: <string, see below>````pipeline-id-format` is a Python format string with two placeholders:- `issue-id`: The issue/pr number- `title-slug`: The issue title slugThe default format for issues is "issue/{issue-id:03}_{title-slug}"The default format for pull-requests is "pr/{pr-id:03}_{title-slug}"#### Example:```yamldpp-github:repository: firctionlessdata/datapackage-pipelines-githubcode:ref: masterpull-requests:local:specs: yescode: yesdisallow-processors:- "dump\..*"remote:specs: nopipeline-id-format: "{title-slug}__{issue-id}"issues:closed: nopipeline-id-format: "{title-slug}__{issue-id}"```
Package: datapackage-pipelines-goodtables
Adatapackage-pipelinesprocessor to validate tabular resources usinggoodtables.Install# clone the repo and install it with pip

git clone https://github.com/frictionlessdata/datapackage-pipelines-goodtables.git
pip install -e .UsageAdd the following to the pipeline-spec.yml configuration to validate
each resource in the datapackage. A report is outputted to the logger....-run:goodtables.validateparameters:fail_on_error:True,reports_path:'path/to/datapackage/reports',# where reports will be writtendatapackage_reports_path:'reports',# relative to datapackage.jsonwrite_report:True,goodtables:<key>:<value># options passed to goodtables.validate()fail_on_error: An optional boolean to determine whether the
pipeline should fail on validation error (defaultTrue).reports_path: An optional string to define where Goodtables
reports should be written (default isreports).datapackage_reports_path: An optional string to define the path
to the report, relative to the datapackage.json (see note below).write_report: An optional boolean to determine whether a
goodtables validation report should be written toreports_path(default isTrue).goodtables: An optional object passed togoodtables.validate()to customise its behaviour. See`goodtables.validate()<https://github.com/frictionlessdata/goodtables-py/#validatesource-options>`__
for available options.If reports are written, anddatapackage_reports_pathis defined, areportsproperty will be added to the datapackage, detailing the
path to the report for each resource:..."reports":[{"resource":"my-resource","reportType":"goodtables","path":"path/to/my-resource.json"}]It is recommended thatdatapackage_reports_pathis used to define a
relative path, from the datapackage.json file, that represents where the
report was written.datapackage_reports_pathdoes not define where
the reports will be written, but helps ensure a correct path is defined
in thereportsproperty in datapackage.json. This is useful when the
pipeline concludes with adump_to.pathprocessor.
Package: datapackage-pipelines-spss
Install# clone the repo and install it with pip

git clone https://github.com/frictionlessdata/datapackage-pipelines-spss.git
pip install -e .Usagedatapackage-pipelines-spss usestableschema-spssto read SPSS .sav files and add tableschema descriptors to a
datapackage. You can use datapackage-pipelines-spss as a plugin fordpp.
In the pipeline-spec.yaml it will look like this:...-run:spss.add_spssparameters:source:data/my-file.savThis will add the file defined atsourceas a tabular resource to
the pipeline.sourcecan be either a file path, or a url starting
withhttporhttps.
Package: data_packager
# Data Packager (data_packager) #Provides simple tools to allow for the versioned packaging of arbitrary
assets.For instance, suppose you have a given data set that your apps depend upon,
which change slowly, but have considerable impact on your appllication
behaviors.You want to manage the data set on a versioned basis, such that somebody
can install a specific version of the data into their virtualenv.This gives a simple way for organizing such packages and a simple interface
for accessing assets within them.# License #Copyright (c) 2012 Ethan Rowe <ethan at the-rowes dot com>Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
Package: datapackage-registry
**This project is **DEPRECATED**. You should use<https://github.com/datapackages/datapackage-py> instead. This repository isjust for historical purposes.**datapackage-registry-py=======================A Python library for working with a Data Package Registry.It comes with a cached version of the registry and schemas from<https://github.com/dataprotocols/schemas>.Usage-----```pythonimport datapackage_registry# get the default registry objectsregistry = datapackage_registry.Registry()# see the available profiles in the registryprint(registry.available_profiles)# {#     'base': {#         'id': 'base',#         'schema': 'https://rawgit.com/dataprotocols/schemas/master/data-package.json',#         'specification': 'http://dataprotocols.org/data-packages',#         'title': 'Data Package'#     },#     'tabular': {#         'id': 'tabular',#         'schema': 'https://rawgit.com/dataprotocols/schemas/master/tabular-data-package.json',#         'specification': 'http://dataprotocols.org/tabular-data-package/',#         'title': 'Tabular Data Package'#    }# }# get a profile by its idbase_profile = registry.get('base')```If you’d like to use a custom registry, you can pass its URL or localpath to the `Registry()` constructor, as in:```pythonimport datapackage_registryregistry = datapackage_registry.Registry('http://someplace.com/my-registry.csv')```If you’d like to get a schema that's not in the registry, you can pass its URLor local path to `Registry().get_external()` method, as in:```pythonimport datapackage_registryregistry = datapackage_registry.Registry()try:schema = registry.get_external('http://someplace.com/schema.json')except datapackage_registry.exceptions.DataPackageRegistryException:pass  # handle error```Developer notes---------------These notes are intended to help people that want to contribute to thispackage itself. If you just want to use it, you can safely ignore this.### Updating the local schemas cacheWe cache the schemas from <https://github.com/dataprotocols/schemas>using git-subtree. To update it, use:git subtree pull --prefix datapackage_registry/schemas https://github.com/dataprotocols/schemas.git master --squash
Package: datapackage-to-datasette
datapackage-to-datasetteImport Frictionless DataDatapackages
into SQLite and generateDatasette metadata.Setuppipinstalldatapackage-to-datasetteUsageOn the consoleImport a datapackage from a local filedatapackage-to-datasettemydatabase.db/path/to/datapackage.jsonmetadata.jsonor from a URLdatapackage-to-datasettemydatabase.dbhttps://pkgstore.datahub.io/core/co2-ppm/10/datapackage.jsonmetadata.jsonIf the datasette metadata file already exists, you can pass--write-mode replaceor--write-mode mergeto overwrite
or merge with the existing datasette metadata file.As a libraryfromdatapackage_to_datasetteimportdatapackage_to_datasette,DataImportErrortry:datapackage_to_datasette('mydatabase.db','/path/to/datapackage.json','metadata.json',write_mode='replace')exceptDataImportError:raise
Package: datapackage-validate
**This project is DEPRECATED. You should use<https://github.com/datapackages/datapackage-py> instead. This repository isjust for historical purposes.**# datapackage-validate-pyValidate [Data Package][] datapackage.json files against a jsonschema.[Data Package]: http://data.okfn.org/doc/data-package## Usage```pythonimport datapackage_validatetry:datapackage_validate.validate(datapackage, schema)except datapackage_validate.exceptions.DataPackageValidateException as e:e.errors  # List with validation errors```The `datapackage` can be a json string or python dict.The `schema` can be a json string, python dict, or a schema id corresponding with a schema from the registry of [Data Package Profiles][]. `schema` is optional, and will default to the `base` schema id if not provided.`validate()` returns None. If there were errors during validation, it raises a`datapackage_validate.exceptions.DataPackageValidateException` with a list ofthe validation errors in its `.errors` property.[Data Package Profiles]: https://github.com/dataprotocols/registry
Package: datapacker
Export data from excel to json format text.It requires xlrd.# samplefrom datapacker import excel2class, excel2data# export to c# classese2c = excel2class.Excel2Class(output_path=’your_output_path/’)e2c.make(namespace=’c# namespace’, gameinfoclass=’GameInfo class name’)# export to JSON filee2d = excel2data.Excel2Data(output_path=’your_output_path/’)e2d.password = ‘md5 password’e2d.suffix = ‘.txt’  # json file suffixe2d.make_json(filename=’GameInfo class name’)
Package: data-packer
简介data-packer是一个数据打包工具。将通过一系列操作将数据从传入数据容器导入到传出容器中.
数据打包中的一个字段打包,将由以下步骤构成:从传入数据容器中获取数据遍历字段校验器列表,对该字段进行校验(如过用户传入了).
以检查传入数据的:类型,长度,取值等是否合法遍历字段转换器列表,对该字段值进行转换(如果用户传入了).
以便将传入数据值转换为我们期望的格式/类型等设置转换后的字段值到传出数据容器定义data-packer是字段提取,校验,打包的工具包.将数据打包逻辑中设计到的实体抽象为:字段 定义不同类型的字段,相互组合形成最终的期望的数据组成结构校验器 当需要对原始数据进行校验时使用, 如校验数据类型,取值范围转换器 当原始数据需要进一步加工处理时使用. 比如长度截取,类型转换等数据容器 将原始数据和目的数据抽象为数据容器,便于再数据打包过程中处理组成field包，用来定义常见的字段类型。 如：必填字段(RequiredField)可选字段(DefaultField)缺省值字段(DefaultField)占位字段(PlaceholderField)多选字段(SelectorField)组合字段(CompositedField)checker包: 定义字段校验器类, 用于校验传入容器中的字段值是否合法converter包: 定义字段值转换器类, 用于按需转换传入数据容器中的字段值container包: 定义一些数据容器类,
用于封装原始数据和目的数据,以便于数据打包应用场景微信支付业务中，封装到微信的请求数据，解析微信的返回数据。web业务中，解析并转换web端传入的请求数据。服务提供端：预定义一系列的DataPacker对象，用于验证和转换
前段传入的数据为内部可用的数据容器。服务请求端：按照服务提供方要求的数据格式创建好DataPacker对象后，将请求的原始数据容器快捷方便的转为服务提供方需要的格式。示例参见example目录
Package: datapact
datapact- pytest, but for dataframesdatapactis a Python library for verifying your data.importdatapactdp=datapact.test(df)dp.age.must.be_positive()dp.name.should.not_be_empty()It works with Pandas + Dask DataFrames, and has special support for Jupyter Notebooks.Here's some features:dozens of existing assertions, easy to add your owngreat in-editor documentation via docstrings + typestwo severence levels (.shouldfor warnings,.mustfor failures)failure notifications via E-Mail, MS Teams, Slack or PagerDuty (via Datapact Track)Get Started here:https://datapact.devDatapactTrackDatapact Track is an optional, browser-based data tracking service.It's fully self-hostable via Docker and Postgres, and there's a hosted version available attrack.datapact.dev.Connecting your test suite is one line of code:dp.connect(server="track.datapact.dev",token="..."# get this from the UI)Datapact track gives you:notifications via E-Mail, Slack, MS Teams and PagerDutya central documentation of your datasetshistory of data expectations + realitydata quality trackingTry out Datapact Track attrack.datapact.dev, or follow theself-hosting guideto deploy your own instance.datapactvsGreat ExpectationsBoth datapact and Great Expectations help you improve Data Quality, but with a different approach.Great Expectations has its own JSON-based storage format for expectation suites, and it gives you a custom UI to edit them.
It's way bigger than datapact - in project size, project scope, but also in complexity.datapactis a lot younger, community-run, and more of alibrarythan aframework.
The main differentiator is that it allows you to express your test suites in Python code, right along your other code.
This works in Python Scripts, Jupyter Notebooks, Pipeline Tests - everywhere that Python runs.
And by having your testsin code, you can co-locate them with the rest of your code, and version control + review them just like all of it.If you already know how to use Great Expectations, you should use it.
If you found its learning curve to steep, maybe look atdatapact- it's designed to be easy to get started, and intuitive to use.Contributors ✨Thanks goes to these wonderful people (emoji key):Simon Knott💻📖🤔🚧st-sch🐛This project follows theall-contributorsspecification. Contributions of any kind welcome!
Package: datapad
Datapad: A Fluent API for Exploratory Data AnalysisDatapad is a Python library for processing sequence and stream data using aFluent style API. Data scientists and researchers use it as a lightweight toolset to efficiently explore datasets and to massage data for modeling tasks.It can be viewed as a combination of syntatic sugar for the Pythonitertools moduleand supercharged tooling for working withStructured Sequencedata.Learn more in DocumentationInstallpip install datapadExploratory data analysis with DatapadSee what you can do withdatapadin the examples below.Count all unique items in a sequence:>>>importdatapadasdp>>>data=['a','b','b','c','c','c']>>>seq=dp.Sequence(data)>>>seq.count(distinct=True)\....collect()[('a',1),('b',2),('c',3)]Transform individual fields in a sequence:>>>importdatapadasdp>>>importdatapad.fieldsasF>>>data=[...{'a':1,'b':2},...{'a':4,'b':4},...{'a':5,'b':7}...]>>>seq=dp.Sequence(data)>>>seq.map(F.apply('a',lambdax:x*2))\....map(F.apply('b',lambdax:x*3))\....collect()[{'a':2,'b':6},{'a':8,'b':12},{'a':10,'b':21}]Chain together multiple transforms for the elements of a sequence:>>>importdatapadasdp>>>data=['a','b','b','c','c','c']>>>seq=dp.Sequence(data)>>>seq.distinct()\....map(lambdax:x+'z')\....map(lambdax:(x,len(x)))\....collect()[('az',2),('bz',2),('cz',2)]Check out our documentation below to see what else is possible with Datapad:DocumentationThis project incorporates ideas from:LINQSparkPython ItertoolsPandasDaskTensorflow tf.data.Datasets
Package: datapan
datapanThis is still a test bed. It is not usefuldatapansifts through enormous files in parallelized Rust to only grab the data you want as quickly and memory-efficiently as possilbe.Installation## create/activate venv# sudo apt-get install python3-venv# python3 -m venv datapan_env# source datapan_env/bin/activate# python -m pip install --upgrade pip## install datapanpipinstalldatapanUsageimportdatapansome_dir=""test=datapan.hello_rust(some_dir)print(test)Developer VersionRust (nightly)curlhttps://sh.rustup.rs-sSf|sh# rustup default nightlyrustupupdatenightlyPoetrypipinstallpoetrymakeinstall
maketest