Saravanan666/pypi-org huggingface dataset:
Package: data-printer
NAMEdata_printer- data printer and dumperVERSION0.0.8SYNOPSISfromdata_printerimportp,npimportsysfromcoloredimportfore,back,styleclassA:def__init__(self,**kw):fork,vinkw.items():setattr(self,k,v)data=A(abc="acc",a=A(x=dict(p=10,r=[20,(2,0.01)])),s='Строка\n',b=b'binary\n',r=r'\n')# add ref to themselwes:data.a.x['r'].append(data)# print colored structure to sys.stdoutp(data)# print colored structure to file streamp(data,file=sys.stderr)# print uncolored structure to file streamp(data,file=sys.stderr,color=False)# serialize structure to strings=np(data)# serialize structure to string without indexes: [6] -> [0] 6s=np(data,indexes=False)# serialize structure to colored string (colors as escape sequences)s=np(data,color=True)# default color schemep(data,color=dict(bool=fore.LIGHT_BLUE,none=fore.LIGHT_BLUE,int=fore.LIGHT_YELLOW,float=fore.LIGHT_YELLOW,str=fore.LIGHT_GREEN,bytes=fore.LIGHT_MAGENTA,object=fore.LIGHT_RED,any=fore.LIGHT_GRAY,key=fore.LIGHT_CYAN,ref=fore.RED,punct=fore.WHITE,))# replace two colorss=np(data,color=dict(bool=fore.LIGHT_RED,none=fore.LIGHT_YELLOW,))# print in perl-style (python - default) and without "[0] 7" in list "[7]".p(data,sep="perl",indexes=False)# print in node-stylep(data,sep="node")# self-stylep(data,sep=dict(kword=True,oword=True,kw=" => ",kv=" => ",object_open="bless {",object_close="}, %n",dict_open="{",dict_close="}",list_open="[",list_close="]",tuple_open="[",tuple_close="]",str='"%s"',bytes='do { use bytes; "%s" }',scalar='%n("%s")',# for datetime.datetime and othersnone="undef",true="1",false="0"))# self-style with two and lambda on bytesp(data,sep=dict(str='"%s"',bytes=lambdas:"".join(['Buffer.from(", ',", ".join(["0x%X"%chforchins]),')']),))# filters use for specific objectsimportdatetimep([datetime.datetime.now()],filters={datetime.datetime:lambdaobj,ddp:ddp.echo("⎈ ",ddp.color.key).echo(repr(obj),ddp.color.any),})# print without newline on the endp(data,end="")DESCRIPTIONData recursive printer. Serialize any python3 data to string or print in console or file.Is colorised output.Data printer check many references to one structure.INSTALL$pipinstalldata-printerREQUIREMENTScoloredAUTHORKosmina O. Yaroslavdarviarush@mail.ruLICENSEMIT LicenseCopyright (c) 2020 Kosmina O. Yaroslav
Package: data-process
Data ProcessMake processing 2d data more convenientInstallpip install data-processModule descriptioncollect_csv_data: Collect data in a directory contains CSV filesdate: Generate current date and time; get the range of and start, end day of a period date; convert bettween string anddatetime.datetimetypedate_range: Get the range of date from the start, end, range gap you provideddecimal_helper: Convert float to Decimal typedict_helper: Replace the keys of a dict; infinitedefaultdictdata structure; get data by a series of keysfilter_date: Filter a series date string by the given start, end dategreen_dict: Make thedictcompatibal with thejson.dumps, especially fordate(time)typegroup_by: Group a series ofdictby a list of functions; degroup a deep grouped dict by the depthsio_csv: Read, write csv filesio_json: Read, write json filesio_files: List files in a glob pattern directoriesio_lines: Read as lines from a file except the lines are blank or start with '#'iter_dict: Convert dict for output, handled by the type of values separatelyjoin: Left, right, inner, outer join a series of dict as like in SQLlist_helper: Flat a nested listpandas_helper: Functions wrapper for usingpandasconveniently. Convertpandas.DataFrameto a list of dict; aggregate DataFrame by a field.
Package: data-processing-library
DataProcessingLibraryPython port forSkivsoft.ProcessorSimple data processing library.The key features are:Easy idea for running tasks step-by-stepSynchronous and asynchronous way for executing stepsSupport SOLIDExample of usagefromuuidimportuuid4fromdata_processing_library.processorimportAbstractProcessor,Contextfromdata_processing_library.groupimportProcessorGroupclassHelloContext(Context):def__init__(self):self.name=Nonedefset_name(self,name:str)->None:self.name=namedefget_name(self)->str:returnself.nameclassInputName(AbstractProcessor):defexecute(self,context:HelloContext):context.set_name(uuid4().hex)classOutputGreeting(AbstractProcessor):defexecute(self,context:HelloContext):print(f'Hello,{context.get_name()}!')defrun_processor():hello_context=HelloContext()steps=[InputName(),OutputGreeting(),]processor=ProcessorGroup(steps)processor.execute(hello_context)if__name__=='__main__':run_processor()
Package: data-processing-package
data-processing-packageDescription.
The package package_name is used to:
preprocessing:
- loadDataset
- fillMissingData
- computeNormalization
- splitTrainTestSets
models:
- computeLinearRegression
- RandomForestRegressorInstallationUse the package managerpipto install package_namepipinstalldata-processingUsageUsed to Load datasets Split in X and y values, later split in Train and test sets and bring the data trainned and predicted.AuthorThomasLicenseMIT
Package: data-product-definition-tooling
Data Product Definitions toolingTools for managing Data Product definitionsInstallationpoetryinstallUsagepoetryrunconverter--help# run testspoetryruninvoketest# release a new version (after bumping it in pyproject.toml)poetryruninvokereleasePre-commit hooksrepos:-repo:https://github.com/ioxio-dataspace/data-product-definition-toolingrev:main# You probably want to lock this to a specific taghooks:-id:data-product-definition-converterfiles:"src/.*py$"args:["src","dest"]
Package: data-product-processor
data product processorThe data product processor is a library for dynamically creating and executing Apache Spark Jobs based on a declarative description of a data product.The declaration is based on YAML and covers input and output data stores as well as data structures. It can be augmented with custom, PySpark-based transformation logic.InstallationPrerequisitesPython 3.xApache Spark 3.xInstall with pippip install data-product-processorGetting startedDeclare a basic data productPlease seeData product specificationfor an overview on the files required to declare a data product.Process the data productFrom folder in which the previously created file are stored, run the data-product-processor as follows:data-product-processor \
  --default_data_lake_bucket some-datalake-bucket \
  --aws_profile some-profile \
  --aws_region eu-central-1 \
  --localThis command will run Apache Spark locally (due to the --local switch) and store the output on an S3 bucket (authenticated with the AWS profile used in the parameter).If you want to run the library from a different folder than the data product decleration, reference the latter through the additional argument--product_path.data-product-processor \
  --product_path ../path-to-some-data-product \
  --default_data_lake_bucket some-datalake-bucket \
  --aws_profile some-profile \
  --aws_region eu-central-1 \
  --localCLI Argumentsdata-product-processor --help

  --JOB_ID - the unique id of this Glue/EMR job
  --JOB_RUN_ID - the unique id of this Glue job run
  --JOB_NAME - the name of this Glue job
  --job-bookmark-option - job-bookmark-disable if you don't want bookmarking
  --TempDir - tempoarary results directory
  --product_path - the data product definition folder
  --aws_profile - the AWS profile to be used for connection
  --aws_region - the AWS region to be used
  --local - local development
  --jars - extra jars to be added to the Spark context
  --additional-python-modules - this parameter is injected by Glue, currently it is not in use
  --default_data_lake_bucket - a default bucket location (with s3a:// prefix)ReferencesData product specificationAccess managementTutorialsHow to write and test custom transformation logic?How to reference custom Spark dependencies?How to set up local development?
Package: data-profiler
The data_profiler module extends the standard CPython profiler by
recording the functions’ signatures. For NumPy array types this includes
thedtypeattribute and the array’s shape.It also adds functionality to visualise the augmented profile table in
snakeviz.This module is a direct port ofAccelerate.profileravailable inAnaconda
Accelerate.Documentation is locatedhereInstallingThe easiest way to install data_profiler and get updates is by using
theAnaconda Distribution#> conda install data_profilerTo compile, test and run from source, it is recommended to create a
conda environment containing the following:numpynumba >=0.26.0snakevizjupyterpytestfor instructions on how to do this see thecondadocumentation, specifically the
section onmanaging
environments.Once a suitable environment is activated, installation achieved simply
by running:#> python setup.py installand the installation can be tested with:#> pytestDocumentationDocumentation is locatedhere.Building DocumentationIt is also possible to build a local copy of the documentation from
source. This requires GNU Make and sphinx (available via conda).Documentation is stored in thedocfolder, and should be built with:#> make SPHINXOPTS=-Wn clean htmlThis ensures that the documentation renders without errors. If errors
occur, they can all be seen at once by building with:#> make SPHINXOPTS=-n clean htmlHowever, these errors should all be fixed so that building with-Wnis possible prior to merging any documentation changes or updates.Continuous IntegrationContinuous integration is provided byTravis
CI, the current build state is availablehere.
Package: data-profile-viewer
DataProfileViewerData profile viewer is compatible with Jupyter Notebooks. Supports the metadata format generated bydatamart-profilerlibrary.Install via pippip install data-profile-viewer
pip install datamart-profilerDemoIn Jupyter Notebook:import DataProfileViewer
data = DataProfileViewer.get_lifeexpectancy_data()
DataProfileViewer.plot_data_summary(data)Data Profile Explorationimport DataProfileViewer
import datamart_profilerIn a jupyter notebook, load the datadata_path = 'lifeexpectancydata.csv'
metadata = datamart_profiler.process_dataset(data_path, include_sample=True, plots=True)and then plot it using:DataProfileViewer.plot_data_summary(metadata)Refine Data Profiler ResultsYou might want to correct/refine the type information, or provide additional annotations for the columns. To do so, use the code:DataProfileViewer.plot_edit_profiler(metadata)To retrieve the updated metadata, use the code:updatedMetadata = DataProfileViewer.get_exported_metadata(data_path)
Package: datapro-learning
DataPro algorithmThe task is learning the common structure of data. The DataPro algorithm is the algorithm that finds statistically significant patterns in a set of token sequences.This repository is Implemented from the paper "Learning the Common Structure of Data" by Kristina Lerman and Steven Minton.installationpip install datapro-learningGet startedHow to use model with this lib:fromdataproimportDataProimportpandasaspd# Read data filedf=pd.read_excel("street_road.xlsx")df=df.dropna()# Choose a column.data_sample=df["หน่วยรับผิดชอบ"]# Create datapro object.s=DataPro(alpha=0.05,k_percentage=10)# Train with datas.fit(data_sample)# show resultprint(s.evaluate_score())Referencehttps://www.aaai.org/Papers/AAAI/2000/AAAI00-093.pdf
Package: dataproxy
Data Proxy: a google app-engine application for proxying data to json (jsonp) format.Author: James Gardner <http://jimmyg.org>Author: Stefan Urbanek <stefan.urbanek@gmail.com>Transformation modules======================For each resource type there should be a module in transform/<type>_transform.pyEach module should implement:* ``transformer(flow, url, query)``, should return a Transformer subclass* Transformer subclass with __init__(flow, url, query)Existing modules:* transform/csv_transform - CSV files* transform/xls_transform - Excel XLS filesRandom notes============Mount pointMaximum file sizehttp://someproxy.example.org/mount_point?url=url_encoded&sheet=1&range=A1:K3&doc=no&indent=4&format=jsonpResponse format:headerurl = http://...file.xlsoption = 'row=5&row=7&row_range=10:100000:5000',responsesheet = 'Sheet 1',data = [[...],[...],[...],]* Downloading the entire spreadsheet* Downloading a single sheet (add ``sheet=1`` to the URL)* Downloading a range in a single sheet (add ``range=A1:K3`` to the URL) [a bit nasty for CSV files but will do I think]* Choosing a limited set of rows within the sheet (add ``row=5&row=7&row_range=10:100000:5000`` - rowrange format would be give me a row between 10 and 100000 every 5000 rows)Hurdles-------* Some data sets are not in text-based formats => Don't handle them at this stage* Excel spreadhseets have formatting and different types => Ignore it, turn everything into a string for now* Some data sets are huge => don't proxy more than 100K of data - up to the user to filter it down if needed* We don't want to re-download data sets => Need a way to cache data -> storage API* Some applications might be wildly popular and put strain on the system -> perhaps API keys and rate limiting are needed so that individual apps/feeds can be disabled. How can we have read API keys on data.gov.uk?
Package: datapungi_imf
install code: pip install datapungi_imfdatapungi_imfdatapungi_imf is a python package that provides a simplified way to extract data from the API of IMF (IMF).  Overall it:can read a saved API key (in json/yaml files or environment variables (default)) to avoid having a copy of it on a script.can automatically test:the connectivity to all BEA datasets,the quality of the cleaned up data, andif the provided requests code snippet returns the correct result.SectionsShort sample runsShort sample runs of all driversDescription of a full returnSetting up datapungi_imfTesting the packageSample runsFirst,set the package up(get an API key from BEA, save it somewhere and let datapungi_imf know its location).  After setting datapungi_imf up, you can run the following:'''Short datapungi_imf sample run'''importdatapungi_imfasdpidata=dpi.data()#or data = dpi.data("API Key"), see setting up sectiondata.list()data.params('IFS')data.data('IFS/A.GB.PMP_IX')data.meta('IFS/A.GB.PMP_IX')
Package: datapunt-accounts
Install$ pip install datapunt-userIn order to use this library you need to have access to a Postgress database.Usageimport dpuser

users = dpuser.AuthzMap(**psycopgconf)

users.add('myuser@example.com', 'secretpassword')
users.set('myuser@example.com', 'newsecretpassword')
users.verify_password('myuser@example.com', 'secretpassword')
users.remove('myuser@example.com')ContributeActivate your virtualenv, install the egg ineditablemode, and start coding:$ source env/bin/activate
$ pip install -e .Testing:make test
Package: datapunt-authorization
Interface to authorization logic in the backend of Amsterdam’s (Netherlands)
open data distribution platform, [Datapunt](http://data.amsterdam.nl).In Datapunt authorization levels associated with users are stored in a central
database. When a user accesses a resource she needs to provide a
cryptographically secure token. That token contains the user’s authorization
level. The resource in turn uses that information to decide what information
the user can access.Install$ pip install datapunt-authorizationIn order to use this library you need to have access to a Postgress database.Usageimport authorization
import authorization_levels  # pip install datapunt-authorization-levels

authzmap = authorization.AuthzMap(**psycopgconf)

if authzmap['myuser'] == authorization_levels.LEVEL_EMPLOYEE:
        ...  # do some eployee-e thingsContributeActivate your virtualenv, install the egg ineditablemode, and start coding:$ source env/bin/activate
$ pip install -e .Testing:make testDocumentationThe docs can be found at [datapunt-authorization.readthedocs.io](https://datapunt-authorization.readthedocs.io).
Package: datapunt-data-ingress
Data IngressThis is a simple ingress app which stores all incoming messages in a queue categorized
into collections.Data is always posted in the form of messages to a specific collection,
and can be processed from the queue by consumers.Usage:1. Add the appEnable in installed apps:INSTALLED_APPS=[...,'ingress']2. Create and configure a consumerImplement a consumer to handle the data. Note: the ingress can be deployed without a
consumer when you (probably temporarily) want to ingest data, but not consume it.classMyConsumer(IngressConsumer):collection_name='foobar'defconsume_message(self):...# by default consume_batch() loops and calls consume_message(), but it can be# overridden to have more control, for instance to bulk insert into the db.defconsume_batch(self):...Configure the consumer in settings.py.  Each consumer must implement the BaseConsumer
and implement the proper methods (see above). On consumption consumers are passed
all messages in the collection that corresponds toConsumer.collection_name.# A list of classpaths to implementations of ingress.consumer.IngressConsumer# to handle the data in the queue.INGRESS_CONSUMER_CLASSES=['app.module.MyConsumer']3. Authentication and AuthorizationConfigure at least the authentication classes or permission classes or disable authentication completely:# A list of authentication classes used in the ingress view.# See https://www.django-rest-framework.org/api-guide/authentication/INGRESS_AUTHENTICATION_CLASSES=[]# A list of permission classes used in the ingress view.# See https://www.django-rest-framework.org/api-guide/permissions/INGRESS_PERMISSION_CLASSES=[]# If the queue is used without the endpoint, authentication checks can be# disabled by setting this flag to True. Defaults to False.INGRESS_DISABLE_ALL_AUTH_PERMISSION_CHECKS=True4. Accept data posted to non-existing collections?By default data posted to non-existing collections is not stored, and an error is
returned for such requests.If data posted to non-existing collections should be stored, modify the setting:# Whether or not to accept data posted to a non-existing collection.INGRESS_ACCEPT_NEW_COLLECTIONS=False5. Set encoding when other than UTF-8By default the ingress decodes all incoming data using UTF-8. If the data needs to be
decoded differently before being stored in the database, configure the setting:# Encoding that the data will be in when posted to the ingressINGRESS_ENCODING="utf-16"
Package: datapunt-objectstore
Objecstore Libary$pipinstalldatapunt-objectstoreContains common used objectstore code for our API’s.During import / ETL tasks we offten use data uploaded to the objecstore
from data sources.ContributingFound a bug or want to work on the code? You can branch therepository on
GitHubor file an issue at itsissue tracker.1. Install the dev dependencies in your virtual environment$pipinstall--upgradesetuptools$pythonsetup.pyinstalldevelopThesetuptoolspakage must be upgraded (as shown above) because the version
that is pre-packaged with Python 3.6 causes problems when running the tests.When you get a PYTHONPATH error, use the install-dir argument:$pythonsetup.pyinstalldevelop--install-dirvenv/lib/python3.6/site-packages2. Run the testsThe test suite and test coverage are run as follows:$pythonsetup.pytestThe Python style checker Flake8 can be run as follows:$pythonsetup.pyflake83. Example configuration and usagepython-mobjecstore.databasedumps/backups/postgres.dumpobjectstore_dir--upload-dbuploads given dump with a date and environment informtion to objectstorepython-mobjecstore.databasedumpsdownloaddirobjectstore_dir--download-dbDowload latest dump with the namedatabase.environment.latestdate.dumpfrom given environmnet
with from location directory in objectstore.
- WILL DELETE OLD DUMPS more then 20 days old.
Package: datapunt-pg-partitioning
# pg_partitionUtility to partition new/existing PostgreSql database tables. Data can be migrated. Only date range partition type is supported.Required environment variables:```DATABASE_HOST : hostname/ip of the PostgreSql serverDATABASE_NAME : database to operate onDATABASE_USER : PostgreSql loginDATABASE_PASSWORD : PostgreSql password```#### Install ####```pip install -r requirements.txt```#### Usage ####```usage: pg_partition.py [-h] {create,add} ...positional arguments:{create,add}optional arguments:-h, --help    show this help message and exit```#### Create Usage ####```usage: pg_partition.py create [-h] [--type {day,week,month}] table columnpositional arguments:table                 table to be partitionedcolumn                column to be used for rangeoptional arguments:-h, --help            show this help message and exit--type {day,week,month}```#### Add Usage ####```usage: pg_partition.py add [-h] [--num NUM] [--date DATE]table {day,week,month}positional arguments:table             table to be partitioned{day,week,month}optional arguments:-h, --help        show this help message and exit--num NUM         partitions to create--date DATE       start date (yyyymmdd)```#### Creating partitions ####```# Convert 'table' to a range partitioned table by 'column'# No data is migratedpython pg_partition.py create table column# Convert 'table' to a range partitioned table by 'column'# Data is migrated in week partitionspython pg_partition.py create table column --type week```#### Adding partitions ####```# Adds 10 day partitions starting from 20180101 to 'table'python pg_partition.py add table day --num 10 --date 20180101```
Package: datapusher
DataPusherDataPusher is a standalone web service that automatically downloads any CSV or
XLS (Excel) data files from a CKAN site's resources when they are added to the
CKAN site, parses them to pull out the actual data, then uses the DataStore API
to push the data into the CKAN site's DataStore.This makes the data from the resource files available via CKAN's DataStore API.
In particular, many of CKAN's data preview and visualization plugins will only
work (or will work much better) with files whose contents are in the DataStore.To get it working you have to:Deploy a DataPusher instance to a server (or use an existing DataPusher
instance)Enable and configure thedatastoreplugin on your CKAN site.Enable and configure thedatapusherplugin on your CKAN site.For details see theDataPusher documentation.Note that if you installed CKAN using thepackage installoption then a
DataPusher instance should be automatically installed and configured to work
with your CKAN site.DataPusher is a replacement forDataStorer.
It's built usingCKAN Service ProviderandMessytables.The original author of DataPusher was
Dominik Moritzdominik.moritz@okfn.org. For the current list of contributors
seegithub.com/ckan/datapusher/contributorsDevelopmentTo install DataPusher for development:gitclonehttps://github.com/ckan/datapusher.gitcddatapusher
pipinstall-rrequirements-dev.txtTo run the tests:nosetestsTo build the documentation:pipinstall-rdoc-requirements.txt
pythonsetup.pybuild_sphinxReleasing a New VersionTo release a new version of DataPusher:Increment the version number indatapusher/init.pyBuild a source distribution of the new version and publish it toPyPI:pythonsetup.pysdistbdist_wheel
pipinstall--upgradetwine
twineuploaddist/*You may want to test installing and running the new version from PyPI in a
clean virtualenv before continuing to the next step.Commit your setup.py changes to git, tag the release, and push the changes
and the tag to GitHub:gitcommitsetup.py-m"Bump version number"gittag0.0.1
gitpush
gitpushorigin0.0.1(Replace both instances of 0.0.1 with the number of the version you're
releasing.)