Saravanan666/pypi-org huggingface dataset:
Package: dataexplore_plot
version number: 0.0.1
author: Tan Kok HuaOverviewPlotting module for use in data exploration for ML and KaggleInstallation / UsageTo install use pip:$ pip install dataexplore_plotOr clone the repo:$ git clonehttps://github.com/spidezad/dataexplore_plot.git$ python setup.py installContributingTBDExampleTBD
Package: data-extraction-c.lynch278
DataExtractionOverviewThis program is used to extract data from an implementation of Collibra into a SQL envrionment. In our case, we are extracting from Gore's instance of Collibra and into EXL_MDSDev. The program will extract the following objects:AssetsAttributesAttribute TypesDomainsCommunitiesRelationsRelation TypesResponsibilities
Each of these object types are stored as their own table in SQL.Instruction For UseSetupThe source executable file is kept in -- Artifact URL --. This folder contains amain.exe file and all of its dependencies. You will need add a 'prod_config.yml' file into the root of this folder. This config file can be found within the source code of the project and should be structured like so:API_CONFIG:limit:1000000AUTH:username:<Valid admin username in environment>password:<Valid admin password in environment>auth-header:<Auto-generated basic auth token, generated using postman>ENVIRONMENT:gore:wlgore-<Envrionment Instance (dev,test,prod)>.collibra.comRunningOpen a cmd prompt in the root of the project folder. Typemain.exe and hit enter. The program will start to run and log its progress. During the run, the program will extract all data and overwrite the raw sql tables.SQL TablesThe following sql tables are created/overwritten during the run on this program:collibra_assets_rawcollibra_attributes_rawcollibra_attribute_types_rawcollibra_communities_rawcollibra_domains_rawcollibra_relations_rawcollibra_relation_types_rawcollibra_responsibilities_rawSQL Stored ProceduresThe following stored procedures are run on the raw tables to manipulate the data and add batch id's:collibra.load_collibra_assetscollibra.load_collirba_attributescollibra.load_collibra_attribute_typescollibra.load_collibra_communitiescollibra.load_collibra_domainscollibra.load_collibra_relationscollibra.load_collibra_relation_typescollibra.load_collibra_responsibilitiesAll of these procedures may be run simultaneously with the collibra._load_entire_batch procedure.Migrating from dev/test to prodIn order to change the environment in which the data extracton runs, the prod_config.yml file within the src folder will need to be changed. The username and password will need to be changed to that of a user in the new environment. Additionally the gore environment variable will need to be changed to the prod instance's URL.
Example of a correctly configured prod_config.yml file for the prod environment:API_CONFIG:limit:1000000AUTH:username:<Valid admin username in PROD environment>password:<Valid admin password in PROD environment>auth-header:<Auto-generated basic auth token, generated using postman>ENVIRONMENT:gore:wlgore-prod.collibra.com
Package: data-extractor
CombineXPath,CSS SelectorsandJSONPathfor Web data extracting.QuickstartsInstallationInstall the stable version from PYPI.pipinstall"data-extractor[jsonpath-extractor]"# for extracting JSON datapipinstall"data-extractor[lxml]"# for extracting HTML dataOr install the latest version from Github.pipinstall"data-extractor[jsonpath-extractor] @ git+https://github.com/linw1995/data_extractor.git@master"Extract JSON dataCurrently supports to extract JSON data with below optional dependenciesjsonpath-extractorjsonpath-rwjsonpath-rw-extinstall one dependency of them to extract JSON data.Extract HTML(XML) dataCurrently supports to extract HTML(XML) data with below optional dependencieslxmlfor usingXPathcssselectfor usingCSS-SelectorsUsagefromdata_extractorimportField,Item,JSONExtractorclassCount(Item):followings=Field(JSONExtractor("countFollowings"))fans=Field(JSONExtractor("countFans"))classUser(Item):name_=Field(JSONExtractor("name"),name="name")age=Field(JSONExtractor("age"),default=17)count=Count()assertUser(JSONExtractor("data.users[*]"),is_many=True).extract({"data":{"users":[{"name":"john","age":19,"countFollowings":14,"countFans":212,},{"name":"jack","description":"","countFollowings":54,"countFans":312,},]}})==[{"name":"john","age":19,"count":{"followings":14,"fans":212}},{"name":"jack","age":17,"count":{"followings":54,"fans":312}},]Changelogv0.10.2Buildupgrade jsonpath-extractor to v0.8.0v0.10.1Fixtypo in .utils.Propertyv0.10.0Featuresupports PEP 561 – Distributing and Packaging Type InformationFixremove LICENSE file from dist filesduplicated extracting if class attrs overlap happened #67remove super class sub-extractors error #68Refactorremove duplciated module “data_extractor.abc”remove the lazy build mechanism of extractorsJSON backend invoking mechanismmake all properties of extractors immutableDocumentfix wrong docstring of “data_extractor.utils.Property”ContributingEnvironment SetupClone the source codes from Github.gitclonehttps://github.com/linw1995/data_extractor.gitcddata_extractorSetup the development environment.
Please make sure you install thepdm,pre-commitandnoxCLIs in your environment.makeinitmakePYTHON=3.7init# for specific python versionLintingUsepre-commitfor installing linters to ensure a good code style.makepre-commitRun linters. Some linters run via CLInox, so make sure you install it.makecheck-allTestingRun quick tests.makeRun quick tests with verbose.makevtestRun tests with coverage.
Testing in multiple Python environments is powered by CLInox.makecov
Package: dataf
DataFDataF is a framework to create data oriented project in a fast way.Data and database manipulation using SQLAlchemy.Database migration with alembic.Creating Web application using Flask.Creating CLI.Clean settings using Yaml.Logging.Dependencies:python >= 3.5SQLAlchemyPyYAMLslackclientflaskflasggerdocutilsmakoalembicQuick install:pip install datafCreate project:dataf create_project nameInstall without pip:git clone https://github.com/BenjaminBoumendil/dataf.git

make

pip install dataf/dist/dataf-x.x.x.tar.gz
Package: datafaser
Datafaser collects and generates data, then fills in templates to generate files.Results of each phase are checked against data schemas and content descriptions.RequirementsPython 3.6Setup. setup-developmentTestingnosetests –with-coverage –cover-package=datafaserUse. local/bin/activate
python datafaser –helpFor example, to output the schema of datafaser as yaml:python datafaser utilities/convert-json-to-yaml.yaml < datafaser/data/schema.jsonDocumentationIdeas in doc folder are currently a bit out of date.
Package: datafest-archive
datafest-archiveDataFestArchive is a Python package designed to generate the DataFestArchive website from past versions of DataFestInstallationpipinstalldatafest-archiveUsageDataFestArchive is a Python package designed to generate the DataFestArchive website from past versions of DataFestUsage:$datafest-archive[OPTIONS]COMMAND[ARGS]...Options:--help: Show this message and exit.Commands:project-call: Reads the spreadsheet and imports the data...website: Create pages of projects and people...datafest-archive maindatafest-archive project-callReads the spreadsheet and imports the data into the database (sqlite3).Usage:$datafest-archiveproject-call[OPTIONS]COMMAND[ARGS]...Options:--help: Show this message and exit.Commands:import-datadatafest-archive project-call import-dataUsage:$datafest-archiveproject-callimport-data[OPTIONS]SPREADSHEET_PATHDATABASE_FILEArguments:SPREADSHEET_PATH: [required]DATABASE_FILE: [required]Options:--help: Show this message and exit.datafest-archive websiteCreate pages of projects and people (students and advisors) from the database (sqlite3) using wowchemy-hugo-academic.Usage:$datafest-archivewebsite[OPTIONS]COMMAND[ARGS]...Options:--help: Show this message and exit.Commands:createdatafest-archive website createUsage:$datafest-archivewebsitecreate[OPTIONS]PATHWEBSITE_OUTPUT_DIRECTORYArguments:PATH: [required]WEBSITE_OUTPUT_DIRECTORY: [required]Options:--help: Show this message and exit.datafest-archiveis a command line tool that can be used to generate the DataFestArchive website from past versions of DataFest.Development documentationRefer toREADME-dev.mdfor development documentation.📈 ReleasesYou can see the list of available releases on theGitHub Releasespage.We followSemantic Versionsspecification.We useRelease Drafter. As pull requests are merged, a draft release is kept up-to-date listing the changes, ready to publish when you’re ready. With the categories option, you can categorize pull requests in release notes using labels.List of labels and corresponding titlesLabelTitle in Releasesenhancement,feature🚀 Featuresbug,refactoring,bugfix,fix🔧 Fixes & Refactoringbuild,ci,testing📦 Build System & CI/CDbreaking💥 Breaking Changesdocumentation📝 Documentationdependencies⬆️ Dependencies updatesYou can update it inrelease-drafter.yml.GitHub creates thebug,enhancement, anddocumentationlabels for you. Dependabot creates thedependencieslabel. Create the remaining labels on the Issues tab of your GitHub repository, when you need them.🛡 LicenseThis project is licensed under the terms of theMITlicense. SeeLICENSEfor more details.📃 Citation@misc{datafest-archive,author={ckids-datafirst},title={DataFestArchive is a Python package designed to generate the DataFestArchive website from past versions of DataFest},year={2023},publisher={GitHub},journal={GitHub repository},howpublished={\url{https://github.com/ckids-datafirst/archive}}}CreditsThis project was generated withpython-package-template
Package: data-file-merge
data-file-mergeThis python module/CLI allows you to merge multiple JSON files into a single destination JSON file using a set of rules defined in thedfm config filePlease refer to theproject wikifor complete documentation includingusageandconfiguration file syntax.Hello World ExampleYou can install the CLI for your operating system fromGitHub here.Using the following config file:{"SourceFiles":[{"SourceFileLocation":{"Path":"foo/bar.json"},"SourceFileNode":"$","DestinationFileNode":"$"},{"SourceFileLocation":{"Path":"foo/baz.json"},"SourceFileNode":"$","DestinationFileNode":"$"},],"DestinationFile":{"DestinationFileLocation":{"Path":"hello/world.json"}}}...you are able to merge the content offoo/bar.jsonandfoo/baz.jsonintohello/world.json(this can either be an existing or new file). The$s indicate that we are merging from the root node of bothfoo/*.jsonfiles to the root ofhello/world.json.
This merge can be achieved with the CLI commanddfm merge --root-path <path to 'foo' directory> <path to this config file>.
The specific rules followed by the merging logic can be found inthe wiki.More examples can be found in thedfm example repo.ConsiderationsCurrently only JSON files are supported but YAML files could be supported with ease in future.Doing the reverse operation will soon be supported (splitting a single sourcefile into multiple destination files).You should be aware of:json-path's dollar-notation syntaxpath-lib's path syntaxregex syntaxDevelopmentinstall pre-commit hooks:poetry run pre-commit installcheck and fix linting issues:make formatContributing to the projectYou are welcome to help us rid the world of monolithic data files! Reach out toServerlessSamto either find some tickets to complete or bring up any ideas you have.
Package: data-file-profiler-utils
Data File Profiler UtilsA Python module for simple data file profiling.Usagefrom import data_file_profiler_utils import Manager as ProfileManagerpm = ProfileManager()
pm.profile_file(“/tmp/patient002.vcf”)CreditsThis package was created withCookiecutterand theaudreyr/cookiecutter-pypackageproject template.History0.1.0 (2024-02-10)First release on PyPI.
Package: datafiles
Datafiles: A file-based ORM for Python dataclassesDatafiles is a bidirectional serialization library for Pythondataclassesto synchronize objects to the filesystem using type annotations. It supports a variety of file formats with round-trip preservation of formatting and comments, where possible. Object changes are automatically saved to disk and only include the minimum data needed to restore each object.Some common use cases include:Coercing user-editable files into the proper Python typesStoring program configuration and state in version controlLoading data fixtures for demonstration or testing purposesSynchronizing application state using file sharing servicesPrototyping data models agnostic of persistence backendsWatchmy lightning talkfor a demo of this in action!OverviewTake an existing dataclass such asthis examplefrom the documentation:fromdataclassesimportdataclass@dataclassclassInventoryItem:"""Class for keeping track of an item in inventory."""name:strunit_price:floatquantity_on_hand:int=0deftotal_cost(self)->float:returnself.unit_price*self.quantity_on_handand decorate it with a directory pattern to synchronize instances:fromdatafilesimportdatafile@datafile("inventory/items/{self.name}.yml")classInventoryItem:...Then, work with instances of the class as normal:>>>item=InventoryItem("widget",3)# inventory/items/widget.ymlunit_price:3.0Changes to the object are automatically saved to the filesystem:>>>item.quantity_on_hand+=100# inventory/items/widget.ymlunit_price:3.0quantity_on_hand:100Changes to the filesystem are automatically reflected in the object:# inventory/items/widget.ymlunit_price:2.5# <= manually changed from "3.0"quantity_on_hand:100>>>item.unit_price2.5Objects can also be restored from the filesystem:>>>fromdatafilesimportMissing>>>item=InventoryItem("widget",Missing)>>>item.unit_price2.5>>>item.quantity_on_hand100InstallationInstall this library directly into an activated virtual environment:$ pip install datafilesor add it to yourPoetryproject:$ poetry add datafilesDocumentationTo see additional synchronization and formatting options, please consult thefull documentation.
Package: datafirst
datafirstAwesomedatafirstis a Python cli/package created withhttps://github.com/TezRomacH/python-package-templateInstallationpipinstalldatafirst🛡 LicenseThis project is licensed under the terms of theMITlicense. SeeLICENSEfor more details.📃 Citation@misc{datafirst,author={ckids-datafirst},title={Awesome `datafirst` is a Python cli/package created with https://github.com/TezRomacH/python-package-template},year={2023},publisher={GitHub},journal={GitHub repository},howpublished={\url{https://github.com/ckids_datafirst/datafirst}}}CreditsThis project was generated withpython-package-template
Package: datafit
DataFit: Automated Data Preprocessing in PythonNote: This package is actively under development and is open source.OverviewDataFit is a powerful Python package developed by Syed Syab and Hamza Rustam for automating data preprocessing tasks. Initiated as part of our Final Year Project at the University of Swat, this tool streamlines the data preprocessing pipeline, making it user-friendly for machine learning engineers and data scientists.Project Initialization Date:01/OCT/2023Expected Project Finalization Date:01/Dec/2023 (Initial Release) (Still under development)Team MembersProfessor Naeem Ullah (Supervisor)FacebookEmail:naeem@uswat.edu.pkSyed Syab (Student)GitHubLinkedInEmail:syab.se@hotmail.comHamza Rustam (Student)GitHubLinkedInEmail:hs4647213@gmail.comPackage FunctionalityThe DataFit package is designed with a user-friendly interface, ensuring accessibility for all users. Its current functionality includes:Displaying information about the datasetHandling null valuesDeleting multiple columnsHandling categorical valuesNormalizationStandardizationExtracting numeric valuesTokenizationUsageTo use the package, install it using:pipinstalldatafitOnce installed, import it like Pandas and start using it:importdatafit.datafitasdf# Display information about the datadf.information(data)To handle categorical values:importdatafit.datafitasdf# Specify columns to handle or use None for all columnsdf.handleCategoricalValues(data,["column1","column2"])To extract numerical values from columns:importdatafit.datafitasdf# Specify columns for extractiondf.extractValues(data,["column1","column2"])New Updates inversion=0.2023.2.13:Description updatedNote:This package is actively under development. Feel free to share and follow onGitHubandLinkedInfor updates.Your support is appreciated!
Package: dataflake.docbuilder
dataflake.docbuilderThis package provides a set of scripts to automate building
Sphinx-based package documentation for packages hosted on a
source code revision control server. At this point only Git
repositories are supported.check out the current development trunk and all tagged versionsbuild all Sphinx-based documentation in them, if it existsstitch together the trunk and release versions in a single
HTML file to provide a simple jump-off page for all package
versionsrebuild the Sphinx documentation if there were any changes
since the last build
Package: dataflake.fakeldap
dataflake.fakeldapThis package offers a mockpython-ldaplibrary that can be used
for testing code relying onpython-ldapwithout having to configure
and populate a real directory server.Starting with version 3.0 the library will behave just likepython-ldapversion 3.3 or higher:distinguished names, relative distinguished names, attribute names and
queries are expected to be native un-encoded string values.attribute values are expected to be bytes values.If you pass the wrong type of string, the library will raise aTypeError.
Seehttps://www.python-ldap.org/en/latest/bytes_mode.htmlfor a short
description of this behavior.Since queries are un-encoded string values and attribute values passed in the
query would never match bytes values in the data store, searches will always
test against both unencoded and bytes values.
Package: dataflake.ldapconnection
dataflake.ldapconnectionThis package provides an abstraction layer on top ofpyldap. It
offers a connection object with simplified methods for inserting,
modifying, searching and deleting records in the LDAP directory tree.
Failover/redundancy can be achieved by supplying connection data for
more than one LDAP server.DocumentationFull documentation is available athttp://dataflakeldapconnection.readthedocs.ioBug trackerA bug tracker is available athttps://github.com/dataflake/dataflake.ldapconnection/issues
Package: dataflake.wsgi.bjoern
dataflake.wsgi.bjoernThis package provides a PasteDeploy-compatible entry point to easily integrate
thebjoern WSGI serverinto an
environment that uses PasteDeploy-style.inifiles to compose a WSGI
application.It also includes a script to create a basic WSGI configuration file for Zope,
similar to Zope’s ownmkwsgiinstance, but specifyingbjoerninstead ofwaitressas WSGI server.Please note thatbjoernrequires thelibevevent library to be
installed.Changelog2.1 (2023-12-28)Add support for Pythn 3.12.2.0 (2023-02-02)Drop support for Python 2.7, 3.5, 3.6.1.2 (2022-06-27)Added support for Python 3.9 and 3.10Switched to GitHub Actions for CI
(#2)1.1 (2019-08-14)Added thelistenparameter to pass a port or a host and port.
This is more in line with whatplone.recipe.zope2instanceexpects.1.0 (2019-04-20)Initial release
Package: dataflake.wsgi.cheroot
dataflake.wsgi.cherootThis package provides a PasteDeploy-compatible entry point to easily integrate
thecheroot WSGI serverinto an
environment that uses PasteDeploy-style.inifiles to compose a WSGI
application.It also includes a script to create a basic WSGI configuration file for Zope,
similar to Zope’s ownmkwsgiinstance, but specifyingcherootinstead ofwaitressas WSGI server.Changelog2.1 (2024-01-03)Add support for Python 3.12.2.0 (2023-02-02)Drop support for Python 2.7, 3.5, 3.6.Added support for Python 3.9 and 3.10Switched to GitHub Actions for CI
(#1)1.0 (2019-04-20)Initial release
Package: dataflake.wsgi.werkzeug
dataflake.wsgi.werkzeugThis package provides a PasteDeploy-compatible entry point to easily integrate
thewerkzeug WSGI serverinto an
environment that uses PasteDeploy-style.inifiles to compose a WSGI
application.A second entry point will enable thewerkzeugdebugger, so you get nice
clickable tracebacks with the ability to open a console prompt at any point in
the stack. The debugger isexplained in the werkzeug documentationandshould never be running
in production.It also includes a script to create a basic WSGI configuration file for Zope,
similar to Zope’s ownmkwsgiinstance, but specifyingwerkzeuginstead ofwaitressas WSGI server.Changelog2.1 (2024-01-03)Add support for Python 3.12.2.0 (2023-02-02)Drop support for Python 2.7, 3.5, 3.6.1.1 (2022-06-27)Added support for Python 3.9 and 3.10Switched to GitHub Actions for CI
(#1)1.0 (2019-04-24)Initial release
Package: dataflat
dataflatA library to flatten all this annoiyng nested keys and columns on Dictionaries, Pandas Dataframes
and Spark (pyspark) Dataframes.Installationpip install dataflatGet startedHow to instantiate a Flattener:First import the FlattenerOptions, CaseTranslator and Flattener classesfromdataflat.flattener_handlerimportFlattenerOptions,CaseTranslatorOptions,FlattenerThe following step is assing the required variables for the flattening process.reference_name: Used to assign a key name to each resulting dictionary or DataFrame from the transform function.id_key: Used as identifier value in case there is any nested list, that will be exploded to a new entire dictionary or dataframe, the exploded Dataframes will have a reference to it parent Dataframe or dictionary.black_list: A list of keys or columns that will be skipped during the flattening process.replace_dots: Specify if the dots (used as hierarchical separator) will be replaced with underscores.from_case, to_case: Specify the keys or columns current case (snake, camel...) and the desired output case for each key.# Default values:#   from_case = CaseTranslatorOptions.CAMEL_CASE#   to_case = CaseTranslatorOptions.SNAKE_CASE#   replace_dots = False#   chunk size = 500reference_name='data'id_key='id'black_list=['keys','or','columns','to','skip']replace_dots=Falsefrom_case=CaseTranslatorOptions.CAMEL_CASEto_case=CaseTranslatorOptions.SNAKE_CASELater, select the desired flattener from the options, according to the current type of your data (dict, pandas.DataFrame, spark.DataFrame)custom_flattener=FlattenerOptions.DICTIONARYcustom_flattener=FlattenerOptions.PANDAS_DFcustom_flattener=FlattenerOptions.SPARK_DFFinally instantiate the flattener class, and apply the transform function.flattener=Flattener().handler(custom_flattener,from_case,to_case,replace_dots)flattener_dict.transform(data,id_key,black_list,reference_name)
Package: dataflow-cookiecutter
dataflow-cookiecutterTired of copy-pasting your ad-hoc Dataflow modules?Then you can use thiscookiecuttercommand-line
tool to easily generate standardized Dataflow templates! :zap:InstallationYou can installdataflow-cookiecutterfrom PyPI:pipinstalldataflow-cookiecutterIn addition, you can also clone this repository and install locally:gitclonehttps://github.com/ljvmiranda921/dataflow-cookiecutter.gitcddataflow-cookiecutter
python3setup.pyinstallUsageYou can create a Dataflow project by executing the command:$dataflow-cookiecutternewChoose from a variety of our premade templates.See all available templates
by runningdataflow-cookiecutter ls. For example, you can create a Google
Cloud Storage (GCS) to BigQuery (BQ) pipeline via:$dataflow-cookiecuternew-tGCSToBQLastly, our templates arehighly-compatible to your trusty, oldcookiecuttercommand-line tool (be sure to usecookiecutter>=1.7.1!):$cookiecutterhttps://github.com/ljvmiranda921/dataflow-cookiecutter\--directory<directory-to-desired-template>FAQWhy are you still wrapping cookiecutter?This started as my learning
project to see how cookiecutter's internals work. While building the alpha
version, I realized that I can add more functionality to this CLI more than
templating, so wrapping Cookiecutter seems to be a good approach.I already have cookiecutter, can I use it with your templates?Yes of
course! Look at the Usage section above! However, ensure that your
cookecutter version is>=1.7.1so that you can use the--directoryflag!Why are you using Python 3 for Dataflow templates?It's 2020, we
shouldn't be supportinglegacy Pythonanymore. Besides, Dataflow now hasstreaming support in Python
3.
See more developments for Beam support in Python 3  in theirissue tracker.