Saravanan666/pypi-org huggingface dataset:
Package: data-flow-diagram
DFDDFD (Data Flow Diagram) Generator - Commandline tool to generate
diagrams as images in various formats (SVG, PNG, JPG, PDF, etc.) from
source text files.The source for this project isavailable here.
The package page isavailable here.ScopeThe present tool allows to create diagrams following the "Structured
Analysis / Stuctured Design" (SA/SD) modelling technique as described
by Edward Yourdon.SA/RT (which is SA/SD with a real-time extension), can be found in the
design of real-time and embedded systems.
Example:Firmware modelling of the Active Descent
Systemof theRosetta
Landerspacecraft.Surprisingly, SA/RT is also well-suited to the analyse and design of
web micro-services and IoT architectures.SummaryData Flow Diagram are used to model the flow and processing of
information through a system.process	P	Process
process	P2	Process
entity	E	Entity
store	S	Store
channel	C	Channel

E  --> P	flow
P  ::> P2	signal
P  <-> S	bidirectional flow
P  --- C	undirected flowSyntax and examplesSee thedocumentation page.DependenciesPython3GraphvizInstalling via pip3[sudo] pip3 install data-flow-diagramInstallation troubleshootingIf you get an error at runtime like:ModuleNotFoundError: No module named 'reportlab.graphics._renderPM'you may have to reinstall reportlab:[sudo] python3 -m pip install --upgrade --force-reinstall reportlabUsagedata-flow-diagram -hsays:usage: data-flow-diagram [-h] [--output-file OUTPUT_FILE] [--markdown]
                         [--format FORMAT] [--percent-zoom PERCENT_ZOOM]
                         [--background-color BACKGROUND_COLOR] [--debug]
                         [INPUT_FILE]

Command-line DFD diagram generator. Converts a textual description into a
graphic file.

positional arguments:
  INPUT_FILE            UML sequence input file; if omitted, stdin is used

options:
  -h, --help            show this help message and exit
  --output-file OUTPUT_FILE, -o OUTPUT_FILE
                        output file name; pass '-' to use stdout; if
                        omitted, use INPUT_FILE base name with '.svg'
                        extension, or stdout
  --markdown, -m        consider snippets between opening marker:
                        ```data-flow-diagram OUTFILE, and closing marker:
                        ``` allowing to generate all diagrams contained in
                        an INPUT_FILE that is a markdown file
  --format FORMAT, -f FORMAT
                        output format: gif, jpg, tiff, bmp, pnm, eps, pdf,
                        svg (any supported by Graphviz); default is svg
  --no-graph-title      suppress graph title
  --no-check-dependencies
                        suppress dependencies checking
  --debug               emits debug messages

See https://github.com/pbauermeister/dfd for information, syntax and
examples.
Package: dataflow-gegham-test
DataFlow ·Prototyping for Hub DataflowSetupInstall the package> pip3 install git+https://github.com/snarkai/dataflow@v0.2.2Make sure gdal and opencv is installed as required byimage_io.pySetup credentials in the local folder and don't forget to put those in .gitignore.secrets/agmri.cfg
.secrets/intelinair #AWS credentialsExample to get startedfrom dataflow import hub
import torch

hub.init()

df = hub.agmri().get_polygons_by_type(hub.agmri.UIUC_ENDROW, max=1) 
ds = hub.polygon_sampler(df, sample_count=32, shape=(1024,1024,4))
ds[0]TroubleshootingInstall GDAL and OpenCVIf it throws error on gdal or opencv install those manually./bin/prepare.sh
./bin/gdal.sh
./bin/opencv.shCredentials examplesintelinairfile should look like this[default]
aws_access_key_id = ...
aws_secret_access_key = ...
region = us-east-1agmri.cfg[production]
admin_username = ...
admin_password = ...Known Issues[ ] Credentials are not flexible where to be storedPush a new versiongit commit -a -m 'current changes'
git tag -a 'v0.2.2' -m 'some message here'
git push origin 'v0.2.2'Automated integration testingTo run pytest before git push we need to do create .git/hooks/pre-push file with the following content:#!/bin/bash
docker-compose run testAnd make it executablechmod +x .git/hooks/pre-pushAlso whenever updating requirements.txt and options.txt please rebuild the testdocker-compose build testFormatting and LintingHub uses Black and Flake8 to ensure a consistent code format throughout the project.
Replace .vscode/settings.json content withthe following:{
    "[py]": {
        "editor.formatOnSave": true
    },
    "python.formatting.provider": "black",
    "python.linting.flake8Enabled": true,
    "python.linting.flake8Path": "flake8",
    "python.linting.flake8Args": [
        "--max-line-length=80",
        "--select=B,C,E,F,W,B950",
        "--ignore=E203,E501,W503"
    ],
    "python.linting.pylintEnabled": false,
    "python.linting.enabled": true,
}
Package: dataflows-aws
dataflows-awsDataflows's processors to work with AWSFeaturesdump_to_s3processorchange_acl_on_s3processorContentsGetting StartedInstallationExamplesDocumentationdump_to_s3change_acl_on_s3ContributingChangelogGetting StartedInstallationThe package use semantic versioning. It means that major versions  could include breaking changes. It's recommended to specifypackageversion range in yoursetup/requirementsfile e.g.package>=1.0,<2.0.$pipinstalldataflows-awsExamplesThese processors have to be used as a part of data flow. For example:flow=Flow(load('data/data.csv'),dump_to_s3(bucket=bucket,acl='private',path='my/datapackage',endpoint_url=os.environ['S3_ENDPOINT_URL'],),)flow.process()Documentationdump_to_s3Saves the DataPackage to AWS S3.Parametersbucket- Name of the bucket where DataPackage will be stored (should already be created!)acl- ACL to provide the uploaded files. Default is 'public-read' (seeboto3 docsfor more info).path- Path (key/prefix) to the DataPackage. May contain format string available fordatapackage.jsonEg:my/example/path/{owner}/{name}/{version}content_type- content type to use when storing files in S3. Defaults to text/plain (usual S3 default is binary/octet-stream but we prefer text/plain).endpoint_url- api endpoint to allow using S3 compatible services (e.g. 'https://ams3.digitaloceanspaces.com')change_acl_on_s3Changes ACL of object in given Bucket with given path aka prefix.Parametersbucket- Name of the bucket where objects are storedacl- Available options'private'|'public-read'|'public-read-write'|'authenticated-read'|'aws-exec-read'|'bucket-owner-read'|'bucket-owner-full-control'path- Path (key/prefix) to the DataPackage.endpoint_url- api endpoint to allow using S3 compatible services (e.g. 'https://ams3.digitaloceanspaces.com')ContributingThe project follows theOpen Knowledge International coding standards.The recommended way to get started is to create and activate a project virtual environment.
To install package and development dependencies into your active environment:$ make installTo run tests with linting and coverage:$maketestFor linting,pylama(configured inpylama.ini) is used. At this stage it's already
installed into your environment and could be used separately with more fine-grained control
as described in documentation -https://pylama.readthedocs.io/en/latest/.For example to sort results by error type:$pylama--sort<path>For testing,tox(configured intox.ini) is used.
It's already installed into your environment and could be used separately with more fine-grained control as described in documentation -https://testrun.org/tox/latest/.For example to check subset of tests against Python 2 environment with increased verbosity.
All positional arguments and options after--will be passed topy.test:tox-epy37---vtests/<path>Under the hoodtoxusespytest(configured inpytest.ini),coverageandmockpackages. These packages are available only in tox envionments.ChangelogHere described only breaking and the most important changes. The full changelog and documentation for all released versions can be found in the nicely formattedcommit history.v0.xan initial processors implementation
Package: dataflows-ckan
dataflows-ckanDataflows processors to work with CKAN.Featuresdump_to_ckanprocessorContentsGetting StartedInstallationExamplesDocumentationdump_to_s3ContributingChangelogGetting StartedInstallationThe package use semantic versioning. It means that major versions  could include breaking changes. It's recommended to specifypackageversion range in yoursetup/requirementsfile e.g.package>=1.0,<2.0.$pipinstalldataflows-ckanExamplesThese processors have to be used as a part of data flow. For example:flow=Flow(load('data/data.csv'),dump_to_ckan(host,api_key,owner_org,overwrite_existing_data=True,push_to_datastore=False,push_to_datastore_method='insert',**options,),)flow.process()Documentationdump_to_ckanSaves the DataPackage to a CKAN instance.ContributingCreate a virtual environment and installPoetry.Then install the package in editable mode:$ make installRun the tests:$ make testFormat your code:$ make formatChangelog0.2.0Full port to dataflows, and some refactoring, with a basic integration test.0.1.0an initial port fromhttps://github.com/frictionlessdata/datapackage-pipelines-ckanbased on the great work of @brew and @amercader
Package: dataflows-elasticsearch
dataflows-elasticsearchDataflows's processors to work with ElasticSearchFeaturesdump_to_elasticsearchprocessorContentsGetting StartedInstallationExamplesDocumentationdump_to_esContributingChangelogGetting StartedInstallationThe package use semantic versioning. It means that major versions  could include breaking changes. It's recommended to specifypackageversion range in yoursetup/requirementsfile e.g.package>=1.0,<2.0.$pipinstalldataflows-elasticsearchExamplesThese processors have to be used as a part of a dataflowsFlow. For example:flow=Flow(load('data/data.csv'),dump_to_es(engine='localhost:9200',),)flow.process()Documentationdump_to_esSaves the Flow to an ElasticSearch Index.Parametersindexes- Mapping of indexe names to resource names, e.g.{
  'index-name-1': {
    'resource-name': 'resource-name-1',
  },
  'index-name-2': {
    'resource-name': 'resource-name-2',
  },
  # ...
}mapper_cls- Class to be used to map json table schema types into ElasticSearch typesindex_settings- Options to be used when creating the ElasticSearch indexengine- Connection string for connecting the ElasticSearch instance, or anElasticsearchobject.
Can also be of the formenv://ENV_VAR, in which case the connection string will be fetched from the environment variableENV_VAR.elasticsearch_options- Options to be used when creating theElasticsearchobject (in case it wasn't provided)ContributingThe project follows theOpen Knowledge International coding standards.The recommended way to get started is to create and activate a project virtual environment.
To install package and development dependencies into your active environment:$ make installTo run tests with linting and coverage:$maketestFor linting,pylama(configured inpylama.ini) is used. At this stage it's already
installed into your environment and could be used separately with more fine-grained control
as described in documentation -https://pylama.readthedocs.io/en/latest/.For example to sort results by error type:$pylama--sort<path>For testing,tox(configured intox.ini) is used.
It's already installed into your environment and could be used separately with more fine-grained control as described in documentation -https://testrun.org/tox/latest/.For example to check subset of tests against Python 2 environment with increased verbosity.
All positional arguments and options after--will be passed topy.test:tox-epy37---vtests/<path>Under the hoodtoxusespytest(configured inpytest.ini),coverageandmockpackages. These packages are available only in tox envionments.ChangelogThe full changelog and documentation for all released versions can be found in the nicely formattedcommit history.
Package: datafluent
Data Fluent for PostgreSQLBuild a better understanding of your data in PostgreSQL.The following shows an example report generated by this tool. It gives the numbers of rows, columns, bytes as well as human-friendly size counts for each table within a given PostgreSQL database.The following shows the row count for every column that represents a date grouped by year and month.Installation and UsagePlease readthis guidefor installation instructions and example usage.Further HelpThe database URL is passed through to sqlalchemy. Please see their documentation onDatabase URLsfor more information on their syntax and the drivers supported.If you need to override any parameters in this application, please refer to the help documentation generated by typer:$datafluent--helpUsage: datafluent [OPTIONS]

Options:
  --url TEXT                      [default: postgresql://localhost:5432/intel]
  --output TEXT                   [default: fluency.xlsx]
  --install-completion [bash|zsh|fish|powershell|pwsh]
                                  Install completion for the specified shell.
  --show-completion [bash|zsh|fish|powershell|pwsh]
                                  Show completion for the specified shell, to
                                  copy it or customize the installation.

  --help                          Show this message and exit.
Package: datafog
Open-source PII Detection for Retrieval Systems.Scan, redact, and manage PII in your documents before they get uploaded to a Retrieval Augmented Generation (RAG) system.OverviewDataFog works by scanning and redacting-out PII in filesbeforeare uploaded to a RAG system.How it worksInstallationDataFog can be installed via pip:pipinstalldatafog# python clientDev NotesClone repoRun 'poetry install' to install dependencies (recommend entering poetry shell for preserving dependencies)Justfile commands:just formatto apply formatting.just lintto check formatting and style.just tagto tag your project on gitjust uploadto publish to PyPi.TestingTo run the datafog unit tests, check out this repository and dotoxLicenseThis software is published under theMIT
license.
Package: datafold-sdk
datafold-sdkAn SDK library for the datafold application.InstallationFirst, create your virtual environment for python:>python3-mvenvvenv
>sourcevenv/bin/activate
>pipinstall--upgradepipsetuptoolswheelNow, you're ready to install the datafold SDK:>pipinstalldatafold-sdkDatafold CLIThe datafold CLI is a utility that automates and simplifies certain tasks.
Help for using this utility is available in the utility itself:>datafold--helpPreliminary Datafold SDKThe Datafold SDK are python functions and classes that allow you to integrate
these CLI functions directly in your application source code.In the future, the SDK will contain a more type-safe and integrated "API" client library,
which further facilitates integration with the datafold app server.The only function currently implemented is thesubmit_artifactsfunction to upload
thedbt runartifacts to the app server:importosfromdatafold.sdk.dbtimportsubmit_artifactshost=os.environ.get("DATAFOLD_HOST")api_key=os.environ.get('DATAFOLD_API_KEY')submit_artifacts(host=host,api_key=api_key,ci_config_id=1,run_type='pull_request',target_folder='<abs-path-to-dbt-target-folder>',commit_sha='abcdef1234567890')
Package: datafold-xdiff
datafold-sdkAn SDK library for the datafold application.InstallationFirst, create your virtual environment for python:>python3-mvenvvenv
>sourcevenv/bin/activate
>pipinstall--upgradepipsetuptoolswheelNow, you're ready to install the datafold SDK:>pipinstalldatafold-sdkDatafold CLIThe datafold CLI is a utility that automates and simplifies certain tasks.
Help for using this utility is available in the utility itself:>datafold--helpPreliminary Datafold SDKThe Datafold SDK are python functions and classes that allow you to integrate
these CLI functions directly in your application source code.In the future, the SDK will contain a more type-safe and integrated "API" client library,
which further facilitates integration with the datafold app server.The only function currently implemented is thesubmit_artifactsfunction to upload
thedbt runartifacts to the app server:importosfromdatafold.sdk.dbtimportsubmit_artifactshost=os.environ.get("DATAFOLD_HOST")api_key=os.environ.get('DATAFOLD_APIKEY')submit_artifacts(host=host,api_key=api_key,ci_config_id=1,run_type='pull_request',target_folder='<abs-path-to-dbt-target-folder>',commit_sha='abcdef1234567890')
Package: dataforge
NoteTested with Python 3.11.Dataforge is a Python package containing tools for creating and packaging data
products.ContentsInstallationConfigurationREDCapInstallationNoteInstallation in a virtual environment of the type created by thevenv moduleorvirtualenvis highly
recommended.To install dataforge:pip install dataforgeOptionalFrictionlessplugins can
be enabled by installing with thefrictionlessoption:pip install dataforge[frictionless]OptionalREDCapsupport requiresxmarshal, which must be compiled usingCython. Thus, prior to installation, you
will need a C compiler:Linuxsudoapt-getinstallbuild-essentialMac OS XInstall Apple’s XCode from the App StoreWindowsInstallMicrosoft’s C++ Build ToolsOnce you have a C compiler, you may install dataforge with REDCap support
using:pip install dataforge[redcap]ConfigurationFor ease of use, many arguments and options for executable commands as well as
many function arguments may be specified in a file calledconfig.yamllocated at the root of your project. For example, you could specify arguments
and options for theredcap_exportcommand by creating aconfig.yamlfile containing something like this:sources.redcap.api:
    url: https://redcap.uchicago.edu/api/
    project_name: MyREDCapProject
    outdir: tmp/redcapREDCapManagement of API TokensTo access a REDCap project via the API, you will need to obtain a REDCap API
token. To avoid having to enter the token each time you export data, it is
convenient to add the token to your system keyring. You may do this with the
following command:keyring set <redcap_hostname> [<project_name>]where<redcap_hostname>is the hostname of your REDCap server (e.g.,redcap.uchicago.edu) and<project_name>is an abbreviated (no spaces)
name of the REDCap project (e.g.,MyREDCapProject). Although specifying a
project name is not required, doing so will allow you to store API tokens for
multiple projects. When you are asked to enter a password, enter the API token
you obtained for your project.WarningDo not store your API token in a plaintext or otherwise
unencrypted file, and do not write it down on a piece of paper.
If another person were to obtain your API token, he or she would
have the same access to your REDCap project as you do. If there
is ever a possibility that your token has been compromised, you
should immediately regenerate your token.
Package: dataforgelab
Quant ResearchWhat is this projet?This is a project aimed for quant, or people want to devote themselves in quantative field. I concluded a lot of experience and sample codes in this project covering from data collection, quantative investment platform to actual factor digging process and so on. This project is still carrying on and ready to bring everyone in quantative field a better experience.How can we use this project?Currently, I have no plan for publishing a independent pypi package for this project, for most of the code is created in research process. I you find some of the code in this project do help you, you can try to clone the project to your own computer, and import the corresponding package. More easily, just copy and paste the code in your project.References for QIMLSome paper references and quantative investment and machine learningDouble EnsembleDataWe provide data byBaiduNetDisk.
Package: dataframe2ndjson
dataframe2ndjsondataframe2ndjson converts a dataframe to ndjson file.Installationpip install dataframe2ndjsonHow to use it?import dataframe2ndjsondataframe2ndjson.df_to_ndjson(df,path)LicenseÂ© 2021 Nishank ParasharThis repository is licensed under the MIT license. See LICENSE for details.
Package: dataframe-diff
dataframe_diffdataframe_diff is a micro-library which takes two dataframes as input , compares them and return two dataframes with column wise comparison and additional records.Installationpip install dataframe-diffExamples>>>importpandasaspd>>>df1=pd.read_csv('students_1.csv')>>>df2=pd.read_csv('students_2.csv')>>>fromdataframe_diffimportdataframe_diff>>>df1.head()NameSubjectsMarksGrade0LeonardEng70B1LeonardMath80B2LeonardPhysics90A3SheldonEng90A4SheldonMath99A>>>df2.head()NameSubjectsMarksGrade0LeonardEng75A1LeonardMath85A2LeonardPhysics90A3SheldonEng99A4SheldonMath99A>>>d1_column,d2_additional=dataframe_diff(df1,df2,key=['Name','Subjects'])>>>d1_columnNameSubjectsvalue_xvalue_ycolumn_name0LeonardEng7075Marks1LeonardEngBAGrade2LeonardMath8085Marks3LeonardMathBAGrade4SheldonEng9099Marks5PennyPhysics6575Marks6PennyPhysicsCBGrade>>>d2_additionalNameSubjectsMarksGradesets0RajeshMath93Adf_x1HowardChemistry83Bdf_y
Package: dataframe-mapper
THIS IS EXPERIMENTAL.RequirementsPython 3.5+Pandas 0.20.0+InstallationInstall DataFrame Mapper via pip:$ pip install dataframe-mapperExampleSimple DataFrame Mapper example:from dfmapper import DataFrameMapper, IntColumn, StrColumn

class UserDfm(DataFrameMapper):

    id = IntColumn(min=1, nullable=False)
    username = StrColumn(max_length=30, nullable=False)
    profile = StrColumn()

    def find_by_id(self, id):
        return self.df[self.df.id == id]

user_dfm = UserDfm({
    "id": [1, 2, 3],
    "username": ["Bessie Bennett", "Sandra Matthews", "Jessie Bates"],
    "profile": ["BLAH BLAH BLAH", "PITH PITH PITH", None]
})

user_dfm.is_valid()
#: True

user_dfm.find_by_id(1)
#:    id username       profile
#: 0  1  Bessie Bennett BLAH BLAH BLAHLicenseDataFrame Mapper is licensed under MIT License. SeeLICENSEfor more information.
Package: dataframe-memory
Dataframe Memory ProjectThis tools aims to provide simple solution to save memory when using pandas' data frame.
It is highly inspired by thiskaggle post.[!IMPORTANT]
The very basic principle : for each column, this tool reduces int and float precision as much as possible so thatApproximate methodmethod='approx': no duplicated values appear and  the minimum and maximum can be re-encodedExact methodmethod='exact': preserve absolute information by testing every value.For object data type, the function is trying to create category.Usagefromdata_memoryimportreduce_memoryimportnumpyasnpimportpandasaspddf=pd.DataFrame(np.array([[1,2,"aaa"],[4,5,"bbb"],[7,8,"ccc"]]*10000),columns=['a','b','c'])reduce_memory(df,method="exact",verbose=True)Yields the following decrease of memoryMemory usage input: 5.04 MB
Memory usage output: 0.09 MB
Decreased by: 98.28 %df.info()#   Column  Non-Null Count  Dtype   
---  ------  --------------  -----   
 0   a       30000 non-null  category
 1   b       30000 non-null  category
 2   c       30000 non-null  category[!WARNING]
Inmethod='approx',This tooldestroysinformation andshould not be applied automaticallyto any dataframe but big onesIt preserves relative but not absolute information