Saravanan666/pypi-org huggingface dataset:
Package: pipeline-notifier
Problem StatementData scientists code up a lot of pipelines that can often take a long time due to long pre-processing sequences or massive
amounts of time required. And if errors occur, then the pipeline needs to be re-run. This means we usually want to be notified
as early as possible if errors occurred.So what is pipelinenotifier?Pipeline notifier will send messages to your Slack/Discord/Keybase channel once you set up a webhook. Support can be added for
more channels if desired. Feel free to ping me atjacky.wong@vctr.ai. It will send error messages so you can tell what is wrong
the moment it happens.InstallationSimply run:python3 -m pip install pipeline-notifierData Science BotsAs data scientists, you are running a lot of code over a large amount of time. This often means that that you want to be notified about things via mobile while you have applications running in the background.I ended up re-writing this code across a number of organisations.from pipelinenotifier import KeyBaseNotifier

# Record your name and the project being worked on to properly track information
with KeyBaseBot("Bill Gates", "Modelling", "<webhook_url>") as bot:
    # Insert training loop/reader here.
    score = 0.5
    bot.send_message(f"Scored {score}.")If there is an error, it will send the error message to your channel.
Package: pipelineprofiler
PipelineProfilerAutoML Pipeline exploration tool compatible with Jupyter Notebooks. Supports auto-sklearn and D3M pipeline format.(Shift click to select multiple pipelines)Paper:https://arxiv.org/abs/2005.00160Video:https://youtu.be/2WSYoaxLLJ8Blog:Medium postDemoLive demo (Google Colab):Heart Stat Log dataauto-sklearn classificationIn Jupyter Notebook:importPipelineProfilerdata=PipelineProfiler.get_heartstatlog_data()PipelineProfiler.plot_pipeline_matrix(data)InstallOption 1: install via pip:pip install pipelineprofilerOption 2: Run the docker image:docker build -t pipelineprofiler .
docker run -p 9999:8888 pipelineprofilerThen copy the access token and log in to jupyter in the browser url:localhost:9999Data preprocessingPipelineProfiler reads data from the D3M Metalearning database. You can download this data from:https://metalearning.datadrivendiscovery.org/dumps/2020/03/04/metalearningdb_dump_20200304.tar.gzYou need to merge two files in order to explore the pipelines: pipelines.json and pipeline_runs.json.  To do so, runpython -m PipelineProfiler.pipeline_merge [-n NUMBER_PIPELINES] pipeline_runs_file pipelines_file output_filePipeline explorationimportPipelineProfilerimportjsonIn a jupyter notebook, load the output_filewithopen("output_file.json","r")asf:pipelines=json.load(f)and then plot it using:PipelineProfiler.plot_pipeline_matrix(pipelines[:10])Data postprocessingYou might want to group pipelines by problem type, and select the top k pipelines from each team. To do so, use the code:defget_top_k_pipelines_team(pipelines,k):team_pipelines=defaultdict(list)forpipelineinpipelines:source=pipeline['pipeline_source']['name']team_pipelines[source].append(pipeline)forteaminteam_pipelines.keys():team_pipelines[team]=sorted(team_pipelines[team],key=lambdax:x['scores'][0]['normalized'],reverse=True)team_pipelines[team]=team_pipelines[team][:k]new_pipelines=[]forteaminteam_pipelines.keys():new_pipelines.extend(team_pipelines[team])returnnew_pipelinesdefsort_pipeline_scores(pipelines):returnsorted(pipelines,key=lambdax:x['scores'][0]['value'],reverse=True)pipelines_problem={}forpipelineinpipelines:problem_id=pipeline['problem']['id']ifproblem_idnotinpipelines_problem:pipelines_problem[problem_id]=[]pipelines_problem[problem_id].append(pipeline)forprobleminpipelines_problem.keys():pipelines_problem[problem]=sort_pipeline_scores(get_top_k_pipelines_team(pipelines_problem[problem],k=100))
Package: pipeline-runner
BitBucket Pipeline Runner for Bamboo=========================Given a bitbucket pipeline file, run all steps locally. This will ignore the given Docker image.Usage=====```bash$ pip install pipeline-runner```Will install the latest version and it's required packages. You can then run the the pipeline runner with a pipeline file:```bash$ pipeline-runnerUsage: pipeline-runner [options]Options:-h, --help            show this help message and exit-f FILE, --file=FILE  Parse pipeline from FILE-b BRANCH, --branch=BRANCHParse pipeline for branch BRANCH-v                    Print scripts before running```Why?====We needed a way for us to move our Bamboo build scripts into version control, and have way to easily migrate our scripts later on. So we build this tool to do that
Package: pipeline-runtime
PipelineAI Home:http://pipeline.aiPipelineAI GitHub:https://github.pipeline.aiPipelineAI Quick Start:https://quickstart.pipeline.aiRequirements:
[‘pipeline-logger==1.0.2’, ‘pipeline-monitor==1.0.1’, ‘pipeline-model==1.0.10’, ‘cloudpickle>=0.3.1’]
Package: pipeline-turbo
pipeline-turbo is a package that will accelerate your processing pipeline. It works with the multi-threading concept in the background. It has been successful in both CPU and GPU tasks.The only pre-requisite is to load the function running for a single process and adjust the threads according to your resource availability.Read more about threading here:https://www.activestate.com/blog/how-to-manage-threads-in-python/InstallationUse the package managerpipto install pipeline-turbopipinstallpipeline-turboExample Usage# let's get some data for processingsentences=["Nevertheless, Trump and other Republicans have tarred the protests as havens for terrorists intent on destroying property.","Billie Eilish issues apology for mouthing an anti-Asian derogatory term in a resurfaced video.","Christians should make clear that the perpetuation of objectionable vaccines and the lack of alternatives is a kind of coercion.","There have been a protest by a group of people","While emphasizing heâ€™s not singling out either party, Cohen warned about the danger of normalizing white supremacist ideology."]sentences=sentences*100# Create your process - Here is an example of running a bias detection model across few sentencesfromtransformersimportAutoTokenizer,TFAutoModelForSequenceClassificationfromtransformersimportpipelinetokenizer=AutoTokenizer.from_pretrained("d4data/bias-detection-model")model=TFAutoModelForSequenceClassification.from_pretrained("d4data/bias-detection-model")classifier=pipeline('text-classification',model=model,tokenizer=tokenizer)defbias_classification(text):out=classifier(text)#label_ = out[0]['label']#probability_ = out[0]['score']returnout# without turbo, looping across all the sentences - Normal Methodout_list=[]forsentinsentences:out=bias_classification(sent)out_list.append(out)# with turbo, call the turbo_threading function"""1. Each of the item in 'sentences' list has to be iterated and that has to be defined as the first argument2. It should be followed by the function and its other arguments (if there are additional arguments for the function)3. Define the thread based on your resource availability (5, 10 would be ideal based on your resources)"""frompipeline_turbo.turboimportturbo_threading# import the turbo threading function which does the magicturbo_out=turbo_threading(sentences,bias_classification,num_threads=5)"""Note: You can pass any number of arguments inside the function, but the iterable list has to be defined firstThe performance varies based on the processing speed of your machine/compute"""AboutThis package is created by Deepak John Reji, Afreen Aman. It was first used to speed up some deep learning pipeline projects and later made it open source. This can be used for normal CPU process as well.LicenseMITLicense
Package: pipeline-turing-system
What does this library ?It give a very simple way to multitreat process over all your processors.
I use it to treat very large dataset in just-in-time way (cross process stream).Licence (TL;DR; version)This small program is safe to include in your projects, even commercial, and doesn't require to propagate the licence to projects that include it.
Details in the (LICENCE.md)[./LICENCE.md] (MIT licence)Get startedThis library is very simple.
How have exemple usage in tests, you will figure it out without problem.
Package: pipelineutils
Python library for interfacing with the drone pipeline.This package provides standard functions for interacting with drone pipeline
services and data.Primary use case: development of the drone pipeline.InstallationThe easiest way install pipelineutils is using pip and pulling from PyPI.:pip install pipelineutilsBecause this system is still under rapid development, you may need a
specific branch from the pipelineutils repository. You can either clone the
repository from GitHub and install locally with following commands:git clone https://github.com/terraref/drone-pipeline
git checkout <branch>
cd drone-pipeline/pipelineutils
pip install .Functions and Classesdronepipeline.pyutilities for interacting with the drone pipeline
Package: pipeline-utils
proc_pipeline_utilsShared utility functions for Coral Atlas data pipelinesAuthenticationTo function properly, this code needs credentials that it can use to authenticate against the Google Earth Engine API. Credentials can be provided in one of two ways, and the code will look for them in this order:Service Account Key.
Before running this code, create an environment variable named SERVICE_ACCOUNT_KEY containing the JSON key for a Google Cloud service account with the necessary permissions to perform the actions you're trying to perform in GEE.Personal Credentials.
Before running this code, authenticate with the Google Earth Engine API using the earthengine command line tool:$ earthengine authenticateRemember that authentication is only required if you are planning to give coordinates through earthengine asset as input.DevelopmentThis project usesPipenvto manage virtual environments and dependencies. Development-time dependencies are documented in thePipfile. Follow thePipenvdocumentation to create a virtual environment and install the dependencies.MakefileThe includedMakefileprescribes actions to test, build, and publish this code to a Python Package Index (PyPI) repository as described in the following sections.$ make [test | build | publish]TestingWith the virtual environment active and the dependencies installed, usepytestto run the test suite.To run the unit tests, run:$ pipenv run pytestBuildingThis project uses thesetuptoolsPython package for packaging as describedhere.When building via themake buildcommand, you may optionally append to the package name using theprereleaseargument. For example, if the current version ofpipelines_utilsspecified in thesetup.pymodule is1.0.1, then$ make build prerelease=rc1will produce a package namedpipelines_utils-1.0.1rc1. The default is a beta prerelease name incremented by each git commit (eg/1.0.1b7for the seventh commit on this branch). Specify a final release with$ make build prerelease=""Note that prerelease names must comply withPEP 440.PublishingThis project uses thetwinePython package for distribution as describedhere.When publishing via themake publishcommand, the default PyPI repository istestpypi. To publish topypi.org, specify that repository:$ make publish pypi_repository=pypiThe credentials necessary to publish to the target PyPI repository can be provided in one of two ways.As these environment variables:Env VarNotesPYPI_REPOSITORY_USERNAMEPyPI account username. If authenticatingusing a token, use the literal string__token__.PYPI_REPOSITORY_PASSWORDPyPI account password. If authenticatingusing a token, use the token contents.At themake publishcommand:$ make publish pypi_repository=pypi pypi_repository_username="foo" pypi_repository_password="bar"
Package: pipeline-viz
A Python package that provides a convenient way to visualize Scikit-learn machine learning pipelines. It utilizes libraries such as NetworkX, Matplotlib, and Plotly to generate clear, interactive, and insightful visual representations of your ML pipelinesInstallationYou can install Visualize Pipeline using pip:pipinstallvisualize-pipelineDependenciesVisualize Pipeline depends on the following Python libraries:--NetworkX
--Matplotlib
--Plotly
--Scikit-LearnYou can install these dependencies using pip:pipinstallnetworkxmatplotlibplotlyscikit-learnUsageBasic ExampleHere's a basic example of how to use Visualize Pipeline:fromvisualize_pipelineimportvisualize_pipelinefromsklearn.pipelineimportPipelinefromsklearn.preprocessingimportStandardScalerfromsklearn.linear_modelimportLogisticRegression# Create a simple pipelinepipe=Pipeline([('scale',StandardScaler()),('clf',LogisticRegression())])# Visualize the pipelinegraph=visualize_pipeline(pipe)# Save the graph to an HTML fileconvert_graph_to_plot(graph,'pipeline.html')This will create an interactive HTML file pipeline.html with the visualization of your pipeline.Example in Google Colab with a complex pipelinepipe=Pipeline(steps=[('scaler',StandardScaler()),('classifier',LogisticRegression())])preprocessor=ColumnTransformer(transformers=[('num',StandardScaler(),[0,1]),('cat',OneHotEncoder(),[2,3])])complex_pipeline=Pipeline(steps=[('preprocessor',preprocessor),('classifier',pipe)])graph=visualize_pipeline(complex_pipeline)# fig = convert_graph_to_plot(graph)# Convert the graph to a plotly figurefig=convert_graph_to_plot(graph,filename='pipeline.html')fromIPython.displayimportHTML# Display the HTML filedisplay(HTML('pipeline.html'))Visual ExampleScopeVisualize Pipeline currently supports Scikit-Learn's Pipeline, FeatureUnion, and ColumnTransformer classes. It can visualize pipelines with nested pipelines and feature unions/column transformers.The package is meant for visualizing the structure of your pipelines and does not show the actual data flow or transformations in the pipeline.ContributingContributions are welcome! Please open an issue or submit a pull request on the GitHub repository.
Package: pipelinewise-singer-python
pipelinewise-singer-pythonWrites the Singer format from Python.This is a fork ofSinger's singer-pythonmade forPipelineWise.UsageSetup environmentThis library depends on python3. We recommend using avirtualenvlike this:python3-mvenv~/.virtualenvs/singer-pythonInstallationNext, install this library:source~/.virtualenvs/singer-python/bin/activate
gitclonehttp://github.com/singer-io/singer-pythoncdsinger-python
makeinstallUsage exampleNow, from python code within the samevirtualenv, you can use the library:importsingersinger.write_schema('my_table',{'properties':{'id':{'type':'string','key':True}}},['id'])singer.write_records('my_table',[{'id':'b'},{'id':'d'}])singer.write_state({'my_table':'d'})Logging configurationpipelinewise-singer-pythonby default doesn't use any predefined logging configuration, it's up to the calling
library to define it. However, if the environment variableLOGGING_CONF_FILEis found and set then thepipelinewise-singer-pythonwould use the path provided in the env variable as the logging configuration for the logger.LicenseDistributed under the Apache License Version 2.0
Package: pipelinewise-tap-oracle
pipelinewise-tap-oracleSingertap that extracts data from aOracledatabase and produces JSON-formatted data following theSinger spec.This is aPipelineWisecompatible tap connector.How to use itThe recommended method of running this tap is to use it fromPipelineWise. When running it from PipelineWise you don't need to configure this tap with JSON files and most of things are automated. Please check the related documentation atTap OracleIf you want to run thisSinger Tapindependently please read further.Log based replicationTap-Oracle Log-based replication requires some configuration changes in Oracle database:EnableARCHIVELOGmodeSet retention period a reasonable and long enough period, ie. 1 day, 3 days, etc.Enable Supplemental loggingSetting up Log-based replication on a self hosted Oracle Database:To verify the current archiving mode, if the result isARCHIVELOG, archiving is enabled:SQL> SELECT LOG_MODE FROM V$DATABASETo enableARCHIVELOGmode (if not enabled yet):SQL> SHUTDOWN IMMEDIATE
  SQL> STARTUP MOUNT
  SQL> ALTER DATABASE ARCHIVELOG
  SQL> ALTER DATABASE OPENTo set retention period, use RMAN:RMAN> CONFIGURE RETENTION POLICY TO RECOVERY WINDOW OF 1 DAYS;To enable supplemental logging:SQL> ALTER DATABASE ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNSSetting up Log-based replication on Oracle on Amazon RDSTo set retention period:begin
      rdsadmin.rdsadmin_util.set_configuration(
          name  => 'archivelog retention hours',
          value => '24');
  end;To enable supplemental logging:begin
    rdsadmin.rdsadmin_util.alter_supplemental_logging(p_action => 'ADD');
  end;Install and RunFirst, make sure Python 3 is installed on your system or follow these
installation instructions forMacorUbuntu.It's recommended to use a virtualenv:python3-mvenvvenvpipinstallpipelinewise-tap-oracleorpython3-mvenvvenv.venv/bin/activatepipinstall--upgradepippipinstall.ConfigurationRunning the the tap requires aconfig.jsonfile. Example with the minimal settings:{"host":"foo.com","port":1521,"user":"my_user","password":"password","sid":"ORCL","filter_schemas":"MY_USER"#optional}To run tests:Tests require Oracle on Amazon RDS >= 12.1, and a user calledROOT.Define environment variables that requires running the tests.export TAP_ORACLE_HOST=<oracle-rds-host>
  export TAP_ORACLE_PORT=<oracle-rds-port>
  export TAP_ORACLE_USER=ROOT
  export TAP_ORACLE_PASSWORD=<oracle-rds-password>
  export TAP_ORACLE_SID=<oracle-rds-sid>Install python dependencies in a virtual env and run nose unit and integration testspython3 -m venv venv
  . venv/bin/activate
  pip install --upgrade pip
  pip install .
  pip install noseTo run unit tests:nosetests
Package: pipelinewise-tap-salesforce
pipelinewise-tap-salesforceSingertap that extracts data fromSalesforceand produces JSON-formatted output following theSinger spec.This is aPipelineWisecompatible tap connector.How to use itThe recommended method of running this tap is to use it fromPipelineWise. When running it from PipelineWise you don't need to configure this tap with JSON files and most of things are automated. Please check the related documentation atTap SalesforceIf you want to run thisSinger Tapindependently please read further.Install and RunFirst, make sure Python 3 is installed on your system or follow these
installation instructions forMacorUbuntu.It's recommended to use a virtualenv:python3-mvenvvenvpipinstallpipelinewise-tap-salesforceorpython3-mvenvvenv.venv/bin/activatepipinstall--upgradepippipinstall.Create a Config file{
  "client_id": "secret_client_id",
  "client_secret": "secret_client_secret",
  "refresh_token": "abc123",
  "start_date": "2017-11-02T00:00:00Z",
  "api_type": "BULK",
  "select_fields_by_default": true
}Theclient_idandclient_secretkeys are your OAuth Salesforce App secrets. Therefresh_tokenis a secret created during the OAuth flow. For more info on the Salesforce OAuth flow, visit theSalesforce documentation.Thestart_dateis used by the tap as a bound on SOQL queries when searching for records.  This should be anRFC3339formatted date-time, like "2018-01-08T00:00:00Z". For more details, see theSinger best practices for dates.Theapi_typeis used to switch the behavior of the tap between using Salesforce's "REST" and "BULK" APIs. When new fields are discovered in Salesforce objects, theselect_fields_by_defaultkey describes whether or not the tap will select those fields by default.Run DiscoveryTo run discovery mode, execute the tap with the config file.> tap-salesforce --config config.json --discover > properties.jsonSync DataTo sync data, select fields in theproperties.jsonoutput and run the tap.> tap-salesforce --config config.json --properties properties.json [--state state.json]Based on Stitch documentation
Package: pipelinewise-tap-zendesk
Pipelinewise-tap-zendeskSingertap that extracts data from a Zendesk API and produces JSON-formatted data following
theSinger spec.This is aPipelineWisecompatible tap connector.How to use itThe recommended method of running this tap is to use it fromPipelineWise. When running it from PipelineWise you don't need to configure this tap with JSON files and most of things are automated. Please check the related documentation atKafkaIf you want to run thisSinger Tapindependently please read further.Install and RunFirst, make sure Python 3 is installed on your system or follow these
installation instructions forMacorUbuntu.It's recommended to use a virtualenv:python3-mvenvvenvpipinstallpipelinewise-tap-zendeskorpython3-mvenvvenv.venv/bin/activatepipinstall--upgradepippipinstall-e.[test]ConfigurationAuthenticationUsing OAuthOAuth is the default authentication method fortap-zendesk. To use OAuth, you will need to fetch anaccess_tokenfrom a configured Zendesk integration. Seehttps://support.zendesk.com/hc/en-us/articles/203663836for more details on how to integrate your application with Zendesk.config.json{"access_token":"AVERYLONGOAUTHTOKEN","subdomain":"acme","start_date":"2000-01-01T00:00:00Z"}Using API TokensFor a simplified, but less granular setup, you can use the API Token authentication which can be generated from the Zendesk Admin page. Seehttps://support.zendesk.com/hc/en-us/articles/226022787-Generating-a-new-API-token-for more details about generating an API Token. You'll then be able to use the admins'semailand the generatedapi_tokento authenticate.config.json{"email":"user@domain.com","api_token":"THISISAVERYLONGTOKEN","subdomain":"acme","start_date":"2000-01-01T00:00:00Z"}Run the tap in Discovery Modetap-zendesk --config config.json --discover                # Should dump a Catalog to stdout
tap-zendesk --config config.json --discover > catalog.json # Capture the CatalogRun the tap in Sync Modetap-zendesk --config config.json --catalog catalog.jsonThe tap will write bookmarks to stdout which can be captured and passed as an optional--state state.jsonparameter to the tap for the next sync.To run tests:Install python test dependencies in a virtual env and run nose unit and integration testspython3 -m venv venv
  . venv/bin/activate
  pip install --upgrade pip
  pip install .[test]To run tests:nosetests testTo run pylint:Install python dependencies and run python linterpython3 -m venv venv
  . venv/bin/activate
  pip install --upgrade pip
  pip install .[test]
  pylint --rcfile .pylintrc tap_zendesk
Package: pipelinewise-tap-zuora
pipelinewise-tap-zuoraSingertap that extracts data from theZuora APIand produces JSON-formatted data following theSinger spec.This is aPipelineWisecompatible tap connector.instructionsMake a virtualenv and install this tap:pip install -e .Write config into new file config.json:# config.json content:
{
    "username": "<username>",
    "password": "<password>",
    "api_type": "AQUA", # or REST
    "start_date": "2020-04-01",
    "sandbox": "false",
    "european": "true",
    "partner_id": "<partner_id>"
}Run discovery mode in order to obtain the schema with everything that can be exported:tap-zuora -c config.json --discover > catalog.jsonSave the modified file as e.g.catalog_aqua_selection.jsonto use when running the sync in the next step.Run the tap:tap-zuora -c config.json --catalog catalog_aqua_selection.jsonBased on Stitch documentation