Saravanan666/pypi-org huggingface dataset:
Package: airflow_aws_shared_secrets
airflow-aws-shared-secretsSecretsManagerBackend with cross-account access.Expected properties where:shared_account: account_id from your core aws-accountaws_region: aws_region where the core secrets are stored in{"connections_prefix":"airflow/connections/${environment}","connections_prefix_shared":"airflow/core/connections/${environment}","shared_account":"123456789012","aws_region":"eu-central-1"}UsageWe recommend setting up the SharedSecretsManager in
theAirflow Helmby configuring
the following config values.### [secrets]AIRFLOW__SECRETS__BACKEND:'airflow_aws_shared_secrets.secret_manager.SharedSecretsManagerBackend'AIRFLOW__SECRETS__BACKEND_KWARGS:'{"connections_prefix": "airflow/connections/${environment}", "connections_prefix_shared" : "airflow/core/connections/${environment}", "shared_account": "<my_core_aws_account_id>", "aws_region": "eu-central-1"}'Checkconfigurations-reffor more
Airflow configuration possibilities.
Package: airflow-bigquerylogger
BigQuery logger handler for AirflowInstallationpip install airflow-bigqueryloggerConfigurationAIRFLOW__CORE__REMOTE_LOGGING='true'AIRFLOW__CORE__REMOTE_BASE_LOG_FOLDER='gs://bucket/path'AIRFLOW__CORE__REMOTE_LOG_CONN_ID='gcs_log'AIRFLOW__CORE__LOGGING_CONFIG_CLASS='bigquerylogger.config.LOGGING_CLASS'AIRFLOW__CORE__LOG_BIGQUERY_DATASET='dataset.table'AIRFLOW__CORE__LOG_BIGQUERY_LIMIT=50Google Cloud BigQueryRows that were written to a table recently via streaming (using the tabledata.insertall method) cannot be modified using UPDATE, DELETE, or MERGE statements. I recommend setting up a table retention!CreditsThanks to Bluecore engineering team forthis usefull article.
Package: airflow-commons
airflow-commonsA python package that contains common functionalities for airflowInstallationUse the package manager pip to install airflow-commons.pipinstallairflow-commonsModulesbigquery_operator: With this module you can manage your Google BigQuery operations.mysql_operator: Using this module, you can connect to your MySQL data source and manage your data operations.s3_operator: This operator connects to your s3 bucket and lets you manage your bucket.glossary: This module consists of constants used across projectsql_resources: Template BigQuery and MySQL queries such as merge, delete, select etc. are located here.utils: Generic methods like connection, querying etc. are implemented in this module.UsageSample deduplication code works like:fromairflow_commonsimportbigquery_operatorbigquery_operator.deduplicate(service_account_file="path_to_file",start_date="01-01-2020 14:00:00",end_date="01-01-2020 15:00:00",project_id="bigquery_project_id",source_dataset="source_dataset",source_table="source_table",target_dataset="target_dataset",target_table="target_table",oldest_allowable_target_partition="01-01-2015 00:00:00",primary_keys=["primary_keys"],time_columns=["time_columns"],allow_partition_pruning=True,)
Package: airflow-connection-plugin
Templating for Airflow connectionsThe connection plugin contains anAirflowmacro fortemplatingconnections in tasks.
You can use it like this:# prints 'mysql'
{{ macros.connection_plugin.get_conn('airflow_db').host }}connection_plugin.get_connreturns theConnection objectthat you can interact with as described in the documentation.Installationpip install airflow-connection-pluginDemoTo start the docker container simply run the following command in the root directory:cd example && docker-compose upAfter that you can reach the airflow frontend viahttp://localhost:8080.
You will find an example DAG that demonstrates how to retrieve different connection information.Attention: Be especially careful when using passwords in templates.
Package: airflow-connections-manager
Airflow connections managerInterface to access Airflow connections. Actually it works in Airflow environments with the API enabled with Basic AuthenticationUsageYou must have the following environment variables declaredAIRFLOW_API_URL=<your Airflow API url like http://localhost:8080/api/v1>
AIRFLOW_API_TOKEN=<your Airflow Basic Auth token like Basic YXRtaW46YHRt8W4=>Samplefrom airflow_connections_manager import AirflowConnectionsManager

connections = AirflowConnectionsManager.list_connections()Building for PyPi deploymenthttps://packaging.python.org/en/latest/tutorials/packaging-projects/python -m pip install --upgrade build

python -m build

python -m twine upload --repository testpypi dist/*
Package: airflowconversion
Python Utility tool for migration of workflows from Oozie to AirflowInputs required :Input folder path (Where the XML files to be converted are stored.)Output folder path (Where the converted python files are to be stored)Queue name (for the DAG)How to use this library:from airflowconversion.ParseXML import conversionconversion(r"<input_folder_path>", r"<output_folder_path>",'<queue_name>')
Package: airflow-customs-by-novigi
Novigi Custom Airflow Operators ,hooks and pluginsThis Repo mainly contains custom airflow operators and hooks which were writen by Novigi Pty LtdVersion No - 1.0.10Bitbucket Link -https://bitbucket.org/novigi/nov20011-airflow-common-extensions/src/master/How do I get set up?Open a terminal and just type "sudo pip install airflow_customs_by_novigi"Who do I talk to?integration@novigi.com.au
Package: airflow-cyberark-secrets-backend
airflow-cyberark-secrets-backendThis is a secrets backend for CyberArk CCP (central credential provider)
for the Apache Airflow platform. It will allow one to pull connections and
variables from their CyberArk safes via the CCP.This library has been tested with Airflow 1.10.14.Documentation for CyberArk CCP can be foundhere.Documentation for Airflow secrets backends can be foundhereUsagepip install airflow-cyberark-secrets-backendUpdate yourairflow.cfgwith the following[secrets]
backend = airflow_cyberark_secrets_backend.CyberArkSecretsBackend

backend_kwargs = {"app_id": "/files/var.json", "ccp_url": "/files/conn.json", "safe": "", "verify": "/path/to/ssl/cert.pem" }The backend_kwargs:app_id : The application ID for CCPccp_url : The host URL for CCP AIM, excluding query paramssafe : The secrets safeverify : The SSL cert path to for CCP SSL, can be False for disable, can be env varCYBERARK_SSL, defaultFalseThis library expects and requires your CyberArk response to have the
the following properties (will be mapped mapped to Airflow keys). This
map is a band-aid required from the little configuration CyberArk PAM (11.xx)
allows.AccountDescription : svc_accountApplicationName : schemaAddress : hostComment : extraContent : passwordLogonDomain : loginPort : portAccountDescription : svc_account field is used to fetch password from
rotating secret where the fetched secret is statis, i.e. if you fetchsecret1which is static, if you specify the CCP URL forsecret2which rotates it will
fetch metadata forsecret1and fill in password fromsecret2in its responseDevelopmentPRs welcomed.The following will install in editable mode with all required development tools.pipinstall-e'.[dev]'Please format (black) and lint (pylint) before submitting PR.
Package: airflow-dag
airflow-dagA tool to manage Airflow dags.InstallationYou can usepipto installairflow-dag:$ pip install airflow-dagUsageYou can use thebuildcommand to convert a yaml config to an Airflow dag:$ airflow-dag build -t examples/ -c examples/notebook.yml -o examples/out$ airflow-dag build --help
Usage: airflow-dag build [OPTIONS]

  Convert a yaml config to an Airflow dag.

Options:
  -t, --template-dir TEXT  Path to dag templates
  -c, --config TEXT        Path to dag config
  -o, --output-dir TEXT    Output path
  --help                   Show this message and exit.If a template path is not provided,airflow-dagwill look into thedefault templates.You can define your own dag templates too, and put them in atemplatesdirectory in Airflow's home folder. The dag yaml configs can be placed in aconfigsdirectory in the same home folder, and the output path can then be the Airflow dags folder. The usage will look like:$ airflow-dag build -t airflow/templates -c airflow/configs/dag.yml -o airflow/dagsVersioningairflow-dagusesSemantic Versioning. For the available versions, see the tags on the GitHub repository.LicenseThis project is licensed under the Apache License, see theLICENSEfile for details.
Package: airflow-dag-artifact
Welcome toairflow_dag_artifactDocumentationA lot of serverless AWS Service supports versioning and alias for deployment. It made the blue / green deployment, canary deployment and rolling back super easy.AWS Lambda Versioning and AliasAWS StepFunction Versioning and AliasAWS SageMaker Model Registry VersioningHowever, Airflow DAG does not support this feature yet. This library provides a way to manage Airflow DAG versioning and alias so you can deploy Airflow DAG with confidence.Pleaseread this tutorialto learn how to use this library.It also has nativeAWS MWAAsupport for DAG deployment automation, with the DAG versioning manage, which is not official supported by Apache Airflow. Pleaseread this exampleto learn how to use this library with AWS MWAA.Installairflow_dag_artifactis released on PyPI, so all you need is to:$pipinstallairflow-dag-artifactTo upgrade to latest version:$pipinstall--upgradeairflow-dag-artifact
Package: airflow-dag-deployer
Installationpip install airflow-dag-deployerDeploy dags with commandlineDags can be deployed as zip archive or independent python file prefixed by project name.Deploying dags as a zip archivedeploydag --project=<project_name> --source=<dags_dir> --destination=<airflow_home> --method=zipDeploying dags as a filedeploydag --project=<project_name> --source=<dags_dir> --destination=<airflow_home> --method=fileDeploying dags with config file for different environmentsSet up of different deployment environment (dev/test/prod)Have adeploydag.json(or any filename with json setting) file like this:{"dev":{"project":"testproject","source":"dags","destination":"airflowhome","method":"zip"}}Run command like this:deploydag --config=deploydag.json --env=dev
Package: airflowdaggenerator
What is AirflowDAGGenerator?Dynamically generates Python Airflow DAG file based on given Jinja2
Template and YAML configuration to encourage reusable code. It also
validates the correctness (by checking DAG contains cyclic dependency
between tasks, invalid tasks, invalid arguments, typos etc.) of the
generated DAG automatically by leveraging airflow DagBag, therefore it
ensures the generated DAG is safe to deploy into Airflow.Why is it useful?Most of the time the Data processing DAG pipelines are same except the
parameters like source, target, schedule interval etc. So having a
dynamic DAG generator using a templating language can greatly benefit
when you have to manage a large number of pipelines at enterprise level.
Also it ensures code re-usability and standardizing the DAG, by having a
standardized template. It also improves the maintainability and testing
effort.How is it Implemented?By leveraging the de-facto templating language used in Airflow itself,
that is Jinja2 and the standard YAML configuration to provide the
parameters specific to a use case while generating the DAG.RequirementsPython 3.6 or laterNote: Tested on 3.6, 3.7 and 3.8 python environments, see tox.ini for detailsHow to use this Package?First install the package using:pipinstallairflowdaggeneratorAirflow Dag Generator should now be available as a command line tool to execute. To verify runairflowdaggenerator-hAirflow Dag Generator can also be run as follows:python-mairflowdaggenerator-hSample Usage:If you have installed the package then:airflowdaggenerator\-config_yml_pathpath/to/config_yml_file\-config_yml_file_nameconfig_yml_file\-template_pathpath/to/jinja2_template_file\-template_file_namejinja2_template_file\-dag_pathpath/to/generated_output_dag_py_file\-dag_file_namegenerated_output_dag_py_fileORpython-mairflowdaggenerator\-config_yml_pathpath/to/config_yml_file\-config_yml_file_nameconfig_yml_file\-template_pathpath/to/jinja2_template_file\-template_file_namejinja2_template_file\-dag_pathpath/to/generated_output_dag_py_file\-dag_file_namegenerated_output_dag_py_fileIf you have cloned the project source code then you have sample jinja2 template and YAML configuration file present under
tests/data folder, so you can test the behaviour by opening a terminal window under project root directory and run the
following command:python-mairflowdaggenerator\-config_yml_path./tests/data\-config_yml_file_namedag_properties.yml\-template_path./tests/data\-template_file_namesample_dag_template.py.j2\-dag_path./tests/data/output\-dag_file_nametest_dag.pyAnd you can see that test_dag.py is created under ./tests/data/output folder.TroubleshootingIn case you get some error while generating the dag using this package like (sqlite3.OperationalError)…, then please
execute following command:airflowinitdb
Package: airflow-declarative
Airflow declarative DAGs via YAML.Compatibility:Python 2.7 / 3.5+Airflow 1.10.4+Key FeaturesDeclarative DAGs in plain text YAML helps a lot to understand how DAG will
looks like. Made for humans, not programmers.It makes extremely hard to turn your DAGs into code mess. Even if you make
complicated YAMLs generator the result would be readable for humans.No more guilty about coupling business logic with task management system
(Airflow). They now could coexists separated.Static analysis becomes a trivial task.It’s a good abstraction to create your own scheduler/worker compatible with
original Airflow one.ExamplesChecktests/dagsdirectory for example of DAGs which will works and which
won’t. Usesrc/airflow_declarative/schema.pymodule for the reference about
YAML file schema. It should be self descriptive.Don’t be shy to experiment:trafaret-configwill help you to understand
what had gone wrong and why and where.UsageWe provide support for two installation options:As a complementary side package for the upstream Airflow.As a built-in Airflow functionality using patches for Airflow.Upstream AirflowThe idea is to put a Python script to thedags_folderwhich would
load the declarative dags via airflow_declarative. More details:Installation using Upstream Airflow.importosimportairflow_declarative# Assuming that the yaml dags are located in the same directory# as this Python module:root=os.path.dirname(__file__)dags_list=[airflow_declarative.from_path(os.path.join(root,item))foriteminos.listdir(root)ifitem.endswith((".yml",".yaml"))]globals().update({dag.dag_id:dagfordagsindags_listfordagindags})Patched AirflowWe provide ready to use patches in thepatchesdirectory. To use them
you will need to apply a patch to a corresponding Airflow version and then
build it yourself. More details:Installation using Patched Airflow.
Package: airflow-diagrams
airflow-diagramsAuto-generated Diagrams from Airflow DAGs. 🔮 🪄This project aims to easily visualise yourAirflowDAGs on service level
from providers like AWS, GCP, Azure, etc. viadiagrams.BeforeAfter🚀 Get startedTo install it fromPyPIrun:pip install airflow-diagramsNOTE:Make sure you haveGraphvizinstalled.Then just call it like this:Examples of generated diagrams can be found in theexamplesdirectory.🤔 How it Worksℹ️ It connects, by using the officialApache Airflow Python Client, to your Airflow installation to retrieve all DAGs (in case you don't specify anydag_id) and all Tasks for the DAG(s).🪄 It processes every DAG and its Tasks and 🔮 tries to find a diagram node for every DAGs task, by usingFuzzy String Matching, that matches the most. If you are unhappy about the match you can also provide amapping.ymlfile to statically map from Airflow task to diagram node.🎨 It renders the results into a python file which can then be executed to retrieve the rendered diagram. 🎉❤️ ContributingContributions are very welcome. Please go ahead and raise an issue if you have one or open a PR. Thank you.
Package: airflow-ditto
DittoDitto is a framework which allows you to do transformations to an Airflow DAG, to convert it into another DAG which is flow-isomorphic with the original DAG. i.e. it will orchestrate a flow of operators which yields the same results, but was just transformed to run in another environment or platform. The framework was built to transform EMR DAGs to run on Azure HDInsight, but you can extend the rich API for any other kind of transformation. In fact you can transform DAGs such that the result is not isomorphic too if you want (although at that point you're better off writing a whole new DAG).The purpose of the framework is to allow you to maintain one codebase and be able to run your airflow DAGs on different execution environments (e.g. on different clouds, or even different container frameworks - spark on YARN vs kubernetes). It is not meant for a one-time transformation, but for continuous and parallel DAG deployments, although you can use it for that purpose too.At the heart, Ditto is a graph manipulation library, which extendable APIs for the actual transformation logic. It does come with out of the box support for EMR to HDInsight transformation though.Installationpip install airflow-dittoA quick exampleDitto is created for conveniently transforming a large number of DAGs which follow a similar pattern quickly. Here's how easy it is to use Ditto:ditto=ditto.AirflowDagTransformer(DAG(dag_id='transformed_dag',default_args=DEFAULT_DAG_ARGS),transformer_resolvers=[ClassTransformerResolver({SlackAPIOperator:TestTransformer1,S3CopyObjectOperator:TestTransformer2,BranchPythonOperator:TestTransformer3,PythonOperator:TestTransformer4})])new_dag=ditto.transform(original_dag)You can put the above call in any python file which is visible to airflow and the resultant dag loads up thanks to how airflow's dagbag finds DAGs.Source DAG(airflow view)Transformed DAGRead the detailed documentationhere
Package: airflow-docker
airflow-dockerDescriptionAn opinionated implementation of exclusively using airflow
DockerOperators for all Operators.Default Operatorfromairflow_docker.operatorimportOperatortask=Operator(image='some-image:latest',...)Default Sensorfromairflow_docker.operatorimportSensorsensor=Sensor(image='some-image:latest',...)Task Codefromairflow_docker_helperimportclientclient.sensor(True)Branch OperatorDag Taskfromairflow_docker.operatorimportBranchOperatorbranching_task=BranchOperator(image='some-image:latest',...)Task Codefromairflow_docker_helperimportclientclient.branch_to_tasks(['task1','task2'])Short Circuit OperatorDag Taskfromairflow_docker.operatorimportShortCircuitOperatorshort_circuit=ShortCircuitOperator(image='some-image:latest',...)Task Codefromairflow_docker_helperimportclientclient.short_circuit()# This task will short circuit if this function gets calledContext UsageDag Taskfromairflow_docker.operatorimportOperatortask=Operator(image='some-image:latest',provide_context=True,...)Task Codefromairflow_docker_helperimportclientcontext=client.context()ConfigurationThe following operator defaults can be set under theairflowdockernamespace:force_pull (boolean true/false)auto_remove (boolean true/false)network_modeFor example, to setforce_pullto False by default set the following
environment variable like so:exportAIRFLOW__AIRFLOWDOCKER__FORCE_PULL=falsePluginThis package works as an airflow plugin as well. When installed and
running airflow, dags can import like sofromairflow.{type,like"operators","sensors"}.{namespecificedinsidethepluginclass}import*i.e.fromairflow.operators.airflow_dockerimportOperatorTestsWe also ship anairflowdocker/testerimage to verify the integrity of
your DAG definitions before committing them.One can run the tests against your own dags like so:dockerrun-it-v"${pwd}/dags:/airflow/dags"airflowdocker/testeror else see theairflow-docker-composeproject which ships with atestsubcommand for precisely this purpose.
Package: airflow-docker-compose
airflow-docker-composeDescriptionA reasonably light wrapper arounddocker-composeto make it simple to start a local
airflow instance in docker.Usageairflow-docker-compose--help
airflow-docker-composeupConfigurationNote, this library assumes thedocker-composeutility is available in your path.In order to use this tool, you should have a localdagsfolder containing your dags.
You should also have apyproject.tomlfile which minimally looks like[tool.airflow-docker-compose]docker-network='network-name'In order to set airflow configuration, you can use theairflow-environment-variableskey.
This allows you to set anyairflow.cfgvariables like so:[tool.airflow-docker-compose]airflow-environment-variables={AIRWFLOW_WORKER_COUNT=4AIRFLOW__AIRFLOWDOCKER__FORCE_PULL='false'}
Package: airflow-docker-helper
Airflow Docker HelperDescriptionA light sdk to be used by the operators in airflow-docker and in task code to participate in host/container communication.Installationpipinstallairflow-docker-helperUsageSensorfromairflow_docker_helperimportclientifsensed:client.sensor(True)else:client.sensor(False)Short Circuitfromairflow_docker_helperimportclientifshould_short_circuit:client.short_circuit()BranchingYou can pass a list...fromairflow_docker_helperimportclientbranch_to_task_ids=['foo','bar']client.branch_to_tasks(branch_to_task_ids)... or a string.fromairflow_docker_helperimportclientclient.branch_to_tasks('some-other-task')TestingThis library ships with a test client that mocks out all io and filesystem calls.  This client
also provides all of the relevant mocked out files to allow for assertions around the io.Some higher level assertions are provided. These assertions are based on the lower level file mocks.fromairflow_docker_helper.testingimporttest_clientclient=test_client()client.assert_not_short_circuited()# Passesclient.short_circuit()client.assert_short_circuited()# Passesclient.sensor(True)client.assert_sensor_called_with(True)# Passesclient.assert_sensor_called_with(False)# Failsclient.assert_branched_to_tasks([])# Passesclient.branch_to_tasks(['foo','bar'])client.assert_branched_to_tasks(['bar','foo'])# PassesFor power users, the mocks may be used directly:>>>fromairflow_docker_helper.testingimporttest_client>>>client=test_client()>>>client.branch_to_tasks(['foo','bar'])>>>client._mock_branch_to_tasks_file.mock_calls[call('./__AIRFLOW_META__/branch_operator.txt','wb'),call().__enter__(),call().write(b'["foo", "bar"]'),call().__exit__(None,None,None)]>>>client.short_circuit()>>>client._mock_short_circuit_file.mock_calls[call('./__AIRFLOW_META__/short_circuit.txt','wb'),call().__enter__(),call().write(b'false'),call().__exit__(None,None,None)]>>>client.sensor(True)>>>client._mock_sensor_file.mock_calls[call('./__AIRFLOW_META__/sensor.txt','wb'),call().__enter__(),call().write(b'true'),call().__exit__(None,None,None)]