Saravanan666/pypi-org huggingface dataset:
Package: airflow-duckdb
Airflow DuckDB on KubernetesDuckDBis an in-memory analytical database to run analytical queries on large data sets.Apache Airflowis an open-source platform for developing, scheduling, and monitoring
batch-oriented workflows.Apache Airflow is not an ETL tool, but more of a workflow scheduler that can be used to schedule and monitor ETL jobs.
Airflow users create DAGs to schedule Spark, Hive, Athena, Trino, BigQuery, and other ETL jobs to process their data.By using DuckDB with Airflow, the users can run analytical queries on local or remote large data sets and store the
results without the need to use these ETL tools.To use DuckDB with Airflow, the users can use the PythonOperator with the DuckDB Python library, the BashOperator with
the DuckDB CLI, or one of the available Airflow operators that support DuckDB (e.g.airflow-provider-duckdbdeveloped by Astronomer). All of these
operators will be running in the worker pod and limited by its resources, for that reason, some users use the
Kubernetes Executor to run the tasks in a dedicated Kubernetes pod to request more resources when needed.Setting up Kubernetes Executor could be a bit challenging for some users, especially maintaining the workers docker
image. This project provides an alternative solution to run DuckDB with Airflow using the KubernetesPodOperator.How to useThe developed operator is completely based on the KubernetesPodOperator, so it needs cncf-kubernetes provider to be
installed in the Airflow environment (preferably the latest version to profit from all the features).Install the packageTo use the operator, you need to install the package in your Airflow environment. You can install the package using pip:pipinstallairflow-duckdbUse the operatorThe operators supports all the parameters of the KubernetesPodOperator, and it has some additional parameters to
simplify the usage of DuckDB.Here is an example of how to use the operator:withDAG("duckdb_dag",...)asdag:DuckDBPodOperator(task_id="duckdb_task",query="SELECT MAX(col1) AS  FROM READ_PARQUET('s3://my_bucket/data.parquet');",do_xcom_push=True,s3_fs_config=S3FSConfig(access_key_id="{{ conn.duckdb_s3.login }}",secret_access_key="{{ conn.duckdb_s3.password }}",),container_resources=k8s.V1ResourceRequirements(requests={"cpu":"1","memory":"8Gi"},limits={"cpu":"1","memory":"8Gi"},),)FeaturesThe current version of the operator supports the following features:Running one or more DuckDB queries in a Kubernetes podConfiguring the pod resources (requests and limits) to run the queriesConfiguring the S3 credentials securely with a Kubernetes secret to read and write data from/to S3
(AWS S3, MinIO or GCS with S3 compatibility)Using Jinja templating to configure the queryLoading the queries from a filePushing the query result to XComThe project also provides a Docker image with DuckDB CLI and some extensions to use it with Airflow.
Package: airflow-exporter
Airflow prometheus exporterExposes dag and task based metrics from Airflow to a Prometheus compatible endpoint.Compatibility with Airflow versions>=2.0Current version is compatible with Airflow 2.0+<=1.10.14, >=1.10.3Version v1.3.2 is compatibleNote: Airflow 1.10.14 with Python 3.8 usersYou should installimportlib-metadatapackage in order for plugin to be loaded. See#85for details.<1.10.3Versionv0.5.4is compatibleInstallpipinstallairflow-exporterThat's it. You're done.Exporting extra labels to PrometheusIt is possible to add extra labels to DAG-related metrics by providinglabelsdict to DAGparams.Exampledag=DAG('dummy_dag',schedule_interval=timedelta(hours=5),default_args=default_args,catchup=False,params={'labels':{'env':'test'}})Labelenvwith valuetestwill be added to all metrics related todummy_dag:airflow_dag_status{dag_id="dummy_dag",env="test",owner="owner",status="running"} 12.0MetricsMetrics will be available athttp://<your_airflow_host_and_port>/admin/metrics/airflow_task_statusLabels:dag_idtask_idownerstatusValue: number of tasks in a specific status.airflow_dag_statusLabels:dag_idownerstatusValue: number of dags in a specific status.airflow_dag_run_durationLabels:dag_id: unique identifier for a given DAGValue: duration in seconds of the longest DAG Run for given DAG. This metric
is not available for DAGs that have already finished.airflow_dag_last_statusLabels:dag_idownerstatusValue: 0 or 1 depending on wherever the current state of eachdag_idisstatus.LicenseDistributed under the BSD license. SeeLICENSEfor more
information.
Package: airflow-fs
airflow-fsairflow-fs is Python package that provides hooks and operators for manipulating
files across a variety of file systems using Apache Airflow.Why airflow-fs?Airflow-fs implements a single interface for different file system hooks, in contrast
to Airflows builtin file system hooks/operators. This approach allows us to interact
with files independently of the underlying file system, using a common set of operators
for performing general operations such as copying and deleting files.Currently, airflow-fs supports the following file systems: local, FTP, HDFS, S3 and SFTP.
Support for additional file systems can be added by implementing an additional file
system hook that adheres to the required hook interface. See the documentation for more
details.DocumentationDetailed documentation is available at:https://jrderuiter.github.io/airflow-fs.LicenseThis software is freely available under the MIT license.HistoryVersion 0.1.0Initial release supporting local, FTP, HDFS, S3 and SFTP file systems.
Package: airflow-gcpsecretmanager-adapter
airflow-gcpsecretmanager-adapterOverviewThis repository contains a Python package to allow Airflow variables that do not align with GCP Secret Manager naming convention to be used with the secrets backend. GCP Secret Manager has constraints on the characters that may be used when naming a secret (alphanumerics, hyphens and underscores only), however variables in Airflow are not restricted in any way. Therefore in some cases it may be necessary to transform the name of the Airflow variable to a format acceptable for Secret Manager to use. For example the Airflow variable name:global::myvaris not permitted as a GCP secret name, so this must instead be translated to:global-myvarwhichispermitted.This package can be installed as an alternative secrets backend within Airflow, as described here:https://airflow.apache.org/docs/apache-airflow/stable/security/secrets/secrets-backend/index.html#roll-your-own-secrets-backendInstallationpipinstallairflow-gcpsecretmanager-adapterCurrent replacementsSource charactersReplaced with::-:-DeploymentFollow the instructions here:https://packaging.python.org/en/latest/tutorials/packaging-projects/to build and deploy a new version of the adapter library.
Package: airflow-gitlab-webhook
Airflow Gitlab Webhook PluginDescriptionA plugin forApache Airflowthat exposes REST endpoint forGitlab Webhooks.System RequirementsAirflow Versions1.10.2 or newerDeployment InstructionsInstall the pluginpip install airflow-gitlab-webhookUpdate the airflow.cfg configuration file adding thegitlab_pluginsection[gitlab_plugin]

 repository_url = http://example.com/mike/diaspora.git
 token = 62b32508-b1ad-44d2-97d1-80021a8d7576
 dag = tutorial

 (Optional) Configure other repositories
 repository_url1 = http://example.com/bla.git
 token1 = my-secret
 dag1 = git_updaterepository_url = Gitlab repository URLtoken = Optional Secure Tokendag = DAG to be run when the push even is receivedConfigure Gitlab Webook (push event) for the repositoryhttps://docs.gitlab.com/ee/user/project/integrations/webhooks.htmlRestart the Airflow Web ServerEndpointspushGitlab Push EventPOST - https://{HOST}:{PORT}/webhooks/gitlab/push
Package: airflow-google-ads-api-report-fetcher
Using gaarf in AirflowIf you want to use Apache Airflow to run any gaarf-based projects you can useairflow-google-ads-api-report-fetcherpackage.Install it withpip install airflow-google-ads-api-report-fetcher-
it will makeairflow_gaarflibrary available.Install the latest development version withpip install -e git+https://github.com/google/ads-api-report-fetcher.git#egg=airflow-google-ads-api-report-fetcher\&subdirectory=py/airflow_gaarfThe library comes with two operators -GaarfOperatorandGaarfBqOperatorwhich can
be used to simplify executinggoogle_ads_queriesandbq_queriesrespectively.SetupConnectionsTemplate pipeline expects two type of connections - go toAdmin - Connections,
add new connection (typeGeneric) and inExtraadd the values specified below:google_ads_default{"google_ads_client":
  	{"developer_token": "",
  	"client_id": "",
  	"client_secret": "",
  	"refresh_token": "",
  	"login_customer_id": "",
  	"client_customer_id": "",
  	"use_proto_plus": "true"
  	}
  }gcp_conn_id{"cloud":
  	{"project_id": "your-project"}
}ExamplesOnce the above connections were setup you may proceed to configuring DAG.examplesfolder contains several DAGs you might find useful:01_gaarf_console_reader_console_writer.py- simple DAG which consist of a singleGaarfOperatorwhat fetches data from an inline query and outputs results to the console.02_gaarf_file_reader_csv_writer.py- DAG that reads query from a file (can be local or remote storage) and save results to CSV.03_gaarf_read_solution_directory- DAG that reads queries from a directory with queries and for reach query builds its own task.
Package: airflow-gpg-plugin
Airflow GPG PluginAirflow plugin with hooks and operators to work with GPG encryption and decryption.InstallationUse the package managerpipto install foobar.pipinstallairflow-gpg-pluginUsageAdd an airflow connection from shell.loginis the email address in the GPG key.passwordis the passphrase of the GPG key.airflowconnectionsadd'gpg_default_conn'\--conn-type'gpg'\--conn-login'gpgexamplepassphrase@example.com'\--conn-password'gpgexamplepassphrase'\--conn-host''\--conn-port''\--conn-extra'{"key_file": "tests/resources/gpgexamplepassphrase.asc"}'Using operators to encrypt and decrypt files.importosfromdatetimeimportdatetimefromairflowimportDAGfromairflow_gpg_plugin.operators.gpg_decrypt_file_operatorimportGPGDecryptFileOperatorfromairflow_gpg_plugin.operators.gpg_encrypt_file_operatorimportGPGEncryptFileOperatorgpg_conn_id="gpg_default_conn"dag=DAG(dag_id="gpg_example",start_date=datetime(2021,1,1),schedule_interval=None)encrypt=GPGEncryptFileOperator(task_id="gpg_encrypt",dag=dag,conn_id=gpg_conn_id,input_file_path=os.curdir+"/README.md",output_file_path=os.curdir+"/README.md.gpg")decrypt=GPGDecryptFileOperator(task_id="gpg_decrypt",dag=dag,conn_id=gpg_conn_id,input_file_path=os.curdir+"/README.md.gpg",output_file_path=os.curdir+"/README.md.txt")encrypt>>decryptContributingPull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.Please make sure to update tests as appropriate.LicenseMIT
Package: airflow-grpc
Airflow Grpc OperatorFork on Airflow 2.0Fix Airflow 1.10.x use grpc operatorRequirements:apache-airflow
grpcio
protobufHow to install:pip install airflow-grpcHow to use:airflow 1.10.xfrom airflow_grpc.grpc_operator import GrpcOperator
args = {
    'owner': 'Airflow',
    'start_date': days_ago(1),
}

dag = DAG(
    dag_id='dag_id',
    default_args=args,
    schedule_interval=None
)

def callback(response: Any, **context):
    return response

run_this = GrpcOperator(task_id='task_id',
                        dag=dag,
                        grpc_conn_id='your_grpc_connection_id_on_admin_connections',
                        stub_class=YOUR_GRPC_STUB_CLASS,
                        call_func='your_grpc_stub_function',
                        request_data_func=YOUR_GRPC_MESSAGE_FOR_REQUEST,
                        response_callback=YOUR_RESPOSNE_METHOD,
                        xcom_task_id='XCOM_TASK_ID',
                        data=YOUR_REQUEST_DATA_DICT
)airflow 2.0.0First must implement the stub_class methodimport xxx_pb2_grpc, xxx_pb2
from utils.grpc_hook import BaseGrpcHook


class xxxGrpcHook(BaseGrpcHook):
    def stub_class(self, channel):
        return xxx_pb2_grpc.xxxServiceStub(channel)Use the implemented classxxxGrpcHook().run('MethodName', {'request': xxx_pb2.xxxRequestMethod(**data_dict)})
Package: airflow-kaldea
airflow-kaldeaThis is a collection ofAirflowoperators to provide integration withKaldea.fromairflow.modelsimportDAGfromairflow_kaldea.operators.kaldea_job_operatorimportKaldeaJobOperatordefault_args={}dag=DAG(dag_id='data_dag',default_args=default_args,schedule_interval='0 * * * *',)kaldea_job=KaldeaJobOperator(task_id='kaldea_job',kaldea_job_id='kaldea_job_id',kaldea_task_id='kaldea_task_id',dag=dag,)InstallationInstall from PyPI:pipinstallairflow-kaldea
Package: airflow-kdb-provider
Airflow KDB ProviderA lightweight KDB provider for Apache Airflow, featuring theKDBAirflowOperator. This provider allows for seamless integration between Airflow and KDB+/q, making it easier to automate data pipelines that involve KDB+/q.InstallationYou can install the Airflow KDB Provider package from PyPI using the following command:pipinstallairflow-kdb-providerUsageTo use the KDBAirflowOperator in your Airflow DAG, you must first import it and create an instance of the operator. Here is an example:from airflow_kdb_provider.operators.kdb_operator import KDBOperatorkdb_operator=KDBOperator(task_id='run_kdb_script',command='/path/to/kdb_script.q',params={'param1':'value1','param2':'value2'},conn_id='kdb_conn',dag=dag)In this example, we create an instance of the KDBOperator and specify the following parameters:task_id: the task ID for this operator
command: the path to the KDB+/q script that we want to execute
params: a dictionary of parameters that will be passed to the KDB+/q script as command-line arguments
conn_id: the connection ID for the KDB+/q server that we want to use (this should be defined in Airflow's Connections interface)
dag: the DAG that this operator belongs to
Once you have created an instance of the KDBOperator, you can add it to your DAG like any other Airflow operator:some_other_operator>>kdb_operator>>some_other_operator2In this example, we have added the kdb_operator to our DAG and specified that it should be executed after some_other_operator and before some_other_operator2.
Package: airflow-kube-pvc-operator
Airflow Kubernetes PVC OperatorAirflow operator for kubernetes PVC typeAspirations:make an operator to take in a yaml file describing a pvc object and have airflow manage its life cycleshould be able to plug in with kube job operatorwould be cool to modify the UI to show the pvc is bound to the Jobthrow a warning if resource quota for storage limit in namespace is not set, Airflow should help onboard people to kube 'best practices'NotesNot to be used yet
Package: airflow-mailgun-email
airflow-mailgun-emailAirflow Email Backend to send email via Mailgun APIHow to configureInairflow.cfg[email]
email_backend = airflow_mailgun_email.email_mailgun.send_email_mailgun

[mailgun]
domain_name = <your mailgun email domain name>
api_password = <api key>How to build in Dev?pip install --editable .
Package: airflow-metrics-gbq
Airflow Metrics to BigQuerySends airflow metrics to BigqueryInstallationpipinstallairflow-metrics-gbqUsageActivate statsd metrics inairflow.cfg[metrics]statsd_on=Truestatsd_host=localhoststatsd_port=8125statsd_prefix=airflowRestart the webserver and the schedulersystemctlrestartairflow-webserver.service
systemctlrestartairflow-scheduler.serviceCheck that airflow is sending out metrics:nc-l-ulocalhost8125Install this packageCreate required tables (counters, gauges and timers), an example is sharedhereCreate materialized views which refresh when the base table changes, as describehereCreate a simple python scriptmonitor.pyto provide configuration:fromairflow_metrics_gbq.metricsimportAirflowMonitorif__name__=='__main__':monitor=AirflowMonitor(host="localhost",# Statsd host (airflow.cfg)port=8125,# Statsd port (airflow.cfg)gcp_credentials="path/to/service/account.json",dataset_id="monitoring",# dataset where the monitoring tables arecounts_table="counts",# counters tablelast_table="last",# gauges tabletimers_table="timers"# timers table)monitor.run()Run the program, ideally in the background to start sending metrics to BigQuery:pythonmonitor.py&The logs can be viewed in the GCP console under theairflow_monitoringapp_name in Google Cloud Logging.Future releasesIncrease test coverage (unit and integration tests)Add proper typing and mypy support and checksProvide more configurable optionsProvide better documentation
Package: airflow-multi-dagrun
Multi dag runThis plugin contains operators for triggering a DAG run multiple times
and you can dynamically specify how many DAG run instances create.It can be useful when you have to handle a big data and you want to split it
into chunks and run multiple instances of the same task in parallel.When you see a lot launched target DAGs you can set up more workers.
So this makes it pretty easy to scale.Installpipinstallairflow_multi_dagrunExampleCode for scheduling dagsimportdatetimeasdtfromairflowimportDAGfromairflow_multi_dagrun.operatorsimportTriggerMultiDagRunOperatordefgenerate_dag_run():foriinrange(100):yield{'index':i}default_args={'owner':'airflow','start_date':dt.datetime(2015,6,1),}dag=DAG('reindex_scheduler',schedule_interval=None,default_args=default_args)ran_dags=TriggerMultiDagRunOperator(task_id='gen_target_dag_run',dag=dag,trigger_dag_id='example_target_dag',python_callable=generate_dag_run,)This code will schedule dag with idexample_target_dag100 times and pass payload to it.Example of triggered dag:dag=DAG(dag_id='example_target_dag',schedule_interval=None,default_args={'start_date':datetime.utcnow(),'owner':'airflow'},)defrun_this_func(dag_run,**kwargs):print("Chunk received:{}".format(dag_run.conf['index']))chunk_handler=PythonOperator(task_id='chunk_handler',provide_context=True,python_callable=run_this_func,dag=dag)Run exampleThere is docker-compose config, so it requires docker to be installed:docker,docker-composemake init- create dbmake add-admin- createadminuser (is asks a password)make web- start docker containers, run airflow webservermake scheduler- start docker containers, run airflow schedulermake downwill stop and remove docker containersContributionsIf you have found a bug or have some idea for improvement feel free to create an issue
or pull request.LicenseApache 2.0
Package: airflownetwork
AirflowNetworkTable of ContentsAboutInstallationChecksLicenseAboutThis is a small library of functions and classes to examine EnergyPlus AirflowNetwork models. A driver program is provided to analyze models in the epJSON format. To summarize the model contents:airflownetwork summarize my_model.epJSONTo create a graph of the model in the DOT format:airflownetwork graph my_model.epJSONTo generate an audit of the model:airflownetwork audit my_model.epJSONFurther help is available on the command line:airflownetwork --helpInstallationpip install airflownetworkChecksThe script checks for a number of issues that may cause an AirflowNetwork to function poorly. These include:Link CountsModels with a large number of links (particularly those with too many links between adjacent zones) may model the building correctly and the simulation results may be correct, but the model performance may be quite poor. A situation that has been observed in user models is the use of individual window elements to model each and every window in a buildings. This may be correct, but as the number of windows increases the performance of the model will suffer, and the performance hit may be avoidable if windows the beahve the same are lumped together. For example, if 10 windows connect a zone to the ambient, but all 10 windows experience the same wind pressure and temperature difference, a single winwdow that represents all 10 will be sufficient and eliminate 9 of the 10 calculations required.Theauditcommand counts the links between zones and flags those that are considered excessive.ConnectednessModels in which there are zones that are "isolated" (i.e., are not connected to the rest of the model via linkages) have been known to be sensitive to convergence issues. For the most part, the solution procedure can handle multiple isolated subnetworks in a single matrix solution, issues that are encountered with these models can be hard to diagnose. The easiest fix to connect together subnetworks is to add one or more linkages very high flow resistance (e.g., a crack element with a very small flow coefficient).Theauditcommand checks that the multizone network (just with surfaces) and the full network (multizone + dsitribution) are fully connected. Models with intrazone features are not currently supported.Licenseairflownetworkis distributed under the terms of theBSD-3-Clauselicense.
Package: airflow-notify-sns
Publish Airflow Notification to a SNS TopicThis package adds a callback function to use in failures to DAGs and Tasks in a Airflow project.Installationpipinstallairflow-notify-snsUsagefromdatetimeimporttimedelta# Airflow native imports to create a DAGfromairflowimportDAG,utilsfromairflow.operators.bash_operatorimportBashOperator# Here is function importfromairflow_notify_snsimportairflow_notify_sns# Dag Definitiondag=DAG(dag_id='test_dag',default_args={'owner':'airflow','depends_on_past':False,'start_date':utils.dates.days_ago(1),'retries':3,'retry_delay':timedelta(minutes=5)},schedule_interval="@daily",dagrun_timeout=timedelta(minutes=60),sla_miss_callback=airflow_notify_sns,on_failure_callback=airflow_notify_sns)# Add your tasks heret=BashOperator(dag=dag,task_id='test_env',bash_command='/tmp/test.sh',env={'EXECUTION_DATE':'{{ ds }}'},on_failure_callback=airflow_notify_sns)When DAG or tasks ends in error, a notification will be send to a SNS Topic using AWS default connection (aws_default).Required VariableThis module will try to find a variable namedairflow_notify_sns_arnin your Airflow environment, containing SNS Topic ARN where message will be published to.If variable is not found, function will abort execution with no error.
Package: airflow-plugin-config-storage
airflow-plugin-config-storageInject connections into the airflow database from configurationQuickstartBasic$pipinstallairflow-plugin-config-storage
$exportAIRFLOW_CONN_POSTGRES_MASTER=postgres://username:password@my.db.example.com/my-schema
$load-airflow-conf-env-var
$airflowwebserverCommonCLI Commandsdelete-all-airflow-connnectionsRemoves all the connections from Airflow. Used to clean out
the default connections.Environment VariablesStructureThe Environment Variables to read from by default are the same
as those defined in theAirflow documentation.CLI Commandsload-airflow-conf-env-varTakes a single optional argument--env-var-prefix ENV_VAR_PREFIXto override the Environment Variable prefix. Default isAIRLFOW_CONN_.AWSNOTE: Not yet implemented, these are proposalsSSMHierarchy/${user_prefix}/airflow/connections/[
    {
        "conn_type": "s3",

    },
    {}
]/${user_prefix}/airflow/