Saravanan666/pypi-org huggingface dataset:
Package: airflow-prometheus-exporter
Airflow Prometheus ExporterThe Airflow Prometheus Exporter exposes various metrics about the Scheduler, DAGs and Tasks which helps improve the observability of an Airflow cluster.The exporter is based on thisprometheus exporter for Airflow.RequirementsThe plugin has been tested with:Airflow >= 1.10.4Python 3.6+The scheduler metrics assume that there is a DAG namedcanary_dag. In our setup, thecanary_dagis a DAG which has a tasks which perform very simple actions such as establishing database connections. This DAG is used to test the uptime of the Airflow scheduler itself.InstallationThe exporter can be installed as an Airflow Plugin using:pip install airflow-prometheus-exporterThis should ideally be installed in your Airflow virtualenv.MetricsMetrics will be available athttp://<your_airflow_host_and_port>/admin/metrics/Task Specific Metricsairflow_task_statusNumber of tasks with a specific status.All the possible states are listedhere.airflow_task_durationDuration of successful tasks in seconds.airflow_task_fail_countNumber of times a particular task has failed.airflow_xcom_paramvalue of configurable parameter in xcom tablexcom fields is deserialized as a dictionary and if key is found for a paticular task-id, the value is reported as a guageAdd task / key combinations in config.yaml:xcom_params:-task_id:abckey:count-task_id:defkey:errorsa task_id of 'all' will match against all airflow tasks:xcom_params:
 -
    task_id: all
    key: countDag Specific Metricsairflow_dag_statusNumber of DAGs with a specific status.All the possible states are listedhereairflow_dag_run_durationDuration of successful DagRun in seconds.Scheduler Metricsairflow_dag_scheduler_delayScheduling delay for a DAG Run in seconds. This metric assumes there is acanary_dag.The scheduling delay is measured as the delay between when a DAG is marked asSCHEDULEDand when it actually startsRUNNING.airflow_task_scheduler_delayScheduling delay for a Task in seconds. This metric assumes there is acanary_dag.airflow_num_queued_tasksNumber of tasks in theQUEUEDstate at any given instance.
Package: airflow-provider-aerospike
Aerospike Provider for Apache AirflowInstallationrequirements:python3.8.0+aerospike14.0.0+apache-airflow2.2.0+You can install this package as:pipinstallairflow-provider-aerospikeConfigurationIn the Airflow interface, configure aConnectionfor Aerospike. Configure the following fields:Conn Id:aerospike_conn_idConn Type:AerospikePort: Aerospike cluster port (usually at 3000)Host: Cluster node address (The client will learn about the other nodes in the cluster from the seed node)Operatorscurrently, the provider supports simple operations such as Fetching single or multiple keys and Creating/Updating keys.Sensorscurrently, the provider supports simple methods such as checking if single or multiple keys exist.
Package: airflow-provider-alembic
Alembic Airflow ProviderAn Airflow Provider to use Alembic to manage database migrationsRead more hereSetupLocallyInstall the Alembic CLI withpip install alembicIn AirflowAddairflow-provider-alembicto yourrequirements.txtor equivalentUsageCreate the required files for Alembic in either yourdagsfolder or theincludefoldermkdirdags/migrationscddags/migrations
alembicinit.Create a revisionalembicrevision-m"My Database Revision"Edit the revision - adding, modifying, or removing objects as needed...defupgrade():# Use ORM to create objectsop.create_table('foo',sa.Column('id',sa.Integer,primary_key=True),sa.Column('name',sa.String(50),nullable=False),sa.Column('description',sa.Unicode(200)),)# Or run raw SQLop.execute("SELECT 1;")defdowngrade():# Specify the opposite of your upgrade, to rollbackop.drop_table('account')Add a Connection to Airflow
For demo purposes, we will add an in-memory SQLite3 Connection namedsqlitevia our.envfile:AIRFLOW_CONN_SQLITE="sqlite:///:memory:"Restart (or start) your project withastro dev restartAdd a DAG, to run your revision. Because this has@once, it will run as soon as the DAG is turned on. Future runs for future revisions will need to be triggered.importosfromdatetimeimportdatetimefromairflow.modelsimportDAGfromairflow.models.paramimportParamfromairflow_provider_alembic.operators.alembicimportAlembicOperatorwithDAG("example_alembic",schedule="@once",# also consider "None"start_date=datetime(1970,1,1),params={"command":Param("upgrade"),"revision":Param("head")})asdag:AlembicOperator(task_id="alembic_op",conn_id="sqlite",command="{{ params.command }}",revision="{{ params.revision }}",script_location="/usr/local/airflow/dags/migrations",)Extra CapabilitiesYou can utilize any of the Alembic commands in theAlembicOperator- such asdowngradeTheAlembicHookhas methods to run anyalembic commands
Package: airflow-provider-appops
AppOpsApp 投放运营工具集AdmobHooksfromappops.hooks.admobimportAdmobHookh=AdmobHook()h.accountsOut[3]:['pub-xxxx']h.mediationreport_json(query_body={"report_spec":{"dateRange":{"startDate":{"year":2021,"month":8,"day":30},"endDate":{"year":2021,"month":8,"day":30}},"dimensions":["DATE","FORMAT"],#"metrics":["ESTIMATED_EARNINGS","AD_REQUESTS","MATCH_RATE","MATCHED_REQUESTS","IMPRESSIONS","CLICKS"],"sortConditions":[{"metric":"ESTIMATED_EARNINGS","order":"ASCENDING"}],"localizationSettings":{"currencyCode":"USD","languageCode":"en-US"}}},accounts=None,# using accounts from connection)OperatorignoreAds
Package: airflow-provider-bigquery-reservation
Airflow BigQuery Reservation ProviderWarningThis package is a pre-released of the official apache-airflow-providers-google package. All of these operators will be integrated to the official package, soon.This repository provides an Apache Airflow provider based onBigQuery Reservation API.Airflow OperatorsBigQueryReservationCreateOperator: Buy BigQuery slots (commitments) and assign them to a GCP project (reserve and assign).BigQueryReservationDeleteOperator: Delete BigQuery commitments and remove associated ressources (rservation and assignment).BigQueryBiEngineReservationCreateOperator: Create or Update a BI engine reservation.BigQueryBiEngineReservationDeleteOperator: Delete or Update a BI engine reservation.You could find DAG sampleshere.RequirementsAGoogle Cloud connectionhas to be defined.
By default, all hooks and operators usegoogle_cloud_default.This connection requires the following roles on the Google Cloud project(s) used in these operators:BigQuery Resource AdminBigQuery Job User-Required forBigQueryReservationCreateOperatorbecause of the reservation attachment check.Defining a new dedicated connection and custom GCP role could be good practices to respect the principle of least privilege.How to installpipinstall--userairflow-provider-bigquery-reservation
Package: airflow-provider-couchbase
Packageapache-airflow-providers-couchbaseRelease:1.0.0couchbaseProvider packageThis is a provider package forcouchbaseprovider. All classes for this provider package
are inairflow.providers.couchbasepython package.InstallationYou can install this package on top of an existing Airflow 2 installation (seeRequirementsbelow
for the minimum Airflow version supported) viapip install apache-airflow-providers-couchbaseThe package supports the following python versions:>=3.7RequirementsPIP packageVersion requiredapache-airflow>=2.3.0couchbase>=4.0.0Changelog1.0.0Initial version of the provider.
Package: airflow-provider-cube
Airflow Cube ProviderCube is the semantic layer for building data applications. It helps
data engineers and application developers access data from modern data
stores, organize it into consistent definitions, and deliver it to
every application.Connection requirementsCubeHookusesGenericconnection. So, in order to create a
connection to a Cube you will need to create an AirflowGenericconnection and specify the following required properties:host- your Cube's host (must contain schema, address and port);password- your Cube'sCUBEJS_API_SECRET;extra.security_context- your usersecurityContext(in a Cube's terms).Rest of the connection fields will be ignored.Note: theCubeBuildOperatoruses Cube's/cubejs-api/v1/pre-aggregations/jobsendpoint that is forbiden by the default. Make sure that specifiedsecurityContextand Cube'scontextToPermissionsfunction are
configured in a way that allow you to run this query. See Cube's
documentation for a more context.Package descriptionThis package provides the commonCubeHookclass, the abstractCubeBaseOperator, theCubeQueryOperatorto run analytical queries
over a Cube and theCubeBuildOperatorto run pre-aggregations build
process.DependenciesPython 3.10PackageVersiontyping2.6.0jwt2.6.0json2.0.9requests2.28.2airflow2.5.1
Package: airflow-provider-dolphindb
Airflow-provider-dolphindbInstalling airflow-provider-dolphindbpipinstallairflow-provider-dolphindbExample DAGsTo create a database and a table in it, and then execute an external .dos script file to insert data, follow these steps:Copy theexample_dolphindb.pyfile to your DAGs folder. If you use the default airflow configurationairflow.cfg, you may need to create the DAGs folder yourself, which is located inAIRFLOW_HOME/dags.Copy theinsert_data.dosfile to the same directory asexample_dolphindb.py.Start your DolphinDB server on port 8848.Start airflow in the development environment:cd/your/project/dir/# Only absolute paths are acceptedexportAIRFLOW_HOME=/your/project/dir/exportAIRFLOW_CONN_DOLPHINDB_DEFAULT="dolphindb://admin:123456@127.0.0.1:8848"python-mairflowstandalonePlease refer to theOfficial documentation for the production environment.Now, you can find the example_dolphindb DAG on your airflow web page. You can try to trigger it.Developer DocumentationInstalling Apache AirflowRefer tohttps://airflow.apache.org/docs/apache-airflow/stable/start.htmlfor further details on this topic.# It is recommended to use the current project directory as the airflow working directorycd/your/source/dir/airflow-provider-dolphindb# Only absolute paths are acceptedexportAIRFLOW_HOME=/your/source/dir/airflow-provider-dolphindb# Install apache-airflow 2.6.3AIRFLOW_VERSION=2.6.3PYTHON_VERSION="$(python--version|cut-d" "-f2|cut-d"."-f1-2)"CONSTRAINT_URL="https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-${PYTHON_VERSION}.txt"pipinstall"apache-airflow==${AIRFLOW_VERSION}"--constraint"${CONSTRAINT_URL}"Additionally, you may need to install Kubernetes to eliminate errors in airflow routines:pipinstallkubernetesInstalling airflow-provider-dolphindb for testingRefer tohttps://pip.pypa.io/en/stable/cli/pip_install/#install-editablefor further details on this topic.python-mpipinstall-e.TestingRun the following commands to validate the installation procedure above.cd/your/source/dir/airflow-provider-dolphindb# Only absolute paths are acceptedexportAIRFLOW_HOME=/your/source/dir/airflow-provider-dolphindbexportAIRFLOW_CONN_DOLPHINDB_DEFAULT="dolphindb://admin:123456@127.0.0.1:8848"pytestPackaging airflow-provider-dolphindbRun the following command.python-mbuild
Package: airflow-provider-duckdb
airflow-provider-duckdbA DuckDB provider for Airflow. This provider exposes a hook/connection that returns a DuckDB connection.This works for either local or MotherDuck connections.Installationpipinstallairflow-provider-duckdbConnectionThe connection type isduckdb. It supports setting the following parameters:Airflow field nameAirflow UI labelDescriptionhostPath to local database filePath to local file. Leave blank (with no password) for in-memory database.schemaMotherDuck database nameName of the MotherDuck database. Leave blank for default.passwordMotherDuck Service tokenMotherDuck Service token. Leave blank for local database.These have been relabeled in the Airflow UI for clarity.For example, if you want to connect to a local file:Airflow field nameAirflow UI labelValuehostPath to local database file/path/to/file.dbschemaMotherDuck database name(leave blank)passwordMotherDuck Service token(leave blank)If you want to connect to a MotherDuck database:Airflow field nameAirflow UI labelValuehostPath to local database file(leave blank)schemaMotherDuck database name<YOUR_DB_NAME>, or leave blank for defaultpasswordMotherDuck Service token<YOUR_SERVICE_TOKEN>Usageimportpandasaspdimportpendulumfromairflow.decoratorsimportdag,taskfromduckdb_provider.hooks.duckdb_hookimportDuckDBHook@dag(schedule=None,start_date=pendulum.datetime(2022,1,1,tz="UTC"),catchup=False,)defduckdb_transform():@taskdefcreate_df()->pd.DataFrame:"""Create a dataframe with some sample data"""df=pd.DataFrame({"a":[1,2,3],"b":[4,5,6],"c":[7,8,9],})returndf@taskdefsimple_select(df:pd.DataFrame)->pd.DataFrame:"""Use DuckDB to select a subset of the data"""hook=DuckDBHook.get_hook('duckdb_default')conn=hook.get_conn()# execute a simple queryres=conn.execute("SELECT a, b, c FROM df WHERE a >= 2").df()returnres@taskdefadd_col(df:pd.DataFrame)->pd.DataFrame:"""Use DuckDB to add a column to the data"""hook=DuckDBHook.get_hook('duckdb_default')conn=hook.get_conn()# add a columnconn.execute("CREATE TABLE tb AS SELECT *, a + b AS d FROM df")# get the tablereturnconn.execute("SELECT * FROM tb").df()@taskdefaggregate(df:pd.DataFrame)->pd.DataFrame:"""Use DuckDB to aggregate the data"""hook=DuckDBHook.get_hook('duckdb_default')conn=hook.get_conn()# aggregatereturnconn.execute("SELECT SUM(a), COUNT(b) FROM df").df()create_df_res=create_df()simple_select_res=simple_select(create_df_res)add_col_res=add_col(simple_select_res)aggregate_res=aggregate(add_col_res)duckdb_transform()
Package: airflow-provider-firebolt
Firebolt Provider for Apache AirflowThis is the provider package for thefireboltprovider. All classes for this provider package are in thefirebolt_providerPython package.ContentsInstallationConfigurationModulesOperatorsHooksInstallationYou can install this package viapipinstallairflow-provider-fireboltairflow-provider-fireboltrequiresapache-airflow2.0+ andfirebolt-sdk1.1+.ConfigurationIn the Airflow user interface, configure a Connection for Firebolt. Configure the following fields:Conn Id:firebolt_conn_id.Conn Type:Firebolt.Client ID: Service account ID.Client Secret: Service account secret.Engine_Name: Firebolt Engine Name.Account: Name of the account you're connecting to.Client id and secret credentials can be obtained by registering aService account.NoteIf you're accessing Firebolt UI viaapp.firebolt.iothen use Username and Password instead of Client ID and Client Secret to connect.ModulesOperatorsoperators.firebolt.FireboltOperatorruns a provided SQL script against Firebolt and returns results.operators.firebolt.FireboltStartEngineOperatoroperators.firebolt.FireboltStopEngineOperatorstarts/stops the specified engine, and waits until it is actually started/stopped. If theengine_nameis not specified, it will use theengine_namefrom the connection, if it also not specified it will start the default engine of the connection database. Note: start/stop operator requires actual engine name, if engine URL is specified instead, start/stop engine operators will not be able to handle it correctly.Hookshooks.firebolt.FireboltHookestablishes a connection to Firebolt.ContributingSee:CONTRIBUTING.MD
Package: airflow-provider-fivetran
Fivetran Provider for Apache AirflowThis package provides an operator, sensor, and hook that integratesFivetraninto Apache Airflow.FivetranOperatorallows you to start Fivetran jobs from Airflow andFivetranSensorallows you to monitor a Fivetran sync job for completion before running downstream processes.Fivetran automates your data pipeline, and Airflow automates your data processing.InstallationPrerequisites: An environment runningapache-airflow.pip install airflow-provider-fivetranConfigurationIn the Airflow user interface, configure a Connection for Fivetran. Most of the Connection config fields will be left blank. Configure the following fields:Conn Id:fivetran_defaultConn Type:FivetranFivetran API Key: Your Fivetran API KeyFivetran API Secret: Your Fivetran API SecretFind the Fivetran API Key and Secret in theFivetran Account Settings, under theAPI Configsection. See our documentation for more information onFivetran API Authentication.The sensor and operator assume theConn Idis set tofivetran_default, however if you are managing multipe Fivetran accounts, you can set this to anything you like. See the DAG in examples to see how to specify a customConn Id.ModulesFivetran OperatorFivetranOperatorstarts a Fivetran sync job. Note that when a Fivetran sync job is controlled via an Operator, it is no longer run on the schedule as managed by Fivetran. In other words, it is now scheduled only from Airflow.FivetranOperatorrequires that you specify theconnector_idof the sync job to start. You can findconnector_idin the Settings page of the connector you configured in theFivetran dashboard.Import into your DAG via:from fivetran_provider.operators.fivetran import FivetranOperatorFivetran SensorFivetranSensormonitors a Fivetran sync job for completion. Monitoring withFivetranSensorallows you to trigger downstream processes only when the Fivetran sync jobs have completed, ensuring data consistency. You can use multiple instances ofFivetranSensorto monitor multiple Fivetran connectors.Note, it is possible to monitor a sync that is scheduled and managed from Fivetran; in other words, you can useFivetranSensorwithout usingFivetranOperator. If used in this way, your DAG will wait until the sync job starts on its Fivetran-controlled schedule and then completes.FivetranSensorrequires that you specify theconnector_idof the sync job to start. You can findconnector_idin the Settings page of the connector you configured in theFivetran dashboard.Import into your DAG via:from fivetran_provider.sensors.fivetran import FivetranSensorExamplesSee theexamplesdirectory for an example DAG.IssuesPlease submitissuesandpull requestsin our official repo:https://github.com/fivetran/airflow-provider-fivetranWe are happy to hear from you. Please email any feedback to the authors atdevrel@fivetran.com.AcknowledgementsSpecial thanks toPete DeJoy,Plinio Guzman, andDavid KoenitzerofAstronomer.iofor their contributions and support in getting this provider off the ground.
Package: airflow-provider-flyte
Flyte Provider for Apache AirflowThis package provides an operator, a sensor, and a hook that integratesFlyteinto Apache Airflow.FlyteOperatoris helpful to trigger a task/workflow in Flyte andFlyteSensorenables monitoring a Flyte execution status
for completion.InstallationPrerequisites: An environment runningapache-airflow.pip install airflow-provider-flyteConfigurationIn the Airflow UI, configure aConnectionfor Flyte.Host (required): The FlyteAdmin host.Port (optional): The FlyteAdmin port.Login (optional):client_idPassword (optional):client_credentials_secretExtra (optional): Specify theextraparameter as JSON dictionary to provide additional parameters.project: The default project to connect to.domain: The default domain to connect to.insecure: Whether to use SSL or not.command: The command to execute to return a token using an external process.scopes: List of scopes to request.auth_mode: The OAuth mode to use. Defaults to pkce flow.env_prefix: Prefix that will be used to lookup for injected secrets at runtime.default_dir: Default directory that will be used to find secrets as individual files.file_prefix: Prefix for the file in thedefault_dir.statsd_host: The statsd host.statsd_port: The statsd port.statsd_disabled: Whether to send statsd or not.statsd_disabled_tags: Turn on to reduce cardinality.local_sandbox_pathS3 Config:s3_enable_debugs3_endpoints3_retriess3_backoffs3_access_key_ids3_secret_access_keyGCS Config:gsutil_parallelismModulesFlyte OperatorTheFlyteOperatorrequires aflyte_conn_idto fetch all the connection-related
parameters that are useful to instantiateFlyteRemote. Also, you must give alaunchplan_nameto trigger a workflow, ortask_nameto trigger a task; you can give a
handful of other values that are optional, such asproject,domain,max_parallelism,raw_data_prefix,kubernetes_service_account,labels,annotations,secrets,notifications,disable_notifications,oauth2_client,version, andinputs.Import into your DAG via:from flyte_provider.operators.flyte import FlyteOperatorFlyte SensorIf you need to wait for an execution to complete, useFlyteSensor.
Monitoring withFlyteSensorallows you to trigger downstream processes only when the Flyte executions are complete.Import into your DAG via:from flyte_provider.sensors.flyte import FlyteSensorExamplesSee theexamplesdirectory for an example DAG.IssuesPlease file issues and open pull requestshere.
If you hit any roadblock, hit us up onSlack.
Package: airflow-provider-fxiaoke
Apache-airflow-providers-fxiaokeAirflow plugin forfxiaoke APIInstallpipinstallairflow-providers-fxiaokeUsageCreate New connections namedfxiaoke_defaultwith extra json:{"app_id":"", "app_secret":"", "permanent_code":"", "open_user_id":""}HooksFxiaokeHooksOperatorsFxiaokeToGCSOperator
Package: airflow-provider-graphgrid
The airflow-provider-graphgrid package includes Operators and functionality
in order to better streamline Airflow workflows within GraphGrid CDP.Airflow DAGs can leverage theGraphGridDockerOperatorandGraphGridMountviafromgraphgrid_provider.operators.graphgrid_dockerimport\GraphGridDockerOperator,GraphGridMountand use them as if they were a normalDockerOperatort_0=GraphGridDockerOperator(task_id='task_0',dag=dag,mounts=[GraphGridMount(target="/some_path",source="/some_other_path",type="bind")],image="some-image",auto_remove=True,)
Package: airflow-provider-hex
Hex Airflow ProviderProvides an Airflow Operator and Hook to trigger Hex project runs.ThisAirflow Provider Packageprovides Hooks and Operators for interacting with the Hex API.RequirementsAirflow >=2.2Hex API TokenInitial SetupInstall the package.pip install airflow-provider-hexAfter creating a Hex API token, set up your Airflow Connection Credentials in the Airflow
UI.Connection ID:hex_defaultConnection Type:Hex ConnectionHost:https://app.hex.techHex API Token:your-token-hereOperatorsTheairflow_provider_hex.operators.hex.HexRunProjectOperatorOperator runs Hex Projects, either synchronously or asynchronously.In the synchronous mode, the Operator will start a Hex Project run and then
poll the run until either an error or success status is returned, or until
the poll timeout. If the timeout occurs, the default behaviour is to attempt to
cancel the run.In the asynchronous mode, the Operator will request that a Hex Project is run,
but will not poll for completion. This can be useful for long-running projects.The operator accepts inputs in the form of a dictionary. These can be used to
override existing input elements in your Hex project.You may also optionally include notifications for a particular run. See
theHex API documentationfor details.HooksTheairflow_provider_hex.hooks.hex.HexHookprovides a low-level interface to the Hex API.These can be useful for testing and development, as they provide both a genericrunmethod which sends an authenticated request to the Hex API, as well as
implementations of therunmethod that provide access to specific endpoints.ExamplesA simplified example DAG demonstrates how to use theAirflow Operatorfromairflow_provider_hex.operators.heximportHexRunProjectOperatorPROJ_ID='abcdef-ghijkl-mnopq'notifications:list[NotificationDetails]=[{"type":"SUCCESS","includeSuccessScreenshot":True,"slackChannelIds":["HEX666SQG"],"userIds":[],"groupIds":[],}]...sync_run=HexRunProjectOperator(task_id="run",hex_conn_id="hex_default",project_id=PROJ_ID,dag=dag,notifications=notifications)