Saravanan666/pypi-org huggingface dataset:
Package: airflow-providers-siasg
airflow-providers-siasgProvider para Airflow que comunica comSIASGe seus sistemas derivadosInstalaçãopipinstallairflow-providers-siasgConteúdoSub-provider paraDW-SIASGNovo tipo de conexão "Conta do DW-SIASG"Transfers de relatórios do DW-SIASG para arquivo e banco MongoDBExemplos de usotask2=DWSIASGRelatorioParaMongoOperator(task_id='task2',id_conexao='teste',id_relatorio='BFD128CD11EC5B5D670B0080EF6553F5',respostas_prompts=['160030','160130'],timeout_segundos=120,id_conexao_mongo='teste_mongo',banco='teste_siasg',colecao='teste_siasg',)Para mais exemplos, consultarairflow/providers/siasg/example_dags/dw.py
Package: airflow-providers-tesouro-gerencial
airflow-providers-tesouro-gerencialProvider do Airflow para comunicação comTesouro Gerencial.Instalaçãopipinstallairflow-providers-tesouro-gerencialConteúdoHook para conexão com Tesouro Gerencial, que contém métodos para:Entrada e saída de contexto (clásulawith), inicializando e encerrando sessão no Tesouro GerencialExecução e exportação de relatórioTransfers que carregam relatórios do Tesouro Gerencial para:Arquivo local;Banco MongoDBExemplo de UsoTransferência de relatório para arquivo local:fromdatetimeimportdatetime,timedeltafromairflow.decoratorsimportdagfromairflow.providers.tesouro_gerencial.transfers.relatorio_para_arquivoimportRelatorioParaArquivo@dag(schedule_interval=None,start_date=datetime(2021,1,1))defteste_tesouro_gerencial():teste=RelatorioParaArquivo(task_id='teste1',id_conexao_siafi='teste',id_relatorio='970A89D511EC923631090080EFC5BFD1',caminho_arquivo='/tmp/tg.xlsx',respostas_prompts_valor=['622110000','622120000'],retries=10,retry_delay=timedelta(minutes=2))minha_dag=teste_tesouro_gerencial()Transferência para banco MongoDBfromdatetimeimportdatetime,timedeltafromairflow.decoratorsimportdagfromairflow.providers.tesouro_gerencial.transfers.relatorio_para_mongoimportRelatorioParaMongo@dag(schedule_interval=None,start_date=datetime(2021,1,1))defteste_tesouro_gerencial():teste=RelatorioParaMongo(task_id='teste2',id_conexao_siafi='teste',id_relatorio='970D89D511EC423631090080EFA5BFD1',id_conexao_mongo='teste_mongo',banco='teste',colecao='teste',respostas_prompts_valor=['622110000','622120000'],truncar_colecao=True,nulos_para_zero=True,retries=10,retry_delay=timedelta(minutes=2))minha_dag=teste_tesouro_gerencial()
Package: airflow-provider-toloka
Airflow Toloka ProviderThis library allows you to run crowdsourcingTolokaprocesses inApache Airflow- a widely used workflow management systemHere you can find a collection of ready-made Airflow tasks for the most frequently used actions inToloka-Kit.Getting started$ pip install airflow-provider-tolokaA good way to start is to follow theexamplein this repo.TolokaHookTolokaHookis used for getting toloka OAuth token and creatingTolokaClientwith it.
You can getTolokaClientfromTolokaHookby callingget_conn()method.To make an appropriate Airflow Connection you need to create it in the Airflow Connections UI with following parameters:Conn ID:toloka_defaultConn Type:TolokaToken: enter your OAuth token for Toloka.
You can learn more about how to get ithere.Environment: enterproductionorsandboxTasks use thetoloka_defaultconnection id by default, but
if needed, you can create additional Airflow Connections and reference them
as the functiontoloka_conn_idargument.Tasks and SensorsThere are several tasks and sensors that give you easy way to interact with Toloka from Airflow DAGs.
Creating a project and a pool, adding tasks and getting assignments are among them.
You can easily create your own task usingTolokaHookif it is beyond the scope of implemented ones.
And it would be nice to have your pull request with updates.Check out ourexampleto see tasks and sensors in the battlefield.Useful LinksToloka homepage.Apache Airflow homepage.Toloka API documentation.Toloka-kit usage examples.Questions and bug reportsFor reporting bugs please use theToloka/bugreportpage.Join our English-speakingslack communityfor both tech and abstract questions.License© YANDEX LLC, 2022. Licensed under the Apache License, Version 2.0. See LICENSE file for more details.
Package: airflow-provider-vdk
Versatile Data Kit Airflow providerA set of Airflow operators, sensors and a connection hook intended to help schedule Versatile Data Kit jobs using Apache Airflow.UsageTo install it simply run:pip install airflow-provider-vdkThen you can create a workflow of data jobs (deployed by VDK Control Service) like this:from datetime import datetime

from airflow import DAG
from vdk_provider.operators.vdk import VDKOperator

with DAG(
    "airflow_example_vdk",
    schedule_interval=None,
    start_date=datetime(2022, 1, 1),
    catchup=False,
    tags=["example", "vdk"],
) as dag:
    trino_job1 = VDKOperator(
        conn_id="vdk-default",
        job_name="airflow-trino-job1",
        team_name="taurus",
        task_id="trino-job1",
    )

    trino_job2 = VDKOperator(
        conn_id="vdk-default",
        job_name="airflow-trino-job2",
        team_name="taurus",
        task_id="trino-job2",
    )

    transform_job = VDKOperator(
        conn_id="vdk-default",
        job_name="airflow-transform-job",
        team_name="taurus",
        task_id="transform-job",
    )

    [trino_job1, trino_job2] >> transform_jobExampleSeefull example hereDemoYou can see demo during one of the community meetings here:https://www.youtube.com/watch?v=c3j1aOALjVU&t=690sArchitectureSee thevdk enhancement proposal spec here
Package: airflow-provider-whylogs
whylogs Airflow OperatorThis is a package for thewhylogsprovider, the open source standard for data and ML logging. With whylogs, users are able to generate summaries of their datasets (called whylogs profiles) which they can use to:Track changes in their datasetCreate data constraints to know whether their data looks the way it shouldQuickly visualize key summary statistics about their datasetsThis Airflow operator focuses on simplifying whylogs' usage along with Airflow. Users are encouraged to benefit from their existing Data Profiles, which are created with whylogs and can bring a lot of value and visibility to track their data changes over time.InstallationYou can install this package on top of an existing Airflow 2.0+ installation (Requirements) by simply running:$pipinstallairflow-provider-whylogsTo install this provider from source, run these instead:$gitclonegit@github.com:whylabs/airflow-provider-whylogs.git
$cdairflow-provider-whylogs
$python3-mvenv.env&&source.env/bin/activate
$pip3install-e.Usage exampleIn order to benefir from the existing operators, users will have to profile their data first, with theirprocessingenvironment of choice. To create and store a profile locally, run the following command on a pandas DataFrame:importwhylogsaswhydf=pd.read_csv("some_file.csv")results=why.log(df)results.writer("local").write()And after that, you can use our operators to either:Create a Summary Drift Report, to visually help you identify if there was drift in your datafromwhylogs_provider.operators.whylogsimportWhylogsSummaryDriftOperatorsummary_drift=WhylogsSummaryDriftOperator(task_id="drift_report",target_profile_path="data/profile.bin",reference_profile_path="data/profile.bin",reader="local",write_report_path="data/Profile.html",)Run a Constraints check, to check if your profiled data met some criteriafromwhylogs_provider.operators.whylogsimportWhylogsConstraintsOperatorfromwhylogs.core.constraints.factoriesimportgreater_than_numberconstraints=WhylogsConstraintsOperator(task_id="constraints_check",profile_path="data/profile.bin",reader="local",constraint=greater_than_number(column_name="my_column",number=0.0),)NOTE: It is important to note that even though it is possible to create a Dataset Profile with the Python Operator, Airflow tries to separate the concern of orchestration from processing, so that is one of the reasons why we didn't want to have a strong opinion on how to read data and profile it, enabling users to best adjust this step to their existing scenario.A full DAG example can be found on the whylogs_provider packagedirectory.RequirementsThe current requirements to use this Airflow Provider are described on the table below.PIP packageVersion requiredapache-airflow>=2.0whylogs[viz, s3]>=1.0.10ContributingUsers are always welcome to ask questions and contribute to this repository, by submitting issues and communicating with us through ourcommunity Slack. Feel free to reach out and makewhylogseven more awesome to use with Airflow.Happy coding! 😄
Package: airflow-ray-executor
Airflow Ray ExecutorAirflow executor implemented usingray中文Usage$pipinstallairflow-ray-executorEdit yourairflow.cfgto set your executor to class:airflow_ray_executor.RayExecutorand add ray client address to this file, example:executor = airflow_ray_executor.RayExecutor[ray]# ray client address to connect to ray cluster# Ray Executor will start Ray on a single machine if not providedclient = ray://127.0.0.1:10001Please note:
Airflow not support sqlite database when executor neither DebugExecutor nor SequentialExecutor
Package: airflow-rest-api
AIRFLOW-REST-APIAsync wrapper forAirflow REST API. See more indocumentationINSTALLpipinstallairflow-rest-apiUSAGEimportasyncioimportosimportaiohttpfromairflow_rest_api.apiimportAirflowRestApiAIRFLOW_HOST=os.environ.get("AIRFLOW_HOST","http://127.0.0.1:8080")AIRFLOW_USER=os.environ.get("AIRFLOW_USER","admin")AIRFLOW_PASSWORD=os.environ.get("AIRFLOW_PASSWORD","admin")asyncdefmain(ara):fortemplateinara.find_template("/dags"):template_id=template.idbreakauth=aiohttp.BasicAuth(AIRFLOW_USER,AIRFLOW_PASSWORD)asyncwithaiohttp.ClientSession(auth=auth)assession:airflow_tasks=awaitara.execute(session=session,template_id=template_id)ifairflow_tasks.status==200:file_token=airflow_tasks.raw["dags"][0]["file_token"]template_id=360dag_source_code=(awaitara.execute(session=session,method=ara.get_template(template_id).method,url=ara.render_url(template_id=template_id,file_token=file_token))).rawprint(dag_source_code)ara=AirflowRestApi(airflow_host=AIRFLOW_HOST)ara.show_templates()loop=asyncio.get_event_loop()loop.run_until_complete(main(ara))
Package: airflow-restapi-sdk
AIRFLOW_RESTAPI-SDKfromairflow_restapi_sdkimportClient,Stateclient=Client('http://localhost:8080')# 触发DAGclient.dag.trigger('test_dag',conf={'k':'v'})# {#     "execution_date": "2020-08-20T07:51:36+00:00",#     "message": "Created <DagRun test_dag @ 2020-08-20 07:51:36+00:00: manual__2020-08-20T07:51:36+00:00, externally triggered: True>",#     "run_id": "manual__2020-08-20T07:51:36+00:00"# }# 查看DAG运行状态client.dag.state('test_dag','2020-08-20T07:51:36+00:00')# {'state': 'failed'}# 触发DAG并阻塞，直到成功或失败status=client.dag.trigger_join('test_dag',conf={'k':'v'},timeout=300)print(status)# {'state': 'failed'}
Package: airflow-run
Airflow RunPython tool for deploying Airflow Multi-Node Cluster.RequirementsPython >=3.6 (tested)GoalTo provide a quick way to setup Airflow Multi-Node Cluster (a.k.a. Celery Executor Setup).StepsGenerate config yaml file.Run commands to start webserver, scheduler, worker, (rabbitmq, postgres).Add dag files and run initdb.Generate config file:afr--generate_configRunning the tool in the same directory as config file:afr--runpostgresqlafr--runinitdbafr--runrabbitmqafr--runwebserverafr--runschedulerafr--runworker--queue{queuename}afr--runflowerOr, running the tool specifying config path:afr--runpostgresql--config/path/config.yamlOr, use this environment variable to set the config path:exportAIRFLOWRUN_CONFIG_PATH="/some_path/config.yaml"After running webserver, scheduler and worker (postgres and rabbitmq if needed local instances), Add your dag files in the dags subdirectory in the directory you defined in the config file.(* note: make sure you have the correct user permission in the dags, logs subdirectories.)That is it!!Default Config yaml file:private_registry:Falseregistry_url:registry.hub.docker.comusername:""password:""repository:pkuong/airflow-runimage:airflow-runtag:latestlocal_dir:{local directory where you want to mount /dags and /logs folder}webserver_port:8000flower_port:5555custom_mount_volumes:[]env:AIRFLOW__CORE__EXECUTOR:CeleryExecutorAIRFLOW__CORE__LOAD_EXAMPLES:"False"AIRFLOW__CORE__DAGS_FOLDER:/usr/local/airflow/airflow/dagsAIRFLOW__CORE__LOGS_FOLDER:/usr/local/airflow/airflow/logsAIRFLOW_HOME:/usr/local/airflowAIRFLOW__CORE__FERNET_KEY:""rabbitmq:name:rabbitmqusername:{username}password:{password}host:{IP}virtual_host:/image:rabbitmq:3-managementhome:/var/lib/rabbitmqui_port:15672port:5672env:RABBITMQ_DEFAULT_USER:{username}RABBITMQ_DEFAULT_PASS:{password}postgresql:name:postgresqlusername:{username}password:{password}host:{host}image:postgresdata:/var/lib/postgresql/dataport:5432env:PGDATA:/var/lib/postgresql/data/pgdataPOSTGRES_USER:{username}POSTGRES_PASSWORD:{password}Custom mount volumesYou can specify custom mount volumes in the container, for example:custom_mount_volumes:-host_path:/Users/bob/.awscontainer_path:/usr/local/airflow/.awsDocker imageThis tool is using the following public docker image by default.https://hub.docker.com/repository/docker/pkuong/airflow-runBuilding the image:If you want to build your own image, you can run the following:afd--build--config_path={absolutepathtoconfig.yaml}--dockerfile_path={absolutepathtodirectorywhichcontainsDockerfile}ContributorsPaulo Kuong (@pkuong)
Package: airflowscan
airflowscanChecklist and tools for increasing security of Apache Airflow.DISCLAIMERThis project NOT AFFILIATED with the Apache Foundation and the Airflow project,
and is not endorsed by them.ContentsThe purpose of this project is provide tools to increase security ofApache Airflow.
installations. This projects provides the following tools:Configuration file with hardened settings - seehardened_airflow.cfg.Security checklist for hardening default installations - seeCHECKLIST.MD.Static analysis tool to check Airflow configuration files for insecure settings.JSON schema document used for validation by the static analysis tool - seeairflow_cfg.schemaInformation for the Static Analysis Tool (airflowscan)The static analysis tool can check an Airflow configuration file for settings related to security. The tool
convers the config file to JSON, and then uses a JSON Schema to do the validation.RequirementsPython 3 is required and you can find all required modules in therequirements.txtfile.
Only tested on Python 3.7 but should work on other 3.x releases. No plans to 2.x support at
this time.InstallationYou can install this via PIP as follows:pip install airflowscan
airflowscanTo download and run manually, do the following:git clone https://github.com/nightwatchcybersecurity/airflowscan.git
cd airflowscan
pip -r requirements.txt
python -m airflowscan.cliHow to useTo scan a configuration file, do the following command:airflowscan scan some_airflow.cfgReporting bugs and feature requestsPlease use the GitHub issue tracker to report issues or suggest features:https://github.com/nightwatchcybersecurity/airflowscanYou can also send emai toresearch /at/ nightwatchcybersecurity [dot] com
Package: airflow-sendgrid-provider-j2
Airflow SendGrid ProviderThis is a modified version ofhttps://github.com/apache/airflow/tree/master/airflow/providers/sendgridChangelog:added dynamic templates supportchanged errors messagesUsageIftemplate_idandtemplate_substitutionparameters is passed,
thehtml_contentandsubjectparameters will be ignoredExamplesend_email(
        from_email='hello@example.com',
        from_name='Hello',
        subject=null,
        to='world@example.com'',
        html_content=null,
        template_id='00000000-0000-0000-0000-000000000000',
        template_substitution={
            ':number': '1337'
        }
    )
Package: airflow-smartsheet-plugin
airflow-smartsheetSimple hooks and operators for transporting data from Smartsheet.Import Smartsheet into PostgreSQL or export as CSV, PDF or EXCEL file.FeaturesSmartsheetToFileOperator: exporting a Smartsheet sheet to a file/jsonSmartsheetToPostgresOperator: exporting a Smartsheet sheet to a PostgreSQL tableInstallUsing pip:pip3installairflow-smartsheet-pluginUsageCreate a variable in Airflow namedSMARTSHEET_ACCESS_TOKENto store your Smartsheet API access token.You can also pass in an override token in your DAG definition.This plugin is published as a pip package. Refer to theexample DAGfor available parameters.Refer to theenumsfor available PDF paper sizes.
Package: airflow-socrata-plugin
Airflow-SocrataSimple hooks and operators for uploading data to Socrata.FeaturesUpsert or reupload PostgreSQL tables to SocrataInstallUsing pip:pip3installairflow-socrata-pluginUsageCreate a connection namedhttp_socrataof typehttpto store Socrata credentials.You can also pass inconn_nameparameter in DAG definition to override.Create a connection namedetl_postgresof typepostgresto store PostgreSQL credentials.You can also pass inpostgres_conn_nameparameter in DAG definition to override.By default, the plugin looks for the specified table underpublicschema. The schema can be specified withpostgres_schemaparameter.The plugin is published as a pip package. Refer to theexample DAGfor available parameters.
Package: airflow-sops-secrets-backend
Airflow SOPS Secrets Backend for GCP KMSThis packages enables Airflow to pull connections and variables
from files in GCP bucket that are encrypted bySOPSusing GCP KMS.Configure AirflowAdd following toairflow.cfg.[secrets]
backend = airflow_sops.secrets_backend.GcsSopsSecretsBackend
backend_kwargs = {"project_id": "your-project-id"}Available parameters to backend_kwargs:project_id: Optional. GCP project id where the GCS bucket which holds the encrypted connections/variables files reside.,bucket_name: Optional. If not submitted tries retrieving from Composer GCS_BUCKET environment variableconnections_prefix. Optional. Default is "sops/connections". The folder in GCS bucket that holds encrypted connections.variables_prefix: Optional. Default is "sops/variables". The folder in GCS bucket that holds encrypted variables.,encrypted_file_ext: Optional. Default is "enc". The file extension for encrypted sops files. The format is <connection_id or variable_key>.<encrypted_file_ext>.yamlignore_mac: Optional. Default is True. Ignores file checksum when true.GCP Configlocals{gcp_project_id="your-project-id"service_account_name="your-composer-service-account-name"}resource"google_service_account""composer"{account_id=local.service_account_namedisplay_name=local.service_account_nameproject=local.gcp_project_id}resource"google_project_iam_member""composer_worker"{project=local.gcp_project_idrole="roles/composer.worker"member="serviceAccount:${google_service_account.composer.email}"}resource"google_kms_key_ring""secrets"{name=local.gcp_project_idlocation="europe-west1"project=local.gcp_project_id}resource"google_kms_crypto_key""secrets_sops"{name="secrets_sops"key_ring=google_kms_key_ring.secrets.idrotation_period="7776000s"// 90 days}resource"google_kms_crypto_key_iam_member""composer_sops_decrypter"{crypto_key_id=google_kms_crypto_key.secrets_sops.idrole="roles/cloudkms.cryptoKeyDecrypter"member="serviceAccount:${google_service_account.composer.email}"}# some mandatory attributes omittedresource"google_composer_environment""composer"{name="your-composer-environment-name"region="europe-west1"project=local.gcp_project_idconfig{software_config{airflow_config_overrides={secrets-backend="airflow_sops.secrets_backend.GcsSopsSecretsBackend"}pypi_packages={airflow-secrets-sops="==0.0.1"}}node_config{service_account=google_service_account.composer.email}}}SOPSInstallSOPS. Encrypt files
using GCP KMS and upload to GCP bucket sops/connections directoryexportKMS_PATH=$(gcloudkmskeyslist--locationeurope-west1--keyringyour-keyring--projectyour-gcp-project|awk'FNR == 2 {print $1}')sops--encrypt--encrypted-regex'^(password|extra)$'--gcp-kms$KMS_PATHsome-connection.yaml>some-connection.enc.yamlSetuppython-mvenv.venvsource.venv/bin/activate
pipconfigset--siteglobal.extra-index-urlhttps://pypi.org/simple
pipinstall-rrequirements.txtTestpipinstall.airflow-sops-secrets-backend[test]pytestBuildpipinstallairflow-sops-secrets-backend[dev]python-mbuild
Package: airflow-sqlcmd-operator
airflow-sqlcmd-operatorCustom Airflow BashOperator for the Microsoft sqlcmd.This package utilizes the sqlcmd to run Microsoft SQLServer scripts on Linux like you would use them on SSMS for example.Thesqlcmdsupports SQLServer scripts with commands like GO, USE [db_name], etc, and multiple statements.RequirementsYou must havesqlcmdalready installed and (currently) on following location: "/opt/mssql-tools/bin/sqlcmd".Installing on Ubuntu with apt:curlhttps://packages.microsoft.com/keys/microsoft.asc|apt-keyadd-
curlhttps://packages.microsoft.com/config/debian/10/prod.list>/etc/apt/sources.list.d/mssql-release.list# install required packages for pyodbcapt-getupdateACCEPT_EULA=Yapt-getinstall-ymsodbcsql17unixodbc-devmssql-tools&&apt-getcleanUsageOn a dag, you can call it like this:fromairflow_sqlcmd_operatorimportSqlcmdOperatorsqlcmd=SqlcmdOperator("MyDB","/scripts/folder/mydag","do_stuff.sql",dag=dag)
Package: airflow-supporter
airflow_supporterairflow_supporter provide services to supporter to manage Apache AirflowUsageAlert email on off DAG and turn on DAG automaticallyInsert following example in your DAGfromairflow_supporter.dag.check_off_dagimportcreate_check_off_dagcreate_check_off_dag()Variableenroll following Variable in your airflowcheck_off_dag_variable{
    exclude_dag_list: list[str],  default empty list,
    automatically_turn_on: bool, default true,
    email: Optional[str]
  }Restart failed DagRun@dag(dag_id="restart_failed_dagrun_dag",schedule="* * * * *",is_paused_upon_creation=False,catchup=False,start_date=datetime(year=1970,month=1,day=1),)defrestart_failed_dagrun_dag()->None:restart_failed_dagrun_op.restart_failed_dagrun_op(rv=RestartFailedDagrunVariable())restart_failed_dagrun_dag()Restart stucked Task@dag(dag_id="clear_stucked_task_dag",schedule="* * * * *",is_paused_upon_creation=False,catchup=False,start_date=datetime(year=1970,month=1,day=1),)defclear_stucked_task_dag()->None:restart_stucked_task_op.clear_stucked_task_op(rv=RestartStuckedTaskVariable())clear_stucked_task_dag()
Package: airflow-tm1
Note: this was an old proof of concept for Airflow 1.x and probably isn't useful for most peopleI've started a airflow 2 style provider athttps://github.com/scrambldchannel/airflow-provider-tm1, help wanted ;)airflow-tm1A package that provides a hook to simplify connecting to the IBM Cognos TM1 / Planning Analytics REST API.RequirementsPython 3.7+Airflow 1.xTM1pyInstallationInstall with pippip install airflow-tm1UsageCreate a connection in Airflow with at least the following parameters set:HostLoginPortExtrassslAny other parameter accepted by the TM1py RestService constructor (eg base_url, namespace etc) can also be added as a key in the Extras field in the connection.In your DAG file:fromairflow_tm1.hooks.tm1importTM1Hooktm1_hook=TM1Hook(tm1_conn_id="tm1_default")tm1=tm1_hook.get_conn()This will attempt to connect to the TM1 server using the details provided and initialise an instance of the TM1Service class than be accessed attm1_hook.tm1SeeTM1pyfor more details.LicenseSeeLICENSE
Package: airflow-trigger-multiple-dag-run
airflow-trigger-multiple-dag-runUsagepipinstallairflow-trigger-multiple-dag-runDev setupinitial setuppipinstallpoetry
makesetuppublish to testpypipoetryconfigrepositories.testpypihttps://test.pypi.org/legacy/
poetrypublish--build-rtestpypi-p<password>-u<username:=__token__>useful poetry commandspoetryadd<dependencies>
poetryadd--dev<dependencies># dev dependenciespoetryupdate
poetrylock--no-updateContributionContributions are very welcome. Tests can be run withmake test, please ensure the coverage at least stays the same
before you submit a pull request.LicenseDistributed under the terms of theMITlicense, "airflow-trigger-multiple-dagrun" is free and open source softwareRepositoryRepo-url