Saravanan666/pypi-org huggingface dataset:
Package: apache-airflow-providers-sendgrid
Packageapache-airflow-providers-sendgridRelease:3.4.0SendgridProvider packageThis is a provider package forsendgridprovider. All classes for this provider package
are inairflow.providers.sendgridpython package.You can find package information and changelog for the provider
in thedocumentation.InstallationYou can install this package on top of an existing Airflow 2 installation (seeRequirementsbelow
for the minimum Airflow version supported) viapip installapache-airflow-providers-sendgridThe package supports the following python versions: 3.8,3.9,3.10,3.11RequirementsPIP packageVersion requiredapache-airflow>=2.6.0sendgrid>=6.0.0The changelog for the provider package can be found in thechangelog.
Package: apache-airflow-providers-sftp
Packageapache-airflow-providers-sftpRelease:4.9.0SSH File Transfer Protocol (SFTP)Provider packageThis is a provider package forsftpprovider. All classes for this provider package
are inairflow.providers.sftppython package.You can find package information and changelog for the provider
in thedocumentation.InstallationYou can install this package on top of an existing Airflow 2 installation (seeRequirementsbelow
for the minimum Airflow version supported) viapip installapache-airflow-providers-sftpThe package supports the following python versions: 3.8,3.9,3.10,3.11RequirementsPIP packageVersion requiredapache-airflow>=2.6.0apache-airflow-providers-ssh>=2.1.0paramiko>=2.8.0asyncssh>=2.12.0Cross provider package dependenciesThose are dependencies that might be needed in order to use all the features of the package.
You need to install the specified provider packages in order to use them.You can install such cross-provider dependencies when installing from PyPI. For example:pipinstallapache-airflow-providers-sftp[openlineage]Dependent packageExtraapache-airflow-providers-openlineageopenlineageapache-airflow-providers-sshsshThe changelog for the provider package can be found in thechangelog.
Package: apache-airflow-providers-singularity
Packageapache-airflow-providers-singularityRelease:3.4.0SingularityProvider packageThis is a provider package forsingularityprovider. All classes for this provider package
are inairflow.providers.singularitypython package.You can find package information and changelog for the provider
in thedocumentation.InstallationYou can install this package on top of an existing Airflow 2 installation (seeRequirementsbelow
for the minimum Airflow version supported) viapip installapache-airflow-providers-singularityThe package supports the following python versions: 3.8,3.9,3.10,3.11RequirementsPIP packageVersion requiredapache-airflow>=2.6.0spython>=0.0.56The changelog for the provider package can be found in thechangelog.
Package: apache-airflow-providers-sktvane
apache-airflow-providers-sktvaneAIDP 가 제공하는 자원들에 접근하는 용도NESBigQueryVault기타 공용 목적의 코드PyPIhttps://pypi.org/project/apache-airflow-providers-sktvaneEnvironmentsLocalVAULT_TOKEN은관련 문서에서 확인exportVAULT_ADDR=https://vault-public.sktai.ioexportVAULT_TOKEN={{VAULT_TOKEN}}exportAIRFLOW__CORE__DAGS_FOLDER=.Deploymentmain브랜치에push이벤트 발생 시 배포, 부득이하게 로컬 환경에서 배포할 경우 아래 명령 수행# build$pythonsetup.pysdistbdist_wheel# upload$twineuploaddist/*# remove$rm-rfbuilddistapache_airflow_providers_sktvane.egg-infoComponentsOperatorsairflow.providers.sktvane.operators.nes.NesOperator: AIDP 의NES사용fromairflow.providers.sktvane.operators.nesimportNesOperator...NesOperator(task_id="jupyter_daily_count",input_nb="https://github.com/sktaiflow/notebooks/blob/master/statistics/jupyter_daily_count.ipynb",parameters={"current_date":"{{ ds }}","channel":"#aim-statistics"},)Sensorsairflow.providers.sktvane.sensors.gcp.BigqueryPartitionSensor: AIDP 의BigQuery파티션 체크fromairflow.providers.sktvane.sensors.gcpimportBigqueryPartitionSensor...BigqueryPartitionSensor(task_id=f"{table}_partition_sensor",dataset_id="wind_tmt",table_id=table,partition="dt = '{{ds}}'",)Macrosairflow.providers.sktvane.macros.slack.send_fail_message: AIDP 정의 포맷으로Slack에러 메시지 발송fromairflow.providers.sktvane.macros.slackimportsend_fail_message...defsend_aidp_fail_message(slack_email:str)->None:send_fail_message(slack_channel="#aidp-airflow-monitoring",slack_username=f"Airflow-AlarmBot-{env}",slack_email=slack_email,)airflow.providers.sktvane.macros.gcp.bigquery_client: AIDP 의BigQuery사용fromairflow.providers.sktvane.macros.gcpimportbigquery_client...defbq_query_to_bq(query,dest_table_name,**kwarg):bq_client=bigquery_client()job=bq_client.query(query)job.result()airflow.providers.sktvane.macros.vault.get_secrets: AIDP 의Vault사용fromairflow.providers.sktvane.macros.vaultimportget_secrets...defget_hive_conn():frompyhiveimporthivehiveserver2=get_secrets(path="ye/hiveserver2")host=hiveserver2["ip"]port=hiveserver2["port"]user=hiveserver2["user"]conn=hive.connect(host,port=port,username=user)returnconnairflow.providers.sktvane.macros.date.ds_nodash_plus_days: AIDP 에서 제공하는date유틸리티fromairflow.providers.sktvane.macros.dateimportds_nodash_plus_days...defds_nodash_tomorrow(ds):ds_nodash_plus_days(ds,1)airflow.providers.sktvane.macros.date.ds_nodash_minus_days:ds_nodash_plus_days와 동일airflow.providers.sktvane.macros.date.ym_nodash_add_month:ds_nodash_plus_days와 동일airflow.providers.sktvane.macros.date.first_day_of_this_month:ds_nodash_plus_days와 동일airflow.providers.sktvane.macros.date.last_day_of_this_month:ds_nodash_plus_days와 동일airflow.providers.sktvane.macros.date.get_latest_loaded_dt:ds_nodash_plus_days와 동일
Package: apache-airflow-providers-slack
Packageapache-airflow-providers-slackRelease:8.6.1Slackservices integration including:Slack APISlack Incoming WebhookProvider packageThis is a provider package forslackprovider. All classes for this provider package
are inairflow.providers.slackpython package.You can find package information and changelog for the provider
in thedocumentation.InstallationYou can install this package on top of an existing Airflow 2 installation (seeRequirementsbelow
for the minimum Airflow version supported) viapip installapache-airflow-providers-slackThe package supports the following python versions: 3.8,3.9,3.10,3.11RequirementsPIP packageVersion requiredapache-airflow>=2.6.0apache-airflow-providers-common-sql>=1.3.1slack_sdk>=3.19.0Cross provider package dependenciesThose are dependencies that might be needed in order to use all the features of the package.
You need to install the specified provider packages in order to use them.You can install such cross-provider dependencies when installing from PyPI. For example:pipinstallapache-airflow-providers-slack[common.sql]Dependent packageExtraapache-airflow-providers-common-sqlcommon.sqlThe changelog for the provider package can be found in thechangelog.
Package: apache-airflow-providers-smtp
Packageapache-airflow-providers-smtpRelease:1.6.0Simple Mail Transfer Protocol (SMTP)Provider packageThis is a provider package forsmtpprovider. All classes for this provider package
are inairflow.providers.smtppython package.You can find package information and changelog for the provider
in thedocumentation.InstallationYou can install this package on top of an existing Airflow 2 installation (seeRequirementsbelow
for the minimum Airflow version supported) viapip installapache-airflow-providers-smtpThe package supports the following python versions: 3.8,3.9,3.10,3.11RequirementsPIP packageVersion requiredapache-airflow>=2.6.0The changelog for the provider package can be found in thechangelog.
Package: apache-airflow-providers-snowflake
Packageapache-airflow-providers-snowflakeRelease:5.3.1SnowflakeProvider packageThis is a provider package forsnowflakeprovider. All classes for this provider package
are inairflow.providers.snowflakepython package.You can find package information and changelog for the provider
in thedocumentation.InstallationYou can install this package on top of an existing Airflow 2 installation (seeRequirementsbelow
for the minimum Airflow version supported) viapip installapache-airflow-providers-snowflakeThe package supports the following python versions: 3.8,3.9,3.10,3.11RequirementsPIP packageVersion requiredapache-airflow>=2.6.0apache-airflow-providers-common-sql>=1.10.0snowflake-connector-python>=2.7.8snowflake-sqlalchemy>=1.1.0Cross provider package dependenciesThose are dependencies that might be needed in order to use all the features of the package.
You need to install the specified provider packages in order to use them.You can install such cross-provider dependencies when installing from PyPI. For example:pipinstallapache-airflow-providers-snowflake[common.sql]Dependent packageExtraapache-airflow-providers-common-sqlcommon.sqlapache-airflow-providers-openlineageopenlineageThe changelog for the provider package can be found in thechangelog.
Package: apache-airflow-providers-sqlite
Packageapache-airflow-providers-sqliteRelease:3.7.1SQLiteProvider packageThis is a provider package forsqliteprovider. All classes for this provider package
are inairflow.providers.sqlitepython package.You can find package information and changelog for the provider
in thedocumentation.InstallationYou can install this package on top of an existing Airflow 2 installation (seeRequirementsbelow
for the minimum Airflow version supported) viapip installapache-airflow-providers-sqliteThe package supports the following python versions: 3.8,3.9,3.10,3.11RequirementsPIP packageVersion requiredapache-airflow>=2.6.0apache-airflow-providers-common-sql>=1.3.1Cross provider package dependenciesThose are dependencies that might be needed in order to use all the features of the package.
You need to install the specified provider packages in order to use them.You can install such cross-provider dependencies when installing from PyPI. For example:pipinstallapache-airflow-providers-sqlite[common.sql]Dependent packageExtraapache-airflow-providers-common-sqlcommon.sqlThe changelog for the provider package can be found in thechangelog.
Package: apache-airflow-providers-ssh
Packageapache-airflow-providers-sshRelease:3.10.1Secure Shell (SSH)Provider packageThis is a provider package forsshprovider. All classes for this provider package
are inairflow.providers.sshpython package.You can find package information and changelog for the provider
in thedocumentation.InstallationYou can install this package on top of an existing Airflow 2 installation (seeRequirementsbelow
for the minimum Airflow version supported) viapip installapache-airflow-providers-sshThe package supports the following python versions: 3.8,3.9,3.10,3.11RequirementsPIP packageVersion requiredapache-airflow>=2.6.0paramiko>=2.6.0sshtunnel>=0.3.2The changelog for the provider package can be found in thechangelog.
Package: apache-airflow-providers-tableau
Packageapache-airflow-providers-tableauRelease:4.4.2TableauProvider packageThis is a provider package fortableauprovider. All classes for this provider package
are inairflow.providers.tableaupython package.You can find package information and changelog for the provider
in thedocumentation.InstallationYou can install this package on top of an existing Airflow 2 installation (seeRequirementsbelow
for the minimum Airflow version supported) viapip installapache-airflow-providers-tableauThe package supports the following python versions: 3.8,3.9,3.10,3.11RequirementsPIP packageVersion requiredapache-airflow>=2.6.0tableauserverclientThe changelog for the provider package can be found in thechangelog.
Package: apache-airflow-providers-tabular
Packageapache-airflow-providers-tabularRelease:1.4.1TabularProvider packageThis is a provider package fortabularprovider. All classes for this provider package
are inairflow.providers.tabularpython package.You can find package information and changelog for the provider
in thedocumentation.InstallationYou can install this package on top of an existing Airflow 2 installation (seeRequirementsbelow
for the minimum Airflow version supported) viapip installapache-airflow-providers-tabularThe package supports the following python versions: 3.8,3.9,3.10,3.11RequirementsPIP packageVersion requiredapache-airflow>=2.6.0The changelog for the provider package can be found in thechangelog.
Package: apache-airflow-providers-telegram
Packageapache-airflow-providers-telegramRelease:4.3.1TelegramProvider packageThis is a provider package fortelegramprovider. All classes for this provider package
are inairflow.providers.telegrampython package.You can find package information and changelog for the provider
in thedocumentation.InstallationYou can install this package on top of an existing Airflow 2 installation (seeRequirementsbelow
for the minimum Airflow version supported) viapip installapache-airflow-providers-telegramThe package supports the following python versions: 3.8,3.9,3.10,3.11RequirementsPIP packageVersion requiredapache-airflow>=2.6.0python-telegram-bot>=20.2The changelog for the provider package can be found in thechangelog.
Package: apache-airflow-providers-teradata
Packageapache-airflow-providers-teradataRelease:2.0.0TeradataProvider packageThis is a provider package forteradataprovider. All classes for this provider package
are inairflow.providers.teradatapython package.You can find package information and changelog for the provider
in thedocumentation.InstallationYou can install this package on top of an existing Airflow 2 installation (seeRequirementsbelow
for the minimum Airflow version supported) viapip installapache-airflow-providers-teradataThe package supports the following python versions: 3.8,3.9,3.10,3.11RequirementsPIP packageVersion requiredapache-airflow>=2.6.0apache-airflow-providers-common-sql>=1.3.1teradatasqlalchemy>=17.20.0.0teradatasql>=17.20.0.28Cross provider package dependenciesThose are dependencies that might be needed in order to use all the features of the package.
You need to install the specified provider packages in order to use them.You can install such cross-provider dependencies when installing from PyPI. For example:pipinstallapache-airflow-providers-teradata[common.sql]Dependent packageExtraapache-airflow-providers-common-sqlcommon.sqlThe changelog for the provider package can be found in thechangelog.
Package: apache-airflow-providers-trino
Packageapache-airflow-providers-trinoRelease:5.6.2TrinoProvider packageThis is a provider package fortrinoprovider. All classes for this provider package
are inairflow.providers.trinopython package.You can find package information and changelog for the provider
in thedocumentation.InstallationYou can install this package on top of an existing Airflow 2 installation (seeRequirementsbelow
for the minimum Airflow version supported) viapip installapache-airflow-providers-trinoThe package supports the following python versions: 3.8,3.9,3.10,3.11RequirementsPIP packageVersion requiredapache-airflow>=2.6.0apache-airflow-providers-common-sql>=1.3.1pandas>=1.2.5trino>=0.318.0Cross provider package dependenciesThose are dependencies that might be needed in order to use all the features of the package.
You need to install the specified provider packages in order to use them.You can install such cross-provider dependencies when installing from PyPI. For example:pipinstallapache-airflow-providers-trino[common.sql]Dependent packageExtraapache-airflow-providers-common-sqlcommon.sqlapache-airflow-providers-googlegoogleapache-airflow-providers-openlineageopenlineageThe changelog for the provider package can be found in thechangelog.
Package: apache-airflow-providers-vertica
Packageapache-airflow-providers-verticaRelease:3.7.1VerticaProvider packageThis is a provider package forverticaprovider. All classes for this provider package
are inairflow.providers.verticapython package.You can find package information and changelog for the provider
in thedocumentation.InstallationYou can install this package on top of an existing Airflow 2 installation (seeRequirementsbelow
for the minimum Airflow version supported) viapip installapache-airflow-providers-verticaThe package supports the following python versions: 3.8,3.9,3.10,3.11RequirementsPIP packageVersion requiredapache-airflow>=2.6.0apache-airflow-providers-common-sql>=1.3.1vertica-python>=0.5.1Cross provider package dependenciesThose are dependencies that might be needed in order to use all the features of the package.
You need to install the specified provider packages in order to use them.You can install such cross-provider dependencies when installing from PyPI. For example:pipinstallapache-airflow-providers-vertica[common.sql]Dependent packageExtraapache-airflow-providers-common-sqlcommon.sqlThe changelog for the provider package can be found in thechangelog.
Package: apache-airflow-providers-weaviate
Packageapache-airflow-providers-weaviateRelease:1.3.2WeaviateProvider packageThis is a provider package forweaviateprovider. All classes for this provider package
are inairflow.providers.weaviatepython package.You can find package information and changelog for the provider
in thedocumentation.InstallationYou can install this package on top of an existing Airflow 2 installation (seeRequirementsbelow
for the minimum Airflow version supported) viapip installapache-airflow-providers-weaviateThe package supports the following python versions: 3.8,3.9,3.10,3.11RequirementsPIP packageVersion requiredapache-airflow>=2.6.0weaviate-client>=3.24.2pandas>=1.2.5The changelog for the provider package can be found in thechangelog.
Package: apache-airflow-providers-y42demo2
Packageapache-airflow-providers-y42demoRelease:1.0.32Provider packageThis is a provider package fory42demoprovider. All classes for this provider package
are inairflow.providers.y42demopython package.InstallationYou can install this package on top of an existing Airflow 2 installation (seeRequirementsbelow
for the minimum Airflow version supported) viapip installapache-airflow-providers-y42demoThe package supports the following python versions: 3.9,3.10
Package: apache-airflow-providers-yandex
Packageapache-airflow-providers-yandexRelease:3.9.0This package is for Yandex, including:Yandex.CloudProvider packageThis is a provider package foryandexprovider. All classes for this provider package
are inairflow.providers.yandexpython package.You can find package information and changelog for the provider
in thedocumentation.InstallationYou can install this package on top of an existing Airflow 2 installation (seeRequirementsbelow
for the minimum Airflow version supported) viapip installapache-airflow-providers-yandexThe package supports the following python versions: 3.8,3.9,3.10,3.11RequirementsPIP packageVersion requiredapache-airflow>=2.6.0yandexcloud>=0.228.0The changelog for the provider package can be found in thechangelog.
Package: apache-airflow-providers-zendesk
Packageapache-airflow-providers-zendeskRelease:4.6.0ZendeskProvider packageThis is a provider package forzendeskprovider. All classes for this provider package
are inairflow.providers.zendeskpython package.You can find package information and changelog for the provider
in thedocumentation.InstallationYou can install this package on top of an existing Airflow 2 installation (seeRequirementsbelow
for the minimum Airflow version supported) viapip installapache-airflow-providers-zendeskThe package supports the following python versions: 3.8,3.9,3.10,3.11RequirementsPIP packageVersion requiredapache-airflow>=2.6.0zenpy>=2.0.24The changelog for the provider package can be found in thechangelog.
Package: apache-bookkeeper-client
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements.  See the NOTICE file
distributed with this work for additional information
regarding copyright ownership.  The ASF licenses this file
to you under the Apache License, Version 2.0 (the
“License”); you may not use this file except in compliance
with the License.  You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
“AS IS” BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied.  See the License for the
specific language governing permissions and limitations
under the License.Python Client for Apache BookKeeperApache BookKeeper is a scalable, fault tolerant and low latency storage service optimized for append-only workloads.
Package: apache-dolphinscheduler
PyDolphinSchedulerPyDolphinScheduleris python API forApache DolphinScheduler,
which allow you definition your workflow by python code, aka workflow-as-codes.Quick StartVersion CompatibilityAt Nov 7, 2022 we seperated PyDolphinScheduler from DolphinScheduler, and the version of PyDolphinScheduler 4.0.0
can match multiple versions of DolphinScheduler, for more details, please refer toversionInstallation# Installpython-mpipinstallapache-dolphinscheduler# Verify installation is successful, it will show the version of apache-dolphinscheduler, here we use 0.1.0 as examplepydolphinschedulerversion# 0.1.0NOTE: package apache-dolphinscheduler not work on above Python version 3.10(including itself) in Window operating system
due to dependencepy4jnot work on those environments.Here we show you how to install and run a simple example of PyDolphinSchedulerStart DolphinSchedulerThere are many ways to start DolphinScheduler, here we use docker to start and run it as a standalone server.# Change the version of dolphinscheduler to the version you want to use, here we use 3.1.1 as exampleDOLPHINSCHEDULER_VERSION=3.1.1
dockerrun--namedolphinscheduler-standalone-server-p12345:12345-p25333:25333-dapache/dolphinscheduler-standalone-server:"${DOLPHINSCHEDULER_VERSION}"After the container is started, you can access the DolphinScheduler UI viahttp://localhost:12345/dolphinscheduler.
For more way to start DolphinScheduler and the more detail about DolphinScheduler, please refer toDolphinSchedulerRun a simple exampleWe have many examples inexamplesdirectory, we here pick up a typical one
to show how to run it.# Get the latest code of example from githubcurlhttps://raw.githubusercontent.com/apache/dolphinscheduler-sdk-python/main/src/pydolphinscheduler/examples/tutorial.py-o./tutorial.py# Change tenant to real exists tenant in the host your DolphinScheduler running, by any editor you like# Run the examplepython./tutorial.pyNOTICE: Since Apache DolphinScheduler's tenant is requests while running command, you have to change
tenant value in file tutorial.py. The default value istenant_exists, change it to username exists your host.After that, a new workflow will be created by PyDolphinScheduler, and you can see it in DolphinScheduler web
UI's Project Management page. It will trigger the workflow automatically, so you can see the workflow running
in DolphinScheduler web UI's Workflow Instance page too. For more detail about any function about DolphinScheduler
Project Management, please refer toDolphinScheduler WorkflowDocumentationFor full documentation visitdocument. This
documentation is generated from this repository so please raise issues or pull requests for any additions, corrections, or clarifications.ContributingIf you would like to contribute, check out theopen issues on GitHub.
You can also see the guide tocontributing.ReleaseFollow thereleaseguide to release a new version of PyDolphinScheduler.