Package: astro-sdk-python
astroworkflows made easyAstro Python SDKis a Python SDK for rapid development of extract, transform, and load workflows inApache Airflow. It allows you to express your workflows as a set of data dependencies without having to worry about ordering and tasks. The Astro Python SDK is maintained byAstronomer.PrerequisitesApache Airflow >= 2.1.0.InstallThe Astro Python SDK is available atPyPI. Use the standard Pythoninstallation tools.To install a cloud-agnostic version of the SDK, run:pipinstallastro-sdk-pythonYou can also install dependencies for using the SDK with popular cloud providers:pipinstallastro-sdk-python[amazon,google,snowflake,postgres]QuickstartEnsure that your Airflow environment is set up correctly by running the following commands:exportAIRFLOW_HOME=`pwd`exportAIRFLOW__CORE__XCOM_BACKEND=astro.custom_backend.astro_custom_backend.AstroCustomXcomBackendexportAIRFLOW__ASTRO_SDK__STORE_DATA_LOCAL_DEV=trueairflowdbinitNote:AIRFLOW__CORE__ENABLE_XCOM_PICKLINGno longer needs to be enabled forastro-sdk-python. This functionality is now deprecated as our custom xcom backend handles serialization.TheAIRFLOW__ASTRO_SDK__STORE_DATA_LOCAL_DEVshould only be used for local development. TheXCom backend docsgive further details about how to set this up in non-local environments.Currently, custom XCom backends are limited to data types that are json serializable. Since Dataframes are not json serializable, we need to enable XCom pickling to store dataframes.The data format used by pickle is Python-specific. This has the advantage that there are no restrictions imposed by external standards such as JSON or XDR (which can’t represent pointer sharing); however it means that non-Python programs may not be able to reconstruct pickled Python objects.Read more:enable_xcom_picklingandpickle:Create a SQLite database for the example to run with:# The sqlite_default connection has different host for MAC vs. LinuxexportSQL_TABLE_NAME=`airflowconnectionsgetsqlite_default-oyaml|grephost|awk'{print $2}'`sqlite3"$SQL_TABLE_NAME""VACUUM;"Copy the following workflow into a file namedcalculate_popular_movies.pyand add it to thedagsdirectory of your Airflow project:https://github.com/astronomer/astro-sdk/blob/d5aa768b2d4bca72ef98f8d533fe3f99624b172f/example_dags/calculate_popular_movies.py#L1-L37Alternatively, you can downloadcalculate_popular_movies.pycurl-Ohttps://raw.githubusercontent.com/astronomer/astro-sdk/main/python-sdk/example_dags/calculate_popular_movies.pyRun the example DAG:airflowdagstestcalculate_popular_movies`date-Iseconds`Check the result of your DAG by running:sqlite3"$SQL_TABLE_NAME""select * from top_animation;"".exit"You should see the following output:$sqlite3"$SQL_TABLE_NAME""select * from top_animation;"".exit"ToyStory3(2010)|8.3
InsideOut(2015)|8.2
HowtoTrainYourDragon(2010)|8.1
Zootopia(2016)|8.1
HowtoTrainYourDragon2(2014)|7.9Supported technologiesDatabasesDatabricks DeltaGoogle BigQueryPostgresSnowflakeSQLiteAmazon RedshiftMicrosoft SQLDuckDBFile typesCSVJSONNDJSONParquetFile storesAmazon S3FilesystemGoogle GCSGoogle DriveSFTPFTPAzure WASBAzure WASBSAvailable operationsThe following are some key functions available in the SDK:load_file: Load a given file into a SQL tabletransform: Applies a SQL select statement to a source table and saves the result to a destination tabledrop_table: Drops a SQL tablerun_raw_sql: Run any SQL statement without handling its outputappend: Insert rows from the source SQL table into the destination SQL table, if there are no conflictsmerge: Insert rows from the source SQL table into the destination SQL table, depending on conflicts:ignore: Do not add rows that already existupdate: Replace existing rows with new onesexport_file: Export SQL table rows into a destination filedataframe: Export given SQL table into in-memory Pandas data-frameFor a full list of available operators, see theSDK reference documentation.DocumentationThe documentation is a work in progress--we aim to follow theDiátaxissystem:Getting Started Tutorial: A hands-on introduction to the Astro Python SDKHow-to guides: Simple step-by-step user guides to accomplish specific tasksReference guide: Commands, modules, classes and methodsExplanation: Clarification and discussion of key decisions when designing the projectChangelogThe Astro Python SDK follows semantic versioning for releases. Check thechangelogfor the latest changes.Release managementsTo learn more about our release philosophy and steps, seeManaging Releases.Contribution guidelinesAll contributions, bug reports, bug fixes, documentation improvements, enhancements, and ideas are welcome.Read theContribution Guidelinefor a detailed overview on how to contribute.Contributors and maintainers should abide by theContributor Code of Conduct.LicenseApache Licence 2.0