Package: astro-projects
astroworkflows made easyastroallows rapid and clean development of {Extract, Load, Transform} workflows using Python.
It helps DAG authors to achieve more with less code.
It is powered byApache Airflowand maintained byAstronomer.:warning:DisclaimerThis project's development status is alpha. In other words, it is not production-ready yet.
The interfaces may change. We welcome alpha users and brave souls to test it - any feedback is welcome.InstallAstrois available atPyPI. Use the standard Pythoninstallation tools.To install a cloud-agnostic version ofAstro, run:pip install astro-projectsIf using cloud providers, install using the optional dependencies of interest:pip install astro-projects[amazon,google,snowflake,postgres]Quick-startAfter installing Astro, copy the following example dagcalculate_popular_movies.pyto a local directory nameddags:fromdatetimeimportdatetimefromairflowimportDAGfromastroimportsqlasaqlfromastro.sql.tableimportTable@aql.transform()deftop_five_animations(input_table:Table):return"""SELECT Title, RatingFROM {{input_table}}WHERE Genre1=='Animation'ORDER BY Rating descLIMIT 5;"""withDAG("calculate_popular_movies",schedule_interval=None,start_date=datetime(2000,1,1),catchup=False,)asdag:imdb_movies=aql.load_file(path="https://raw.githubusercontent.com/astro-projects/astro/main/tests/data/imdb.csv",task_id="load_csv",output_table=Table(table_name="imdb_movies",database="sqlite",conn_id="sqlite_default"),)top_five_animations(input_table=imdb_movies,output_table=Table(table_name="top_animation",database="sqlite",conn_id="sqlite_default"),)Set up a local instance of Airflow by running:exportAIRFLOW_HOME=`pwd`exportAIRFLOW__CORE__ENABLE_XCOM_PICKLING=True

airflowdbinitCreate an SQLite database for the example to run with and run the DAG:# The sqlite_default connection has different host for MAC vs. LinuxexportSQL_TABLE_NAME=`airflowconnectionsgetsqlite_default-oyaml|grephost|awk'{print $2}'`sqlite3"$SQL_TABLE_NAME""VACUUM;"airflowdagstestcalculate_popular_movies`date-Iseconds`Check the top five animations calculated by your first Astro DAG by running:sqlite3"$SQL_TABLE_NAME""select * from top_animation;"".exit"You should see the following output:$sqlite3"$SQL_TABLE_NAME""select * from top_animation;"".exit"Toy Story 3 (2010)|8.3Inside Out (2015)|8.2How to Train Your Dragon (2010)|8.1Zootopia (2016)|8.1How to Train Your Dragon 2 (2014)|7.9RequirementsBecauseastrorelies on theTask Flow APIand
it depends on Apache Airflow >= 2.1.0.Supported technologiesDatabasesFile typesFile locationsGoogle BigQueryCSVAmazon S3PostgresJSONFilesystemSnowflakeNDJSONGoogle GCSSQLiteParquetAvailable operationsA summary of the currently available operations inastro. More details are available in thereference guide.load_file: load a given file into a SQL tabletransform: applies a SQL select statement to a source table and saves the result to a destination tabletruncate: remove all records from a SQL tablerun_raw_sql: run any SQL statement without handling its outputappend: insert rows from the source SQL table into the destination SQL table, if there are no conflictsmerge: insert rows from the source SQL table into the destination SQL table, depending on conflicts:ignore: do not add rows that already existupdate: replace existing rows with new onessave_file: export SQL table rows into a destination filedataframe: export given SQL table into in-memory Pandas data-framerender: given a directory containing SQL statements, dynamically create transform tasks within a DAGDocumentationThe documentation is a work in progress, and we aim to follow theDiátaxissystem:Tutorial: a hands-on introduction toastroHow-to guides: simple step-by-step user guides to accomplish specific tasksReference guide: commands, modules, classes and methodsExplanation: Clarification and discussion of key decisions when designing the project.ChangelogWe follow Semantic Versioning for releases. Check thechangelogfor the latest changes.Release ManagementsTo learn more about our release philosophy and steps, checkhereContribution GuidelinesAll contributions, bug reports, bug fixes, documentation improvements, enhancements, and ideas are welcome.Read theContribution Guidelinefor a detailed overview on how to contribute.As contributors and maintainers to this project, you should abide by theContributor Code of Conduct.LicenseApache Licence 2.0