Package: li-airflow-backfill-plugin
Li-Airflow-Backfill-PluginGitHub Link of this pageAirflow Backfill PluginThis is anAirflow Plugin. It provides full-featured UI and APIs for data backfills in Airflow with manageability and scalability.FeaturesWe want users to be able run backfills in a scheduled, managed, scalable, and robust way:Backfills can be triggered directly with desired optionsBackfills can be monitored and manageableBackfills are scalableResilient to Airflow system restarts and failuresAllow scheduled Dags to run while backfillingProper controlling of resources used by backfillsThese features can be easily added to an Airflow instances since it is an Airflow plugin.Quick StartLet's get started by running Airflow and backfill at local docker: (docker is required,refer to Airflow docfor more details in running Airflow in docker):# in project root folder# startdocker-composeup# stopdocker-composedownTo access Airflow web:http://localhost:8080(user/pass airflow:airflow)Backfill User DocIntroBackfill UIBackfill APIsSupported Airflow VersionThe supported Airflow version is 2.5.3.Other versions are exected to work. To quickly test other versions:first, make sure 2.5.3 works at your local docker, and backupdocker-compose.yaml in project root folderfetch and copythe docker-compose.yaml from the testing Airflow versionto the project rootadd the volumns values as the following to it: (refering to the docker-compose.yaml backuped above)volumes:
    - ${AIRFLOW_PROJ_DIR:-.}/dags:/opt/airflow/dags
    - ${AIRFLOW_PROJ_DIR:-.}/logs:/opt/airflow/logs
    - ${AIRFLOW_PROJ_DIR:-.}/plugins/linkedin/airflow/backfill:/opt/airflow/plugins/linkedin/airflow/backfillfollowQuick Startto run Airflow.Deploy Backfill Plugin to an Existing Airflow InstanceDeploy PluginOption 1: Drop files to Airflow plugins folderAsApache Airflow doc says, simply drop all the content in theplugins folder in the project rootto the $AIRFLOW_HOME/plugins folder of the Airflow instance.Restart of the Airflowmay be needed according the Airflow configto enable the backfill plugin.Option 2: Install from PyPiStarting from 1.0.2, Backfill plugin is available in PyPi. After installation, backfill lib will be installed and registered through entry_points insetup.py.pipinstallli-airflow-backfill-plugin==1.0.2Deploy System DagsSome Dags are needed to make backfill work. After enabling backfill plugin, drop all the content in thedags/backfill_dags folderto the configured Airflow Dags folder (default is $AIRFLOW_HOME/dags) of the Airflow instance.Restart of the Airflow is not needed.Plugin DevelopmentRun AirflowAfter making changes to the source code, you can run the Airflow in local docker as described inQuick Startto test. The logs will appear in the logs folder in the project root, and feel free to add testing Dags to thedags folder.Unit Testpytest is used to run unit tests in docker. The test source code is intests folderand the pytest configure ispytest.iniBuild Image once for all:# in project root folderdockerbuild-tairflow-backfill-plugin-tests-1-ftests.Dockerfile.Run Unit Test:# in project root folder./run_tests.shAdvanced TopicsFor detailed design, please refer to theDesign Doc.AssumptionsWriting files to Airflow Dags folderBy default, backfill Dag files will be created in dags/backfill_user_dags folder in workers.
This limitation may be lifted throughbackfill store customerization.Shallow copyThe backfill Dags are shallow copies from the origin Dags, which means if dependencies outside of the Dag definition file change while backfills are running, the actual behavior may change accordingly.DatabaseAbackfill tableis automactically created and leveraged to store backfill meta and status information in the default Airflow database.No other tables are created or modified by the backfill feature.Backfill Store CustomerizationBackfill Dag Id ConventionsThe Backfill Dag Ids are generated bybackfill store. By default, the Id will be origin Dag Id affixed with "backfill" and timestamp.The backfill Dag Id is customizable by setting AIRFLOW__LI_BACKFILL__BACKFILL_STORE env to new store class. For example:name: AIRFLOW__LI_BACKFILL__BACKFILL_STORE
value: 'airflow.providers.my_porvider.backfill.backfill_store.MyBackfillStore'Backfill Dags PersistenceThe backfill Dag files by default are persisted to dags/backfill_user_dags folder.The persistence is customizable, for example, to store Dags through APIs.SecurityAutheticationBackfill, both UI and APIs, is integrated into the existingAirflow authetication model, so they are autheticated as other Airflow UI and APIs. By default and in local docker Airflow instance, username and password are used to autheticate.Access ControlAirflow RBAC is supported throughpermission module.Backfill Menus Access: Backfill menu resources are defined and requires menu access permission to show the menu in the UIBackfill Access: Backfill resource can be create (create backfill), read (list backfill), and operate (cancel or delete backfill)Backfill Dag Access: Backfill Dags inherit Dag access permissions from the orgin DagsOrigin Dags to backfill: Must have can_edit permission for the origin Dag to start backfillBy default, all the backfill permissions are automatically granted to "User" and "Op" roles. This can be customerized through AIRFLOW__LI_BACKFILL__PERMITTED_ROLES. For example:name: AIRFLOW__LI_BACKFILL__PERMITTED_ROLES
value: 'first_role,second_role'