Repository: carbonfact/lea
Pull request #100: Update documentation of skipping feature
Author: login: Emilien-Foissotte
Created: 2025-03-21T08:12:05Z | Merged: 2025-03-27T10:00:53Z

Hi @MaxHalford 👋 

Following our discussion about skipping feature, I've updated the associated section of the README 📖  

I materialized this example with a real SQL script example in `jaffle_shop` to illustrate the skipping feature benefits  during WAP cycle. 

Please feel free to update examples and let me know openly your feedback ! 


Diff:
diff --git a/README.md b/README.md
index 0abd403..60a5e30 100644
--- a/README.md
+++ b/README.md
@@ -142,7 +142,12 @@ You have access to an `env` variable within the template context, which is simpl
 
 ### Development vs. production
 
-By default, lea appends a `_<user>` suffix to schema names. This way you can have a development schema and a production schema. Use the `--production` flag to disable this behavior.
+By default, lea creates an isolation layer with production. The way this is done depends on your warehouse:
+
+- BigQuery : by appending a `_<user>` suffix to schema names
+- DuckDB : by adding a suffix `_<user>` to database file.
+
+In other words, a development environment is used by default. Use the `--production` flag when executing `lea run` to disable this behaviour, and instead target the product environment.
 
 ```sh
 lea run --production
@@ -267,20 +272,24 @@ You may decide to run all scripts without executing tests, which is obviously no
 lea run --unselect tests/
 ```
 
-### Skipping unmodified scripts
+### Skipping unmodified scripts during development
+
+lea doesn't run scripts that have been modified before the last time the associated table was materialized, given that audit
+tables still exists (e.g you are still under **Write** and **Audit** loop of the WAP Pattern).
 
-lea doesn't run scripts that have been modified before the last time the associated table was materialized. For instance:
+For instance:
 
-1. You add a script named `core/expenses.sql`
-2. You execute `lea run --select core.expenses`
-3. `core__expenses` is materialized in your data warehouse
-4. You execute `lea run`
-5. The `core/expenses.sql` script is skipped because it was modified before `core__expenses` was last materialized
-6. You edit `core/expenses.sql`
-7. You execute `lea run`
-8. The `core/expenses.sql` script is run because it was modified after `core__expenses` was last materialized
+1. You execute `lea run` to sync all tables from sources, no errors, all tables are materialized.
+1. You modify a script named `core/expenses.sql` depending on `staging/customers.sql` and `staging/orders.sql`
+1. You execute `lea run core.expenses+` to run again all impacted tables
+1. `core__expenses___audit` is materialized in your data warehouse but the `-- #NO_NULLS` assertion test on a column fails
+1. After reviewing data in `core__expenses___audit`, you edit and fix `core/expenses.sql` to filter out results where NULLs are appearing
+1. You execute `lea run`
+1. The `staging/customers.sql` and `staging/orders.sql` scripts are skipped because they were modified before `staging__customers` and `staging__orders` was last materialized
+1. The `core/expenses.sql` script is run because it was modified after `core__expenses` was last materialized
+1. All audit tables are wipped out from database as the whole DAG has run successfully ! 🎉
 
-You can disable this default behavior to guarantee each script runs:
+**NB**: You can disable this default behavior to guarantee each script runs:
 
 ```sh
 lea run --restart
diff --git a/examples/jaffle_shop/README.md b/examples/jaffle_shop/README.md
index 1eaded4..a66a518 100644
--- a/examples/jaffle_shop/README.md
+++ b/examples/jaffle_shop/README.md
@@ -43,9 +43,92 @@ There are a couple of cool things:
 
 1. The staging schema is populated using SQL scripts and native DuckDB parsing of CSV files.
 2. The `core.orders` table is created using a Jinja SQL script. lea will automatically run the script through Jinja, and then execute the resulting SQL.
-3. Skip feature (TBC)
+3. Skip feature can help fasten development cycle during WAP pattern. If a table is not passing through audit, all materialized tables won't be run again if the associated SQL script has'nt changed.
+   If the script has changed, the audit table will be generated again, and all it's related childs in the DAG.
 
-Now, if you were running in production and not in development mode, you would add a `--production` flag to each command:
+Let's take the example given in [README.md](README.md).
+
+- Tables are materialized since you ran earlier `lea run`
+
+## Write
+
+- Add a new script `core/expenses.sql`
+
+```sh
+echo '''
+with customer_orders as (
+
+    select
+        customer_id,
+
+        min(order_date) as first_order,
+        max(order_date) as most_recent_order,
+        count(order_id) as number_of_orders
+    from staging.orders
+
+    group by customer_id
+
+),
+
+customer_payments as (
+
+    select
+        orders.customer_id,
+        sum(payments.amount) as total_amount
+
+    from staging.payments as payments
+
+    left join staging.orders as orders
+        on payments.order_id = orders.order_id
+
+    group by orders.customer_id
+
+),
+
+expenses as (
+    select
+    -- #UNIQUE
+        customers.customer_id,
+        customers.first_name,
+        customers.last_name,
+        customer_orders.first_order,
+        customer_orders.most_recent_order,
+        customer_orders.number_of_orders,
+        -- #NO_NULLS
+        customer_payments.total_amount as customer_lifetime_value
+    from staging.customers as customers --comment here
+    left join customer_orders  --comment here
+      on customers.customer_id = customer_orders.customer_id  --comment here
+    -- FROM customer_orders  --uncomment here
+    -- left join staging.customers as customers  --uncomment here
+    --     on  customer_orders.customer_id = customers.customer_id  --uncomment here
+    left join customer_payments
+        on customers.customer_id = customer_payments.customer_id
+)
+
+select * from expenses
+''' > scripts/core/expenses.sql
+```
+
+## Audit
+
+- Run the scripts `lea run` : `lea_duckdb_max.tests.core__expenses__customer_lifetime_value___no_nulls___audit` is failing ❌
+- Uncomment and comment lines to reverse the JOIN orders, and exclude customers absent from orders tables.
+
+```sh
+sed -i '' '/--comment here/s/^/--/' scripts/core/expenses.sql
+sed -i '' '/--uncomment here/s/-- //' scripts/core/expenses.sql
+```
+
+- Run again scripts, you should see that all stagings audit tables are not executed again.
+- `core.expenses` is executed as lea detected modification on the script
+- All tests are now passing 🎉
+- Audit tables are wiped out from development warehouse.
+
+## Publish
+
+- As all tests passed, tables are materialized in the development warehouse.
+- If you want now to run it against production and not development warehouse, you would add a `--production` flag to each command:
 
 ```sh
 lea run --production


Reviews:
user: login: MaxHalford
body: Great stuff, thanks for caring about the documentation 👍
state: APPROVED
author_association: MEMBER
submitted_at: 2025-03-21T10:27:09Z
user: login: Emilien-Foissotte
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-21T13:38:56Z

Review comments:
diff_hunk: @@ -142,7 +142,12 @@ You have access to an `env` variable within the template context, which is simpl
 
 ### Development vs. production
 
-By default, lea appends a `_<user>` suffix to schema names. This way you can have a development schema and a production schema. Use the `--production` flag to disable this behavior.
+By default, lea create an isolation layer with production in various way depending on your warehouse choicea :
path: README.md
user: login: MaxHalford
body: ```suggestion
By default, lea create an isolation layer with production. The way this is done depends on your warehouse:
```
created_at: 2025-03-21T10:24:02Z
author_association: MEMBER
diff_hunk: @@ -142,7 +142,12 @@ You have access to an `env` variable within the template context, which is simpl
 
 ### Development vs. production
 
-By default, lea appends a `_<user>` suffix to schema names. This way you can have a development schema and a production schema. Use the `--production` flag to disable this behavior.
+By default, lea create an isolation layer with production in various way depending on your warehouse choicea :
+
+- BigQuery : by appending a `_<user>` suffix to schema names
+- DuckDB : by adding a suffix `_<user>` to database file.
+
+This way you can have a development schema and a production schema. Use the `--production` flag to disable this behavior.
path: README.md
user: login: MaxHalford
body: ```suggestion
In other words, a development environment is used by default. Use the `--production` flag when executing `lea run` to disable this behaviour, and instead target the product environment.
```
created_at: 2025-03-21T10:25:32Z
author_association: MEMBER
diff_hunk: @@ -123,10 +123,12 @@ sed -i '' '/--uncomment here/s/-- //' scripts/core/expenses.sql
 - Run again scripts, you should see that all stagings audit tables are not executed again.
 - `core.expenses` is executed as lea detected modification on the script
 - All tests are now passing 🎉
+- Audit tables are wipped out from development warehouse.
path: examples/jaffle_shop/README.md
user: login: MaxHalford
body: ```suggestion
- Audit tables are wiped out from development warehouse.
```
created_at: 2025-03-21T10:25:44Z
author_association: MEMBER
diff_hunk: @@ -123,10 +123,12 @@ sed -i '' '/--uncomment here/s/-- //' scripts/core/expenses.sql
 - Run again scripts, you should see that all stagings audit tables are not executed again.
 - `core.expenses` is executed as lea detected modification on the script
 - All tests are now passing 🎉
+- Audit tables are wipped out from development warehouse.
 
 ## Publish
 
-Now, if you were running in production and not in development mode, you would add a `--production` flag to each command:
+- As all tests passed, tables are materialized in the development warehouse.
+- If you want now to run it against production and not development warehouse, you would add a `--production` flag to each command:
path: examples/jaffle_shop/README.md
user: login: MaxHalford
body: By the way, at Carbonfact we actually have a command to promote the development environment to production. This works by cloning the development tables to replace the production ones. At some point we'll move this logic to lea!
created_at: 2025-03-21T10:26:50Z
author_association: MEMBER
diff_hunk: @@ -123,10 +123,12 @@ sed -i '' '/--uncomment here/s/-- //' scripts/core/expenses.sql
 - Run again scripts, you should see that all stagings audit tables are not executed again.
 - `core.expenses` is executed as lea detected modification on the script
 - All tests are now passing 🎉
+- Audit tables are wipped out from development warehouse.
 
 ## Publish
 
-Now, if you were running in production and not in development mode, you would add a `--production` flag to each command:
+- As all tests passed, tables are materialized in the development warehouse.
+- If you want now to run it against production and not development warehouse, you would add a `--production` flag to each command:
path: examples/jaffle_shop/README.md
user: login: Emilien-Foissotte
body: Nice and interesting to know you have some ad-hoc commands to do this, thanks for sharing that insight !  

Indeed it would make it smoother and it would be useful to avoid running twice the audits when development stage is successful. (Especially when data sources are the same between `development` and `production`). It would definitely makes a difference on computation resources and time  ! 
created_at: 2025-03-21T13:38:56Z
author_association: CONTRIBUTOR

PR comments:
user: login: Emilien-Foissotte
created_at: 2025-03-27T09:13:43Z
body: I rebased PR after dependabot dependencies updates this week and signed commits to allow merging 👍 
author_association: CONTRIBUTOR
user: login: MaxHalford
created_at: 2025-03-27T10:00:58Z
body: Nice 👏
author_association: MEMBER