Repository: pass-culture/data-gcp
Pull request #4102: (DE) dbt(feat): full refresh bookable venue
Author: login: vbusson-pass
Created: 2025-05-06T15:00:20Z | Merged: 2025-05-13T09:33:39Z

# PR template

## Select PR template in preview mode

* [Hotfix](?expand=1&template=Hotfix_template.md)
* [DA](?expand=1&template=DA_ticket_template.md)
* [DE](?expand=1&template=DE_ticket_template.md)
* [DS](?expand=1&template=DS_ticket_template.md)
* [MEP](?expand=1&template=MEP_template.md)

## PR title format (except for MEP)

There is a linter on the PR title format. Please respect the following format:

<details>
<summary>(ticket) type(topic): comment</summary>

* ticket surrounded by parenthesis, with optionnaly a hyphen followed by one or more digits (e.g., -1234). The first part must be one of the following strings:
  * DA
  * DE
  * AE
  * DS
  * HF
  * BSR
  * PC

* type :
The second part to specify the type of change one of the following :
  * build
  * lint
  * ci
  * docs
  * feat
  * fix
  * perf
  * refactor
  * test
  * chore
  * dbt

* topic within parenthesis: 1 word e.g., (dag)

* comment: tell us your life

examples:

* :white_check_mark: (DE-124) refactor(firebase): update source field
* :x: (DE-124) refactor (firebase): update source field **(space between type and topic)**
* :x: (DE-124) airflow(firebase): update source fiedd in DAG **(wrong type)**
* :x: (DE-124) (DE-124) refactor(firebase refacto): update source field **(topic in two words)**
* :white_check_mark: (BSR) docs(github): add PR title valid format in template

</details>

<!-- Markdown tips:

To tick boxe replace [ ] with [x]

 -->

 <!--
TO DO: check with repo admin : [https://passculture.atlassian.net/browse/PC-<num>](https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/managing-repository-settings/configuring-autolinks-to-reference-external-resources)

JIRA-replace_with_ticket_number

[Notion-link](paste within parenthesis)
-->



Diff:
diff --git a/orchestration/dags/data_gcp_dbt/dbt_project.yml b/orchestration/dags/data_gcp_dbt/dbt_project.yml
index b28213277a..c85da15bd1 100644
--- a/orchestration/dags/data_gcp_dbt/dbt_project.yml
+++ b/orchestration/dags/data_gcp_dbt/dbt_project.yml
@@ -124,6 +124,7 @@ data_tests:
 
 vars:
   ENV_SHORT_NAME: dev
+  pass_start_date: "2018-01-01"
   CI_MATERIALIZATION: none
   region_name: region-europe-west1
   mute_ensure_materialization_override: true
diff --git a/orchestration/dags/data_gcp_dbt/models/intermediate/history/int_history__bookable_venue.sql b/orchestration/dags/data_gcp_dbt/models/intermediate/history/int_history__bookable_venue.sql
index a9a25f2538..b58073accb 100644
--- a/orchestration/dags/data_gcp_dbt/models/intermediate/history/int_history__bookable_venue.sql
+++ b/orchestration/dags/data_gcp_dbt/models/intermediate/history/int_history__bookable_venue.sql
@@ -9,38 +9,60 @@
 
 
 with
+    {% if not is_incremental() %}
+        date_bounds as (
+            select
+                date('{{ var("pass_start_date") }}') as start_date,
+                date('{{ ds() }}') as end_date
+        ),
+    {% endif %}
+
+    dates as (
+        {% if is_incremental() %}select date('{{ ds() }}') as partition_date
+        {% else %}
+            select day as partition_date
+            from
+                date_bounds,
+                unnest(generate_date_array(start_date, end_date, interval 1 day)) as day
+        {% endif %}
+    ),
+
     all_bookable_data as (
         select
             o.venue_id,
             v.venue_managing_offerer_id as offerer_id,
-            date('{{ ds() }}') as partition_date,
+            d.partition_date,
             'individual' as offer_type,
             count(distinct s.offer_id) as total_bookable_offers
-        from {{ ref("snapshot__bookable_offer") }} as s
+        from dates as d
+        inner join
+            {{ ref("snapshot__bookable_offer") }} as s
+            on date(s.dbt_valid_from) <= d.partition_date
+            and (s.dbt_valid_to is null or date(s.dbt_valid_to) > d.partition_date)
         inner join {{ ref("int_applicative__offer") }} as o using (offer_id)
         inner join
             {{ source("raw", "applicative_database_venue") }} as v using (venue_id)
-        where
-            date('{{ ds() }}') >= date(s.dbt_valid_from)
-            and (s.dbt_valid_to is null or date('{{ ds() }}') <= date(s.dbt_valid_to))
         group by venue_id, offerer_id, partition_date, offer_type
+
         union all
+
         select
             o.venue_id,
             v.venue_managing_offerer_id as offerer_id,
-            date('{{ ds() }}') as partition_date,
+            d.partition_date,
             'collective' as offer_type,
             count(distinct sb.collective_offer_id) as total_bookable_offers
-        from {{ ref("snapshot__bookable_collective_offer") }} as sb
+        from dates as d
+        inner join
+            {{ ref("snapshot__bookable_collective_offer") }} as sb
+            on date(sb.dbt_valid_from) <= d.partition_date
+            and (sb.dbt_valid_to is null or date(sb.dbt_valid_to) > d.partition_date)
         inner join
             {{ ref("int_applicative__collective_offer") }} as o using (
                 collective_offer_id
             )
         inner join
             {{ source("raw", "applicative_database_venue") }} as v using (venue_id)
-        where
-            date('{{ ds() }}') >= date(sb.dbt_valid_from)
-            and (sb.dbt_valid_to is null or date('{{ ds() }}') <= date(sb.dbt_valid_to))
         group by venue_id, offerer_id, partition_date, offer_type
     ),
 
@@ -49,8 +71,8 @@ with
             venue_id,
             offerer_id,
             partition_date,
-            individual as total_individual_bookable_offers,
-            collective as total_collective_bookable_offers
+            coalesce(individual, 0) as total_individual_bookable_offers,
+            coalesce(collective, 0) as total_collective_bookable_offers
         from
             all_bookable_data pivot (
                 sum(total_bookable_offers) for offer_type
@@ -62,8 +84,8 @@ select
     venue_id,
     offerer_id,
     partition_date,
-    coalesce(total_individual_bookable_offers, 0) as total_individual_bookable_offers,
-    coalesce(total_collective_bookable_offers, 0) as total_collective_bookable_offers,
-    coalesce(total_individual_bookable_offers, 0)
-    + coalesce(total_collective_bookable_offers, 0) as total_bookable_offers
+    total_individual_bookable_offers,
+    total_collective_bookable_offers,
+    total_individual_bookable_offers
+    + total_collective_bookable_offers as total_bookable_offers
 from pivoted_data


Reviews:
user: login: jarbelot-pass
state: DISMISSED
author_association: CONTRIBUTOR
submitted_at: 2025-05-06T15:02:49Z
user: login: lrainteau-pass
state: DISMISSED
author_association: COLLABORATOR
submitted_at: 2025-05-13T08:53:39Z
user: login: vbusson-pass
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-13T08:58:44Z
user: login: vbusson-pass
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-13T08:59:45Z

Review comments:
diff_hunk: @@ -9,48 +9,70 @@
 
 
 with
+    {% if not is_incremental() %}
+        date_bounds as (
+            select
+                date('{{ var("pass_start_date") }}') as start_date,
+                date('{{ ds() }}') as end_date
+        ),
+    {% endif %}
+
+    dates as (
+        {% if is_incremental() %}select date('{{ ds() }}') as partition_date
+        {% else %}
+            select day as partition_date
+            from
+                date_bounds,
+                unnest(generate_date_array(start_date, end_date, interval 1 day)) as day
+        {% endif %}
+    ),
+
     all_bookable_data as (
         select
             o.venue_id,
             v.venue_managing_offerer_id as offerer_id,
-            date('{{ ds() }}') as partition_date,
+            d.partition_date,
             'individual' as offer_type,
             count(distinct s.offer_id) as total_bookable_offers
-        from {{ ref("snapshot__bookable_offer") }} as s
+        from dates as d
+        inner join
+            {{ ref("snapshot__bookable_offer") }} as s
+            on date(s.dbt_valid_from) <= d.partition_date
+            and (s.dbt_valid_to is null or date(s.dbt_valid_to) > d.partition_date)
         inner join {{ ref("int_applicative__offer") }} as o using (offer_id)
         inner join
             {{ source("raw", "applicative_database_venue") }} as v using (venue_id)
-        where
-            date('{{ ds() }}') >= date(s.dbt_valid_from)
-            and (s.dbt_valid_to is null or date('{{ ds() }}') <= date(s.dbt_valid_to))
-        group by venue_id, offerer_id, partition_date, offer_type
+        group by 1, 2, 3, 4
path: orchestration/dags/data_gcp_dbt/models/intermediate/history/int_history__bookable_venue.sql
user: login: lrainteau-pass
body: mieux d'avoir le nom des colonnes ici plutôt que 1,2,3,4
created_at: 2025-05-13T08:50:59Z
author_association: COLLABORATOR
diff_hunk: @@ -9,48 +9,70 @@
 
 
 with
+    {% if not is_incremental() %}
+        date_bounds as (
+            select
+                date('{{ var("pass_start_date") }}') as start_date,
+                date('{{ ds() }}') as end_date
+        ),
+    {% endif %}
+
+    dates as (
+        {% if is_incremental() %}select date('{{ ds() }}') as partition_date
+        {% else %}
+            select day as partition_date
+            from
+                date_bounds,
+                unnest(generate_date_array(start_date, end_date, interval 1 day)) as day
+        {% endif %}
+    ),
+
     all_bookable_data as (
         select
             o.venue_id,
             v.venue_managing_offerer_id as offerer_id,
-            date('{{ ds() }}') as partition_date,
+            d.partition_date,
             'individual' as offer_type,
             count(distinct s.offer_id) as total_bookable_offers
-        from {{ ref("snapshot__bookable_offer") }} as s
+        from dates as d
+        inner join
+            {{ ref("snapshot__bookable_offer") }} as s
+            on date(s.dbt_valid_from) <= d.partition_date
+            and (s.dbt_valid_to is null or date(s.dbt_valid_to) > d.partition_date)
         inner join {{ ref("int_applicative__offer") }} as o using (offer_id)
         inner join
             {{ source("raw", "applicative_database_venue") }} as v using (venue_id)
-        where
-            date('{{ ds() }}') >= date(s.dbt_valid_from)
-            and (s.dbt_valid_to is null or date('{{ ds() }}') <= date(s.dbt_valid_to))
-        group by venue_id, offerer_id, partition_date, offer_type
+        group by 1, 2, 3, 4
+
         union all
+
         select
             o.venue_id,
             v.venue_managing_offerer_id as offerer_id,
-            date('{{ ds() }}') as partition_date,
+            d.partition_date,
             'collective' as offer_type,
             count(distinct sb.collective_offer_id) as total_bookable_offers
-        from {{ ref("snapshot__bookable_collective_offer") }} as sb
+        from dates as d
+        inner join
+            {{ ref("snapshot__bookable_collective_offer") }} as sb
+            on date(sb.dbt_valid_from) <= d.partition_date
+            and (sb.dbt_valid_to is null or date(sb.dbt_valid_to) > d.partition_date)
         inner join
             {{ ref("int_applicative__collective_offer") }} as o using (
                 collective_offer_id
             )
         inner join
             {{ source("raw", "applicative_database_venue") }} as v using (venue_id)
-        where
-            date('{{ ds() }}') >= date(sb.dbt_valid_from)
-            and (sb.dbt_valid_to is null or date('{{ ds() }}') <= date(sb.dbt_valid_to))
-        group by venue_id, offerer_id, partition_date, offer_type
+        group by 1, 2, 3, 4
path: orchestration/dags/data_gcp_dbt/models/intermediate/history/int_history__bookable_venue.sql
user: login: lrainteau-pass
body: pareil la
created_at: 2025-05-13T08:52:43Z
author_association: COLLABORATOR
diff_hunk: @@ -9,48 +9,70 @@
 
 
 with
+    {% if not is_incremental() %}
+        date_bounds as (
+            select
+                date('{{ var("pass_start_date") }}') as start_date,
+                date('{{ ds() }}') as end_date
+        ),
+    {% endif %}
+
+    dates as (
+        {% if is_incremental() %}select date('{{ ds() }}') as partition_date
+        {% else %}
+            select day as partition_date
+            from
+                date_bounds,
+                unnest(generate_date_array(start_date, end_date, interval 1 day)) as day
+        {% endif %}
+    ),
+
     all_bookable_data as (
         select
             o.venue_id,
             v.venue_managing_offerer_id as offerer_id,
-            date('{{ ds() }}') as partition_date,
+            d.partition_date,
             'individual' as offer_type,
             count(distinct s.offer_id) as total_bookable_offers
-        from {{ ref("snapshot__bookable_offer") }} as s
+        from dates as d
+        inner join
+            {{ ref("snapshot__bookable_offer") }} as s
+            on date(s.dbt_valid_from) <= d.partition_date
+            and (s.dbt_valid_to is null or date(s.dbt_valid_to) > d.partition_date)
         inner join {{ ref("int_applicative__offer") }} as o using (offer_id)
         inner join
             {{ source("raw", "applicative_database_venue") }} as v using (venue_id)
-        where
-            date('{{ ds() }}') >= date(s.dbt_valid_from)
-            and (s.dbt_valid_to is null or date('{{ ds() }}') <= date(s.dbt_valid_to))
-        group by venue_id, offerer_id, partition_date, offer_type
+        group by 1, 2, 3, 4
+
         union all
+
         select
             o.venue_id,
             v.venue_managing_offerer_id as offerer_id,
-            date('{{ ds() }}') as partition_date,
+            d.partition_date,
             'collective' as offer_type,
             count(distinct sb.collective_offer_id) as total_bookable_offers
-        from {{ ref("snapshot__bookable_collective_offer") }} as sb
+        from dates as d
+        inner join
+            {{ ref("snapshot__bookable_collective_offer") }} as sb
+            on date(sb.dbt_valid_from) <= d.partition_date
+            and (sb.dbt_valid_to is null or date(sb.dbt_valid_to) > d.partition_date)
         inner join
             {{ ref("int_applicative__collective_offer") }} as o using (
                 collective_offer_id
             )
         inner join
             {{ source("raw", "applicative_database_venue") }} as v using (venue_id)
-        where
-            date('{{ ds() }}') >= date(sb.dbt_valid_from)
-            and (sb.dbt_valid_to is null or date('{{ ds() }}') <= date(sb.dbt_valid_to))
-        group by venue_id, offerer_id, partition_date, offer_type
+        group by 1, 2, 3, 4
path: orchestration/dags/data_gcp_dbt/models/intermediate/history/int_history__bookable_venue.sql
user: login: vbusson-pass
body: ```suggestion
        group by venue_id, offerer_id, partition_date, offer_type
```
created_at: 2025-05-13T08:58:43Z
author_association: COLLABORATOR
diff_hunk: @@ -9,48 +9,70 @@
 
 
 with
+    {% if not is_incremental() %}
+        date_bounds as (
+            select
+                date('{{ var("pass_start_date") }}') as start_date,
+                date('{{ ds() }}') as end_date
+        ),
+    {% endif %}
+
+    dates as (
+        {% if is_incremental() %}select date('{{ ds() }}') as partition_date
+        {% else %}
+            select day as partition_date
+            from
+                date_bounds,
+                unnest(generate_date_array(start_date, end_date, interval 1 day)) as day
+        {% endif %}
+    ),
+
     all_bookable_data as (
         select
             o.venue_id,
             v.venue_managing_offerer_id as offerer_id,
-            date('{{ ds() }}') as partition_date,
+            d.partition_date,
             'individual' as offer_type,
             count(distinct s.offer_id) as total_bookable_offers
-        from {{ ref("snapshot__bookable_offer") }} as s
+        from dates as d
+        inner join
+            {{ ref("snapshot__bookable_offer") }} as s
+            on date(s.dbt_valid_from) <= d.partition_date
+            and (s.dbt_valid_to is null or date(s.dbt_valid_to) > d.partition_date)
         inner join {{ ref("int_applicative__offer") }} as o using (offer_id)
         inner join
             {{ source("raw", "applicative_database_venue") }} as v using (venue_id)
-        where
-            date('{{ ds() }}') >= date(s.dbt_valid_from)
-            and (s.dbt_valid_to is null or date('{{ ds() }}') <= date(s.dbt_valid_to))
-        group by venue_id, offerer_id, partition_date, offer_type
+        group by 1, 2, 3, 4
path: orchestration/dags/data_gcp_dbt/models/intermediate/history/int_history__bookable_venue.sql
user: login: vbusson-pass
body: ```suggestion
        group by venue_id, offerer_id, partition_date, offer_type
```
created_at: 2025-05-13T08:59:45Z
author_association: COLLABORATOR

PR comments:
user: login: sonarqubecloud[bot]
created_at: 2025-05-13T09:03:58Z
body: ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=pass-culture_data-gcp&pullRequest=4102) **Quality Gate passed**  
Issues  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 New issues](https://sonarcloud.io/project/issues?id=pass-culture_data-gcp&pullRequest=4102&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png '') [0 Accepted issues](https://sonarcloud.io/project/issues?id=pass-culture_data-gcp&pullRequest=4102&issueStatuses=ACCEPTED)

Measures  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=pass-culture_data-gcp&pullRequest=4102&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=pass-culture_data-gcp&pullRequest=4102&metric=new_coverage&view=list)  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=pass-culture_data-gcp&pullRequest=4102&metric=new_duplicated_lines_density&view=list)  
  
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=pass-culture_data-gcp&pullRequest=4102)


author_association: NONE