Repository: NYCPlanning/data-engineering
Pull request #1636: Lion ETL: Centerline & Face Code Fields
Author: login: sf-dcp
Created: 2025-04-28T21:55:56Z | Merged: 2025-05-01T20:16:00Z

Closes #1566.

2.2.1 in the [docs](https://nyco365.sharepoint.com/:w:/r/sites/NYCPLANNING/itd/edm/_layouts/15/Doc.aspx?sourcedoc=%7BD3194825-3026-4295-8A31-6FB030749AA1%7D&file=ETL_V8_02012024%20-%20Copy.docx&action=default&mobileredirect=true)

Latest build [here](https://github.com/NYCPlanning/data-engineering/actions/runs/14767666484/job/41462156546)

### What
Adds 4 types of fields in this PR:
- Local group codes in fields `lgc1` - `lgc9`. You could think of them as various ids for a given centerline segment. `lgc1` is the "preferred" local group code. A centerline segment can have between 1 and 9 local group codes.
- Board of Elections local group code. It's derived based on a flag and the lgc number. If a flag is in a street segment corresponding to `lgc4`, then the value for this field is 4.
- Face code
- `5sc` field. This field is a street code but without a borough in the beginning.

The logic in dbt collects all these fields into one table, `int__centerline_streetcode_and_facecode` which has 1-1 mapping with the final lion table. The visual:

![image](https://github.com/user-attachments/assets/f40e621c-28ea-49b3-b372-ca9a4d2ce7ef)

### Testing data
Please refer to the issue. In summary, there are mismatches that I'm not sure how to address. The dev values in the new fields make sense logically, matching to the docs description. 


Linked issue #1566: CSCL ETL POC - Centerline Street/Face Code Fields
The standard workflow for these cscl chapters is pretty consistent, just with different sections in the documentation and different fields needed in the final product table
- [x] Source data (ingest templates, recipe/dbt setup) added as needed if mentioned in 2.2.1
- [x] intermediate tables added to perform transformations outlines in 2.2.1
- [x] consider/implement constraints/tests of table (i.e. 1-1 rows with centerline, etc)
- [x] final product table modified with int table joined in to align with Table 3 for relevant fields:
  - L2, L5-10, L74-78

Linked issue comments: user: login: sf-dcp
created_at: 2025-05-01T00:49:31Z
body: ### Query 1: `LGC1`

```sql
select 
d."Borough",
d."Sequence Number",
d."Segment ID",
d."LGC1" as lgc1_new,
p."LGC1" as lgc1_original
from  sf_cscl_centerline_street_fields.lion_dat_fields d
inner join production_outputs.citywide_lion p
using("Borough", "Sequence Number", "Segment ID")
where d."LGC1" <> p."LGC1";
```
This results in 13 records:

|Borough|Sequence Number|Segment ID|lgc1_new|lgc1_original|
|-------|---------------|----------|--------|-------------|
|1|02020|0194496|03|01|
|1|02040|0141766|03|01|
|1|00110|0071288|02|01|
|1|02010|0194497|03|01|
|1|02000|0194498|03|01|
|1|00815|0295870|01|03|
|1|00789|0295871|01|03|
|1|01429|0295873|01|03|
|1|01459|0295874|01|03|
|3|00230|0161973|05|01|
|3|00210|0021962|05|01|
|5|00010|0160074|11|01|

This table is important to troubleshoot because the other field `Face Code` uses `lgc1` value in the join (joined precisely on the "preferred" `b7sc` which is comprised of `b5sc` + `lgc1`).

According to the lion guide, the `lgc1` values come from the raw data where the `preferred_lgc_flag` = `Y` for a given segment. Only one record per segment can have the flag with `Y`. Here is the example of the first record in the raw data which confirms the correct value, `3`, in the resulting table above:

![Image](https://github.com/user-attachments/assets/6df45ddf-3b82-40a6-b5c8-df45ebaad97b)


I checked all other records: most of them are "flipped" like that record and  a few of them don't even have the `lgc` value on the list. For example, the last record in the raw data: 

![Image](https://github.com/user-attachments/assets/9bc53d3c-fa68-402c-b8b6-c951ab45a56a)

I'm not sure where these `lgc1` values in the production table come from.
author_association: CONTRIBUTOR
user: login: sf-dcp
created_at: 2025-05-01T01:18:30Z
body: ### Query 2: `LGC2`

```sql
select distinct
d."Borough",
d."Sequence Number",
d."Segment ID",
d."LGC1",
d."LGC2" as lgc2_new,
p."LGC2" as lgc2_original
from  sf_cscl_centerline_street_fields.lion_dat_fields d
inner join production_outputs.citywide_lion p
using("Borough", "Sequence Number", "Segment ID", "LGC1")
where d."LGC2" <> p."LGC2";
```
Results in 47 records: all records have `NULL` in `lgc2_original` like this:

|Borough|Sequence Number|Segment ID|LGC1|lgc2_new|lgc2_original|
|-------|---------------|----------|----|--------|-------------|
|2|00020|0138857|01|02|  |
|2|00020|0179536|01|02|  |
|2|00020|0179552|01|02|  |

With `LGC2` - `LGC9`, these fields come from the same raw data as `LGC1` where `preferred_lgc_flag` = `N`. Then these fields are sorted in the ascended order to fill up the remaining `LGC2` - `LGC9` columns. 

The resulting query records: confirmed with the raw data, these records shouldn't be empty. 
author_association: CONTRIBUTOR
user: login: sf-dcp
created_at: 2025-05-01T01:53:20Z
body: ### Query 3: `5-Digit Street Code (5SC)`

```sql
select distinct
d."Borough",
d."Sequence Number",
d."Segment ID",
d."5-Digit Street Code (5SC)" as dev_5sc,
p."5-Digit Street Code (5SC)" as prod_5sc
from  sf_cscl_centerline_street_fields.lion_dat_fields d
inner join production_outputs.citywide_lion p
using("Borough", "Sequence Number", "Segment ID")
where d."5-Digit Street Code (5SC)" <> p."5-Digit Street Code (5SC)";
```

This results in 113 records. Sample:

|Borough|Sequence Number|Segment ID|dev_5sc|prod_5sc|
|-------|---------------|----------|-------|--------|
|1|00010|0037266|10740|10700|
|1|00015|0195843|10740|10700|
|1|00020|0195844|10740|10700|
|1|00110|0071288|98110|10955|
|1|00789|0295871|11112|11115|
|1|00815|0295870|11112|11115|


According to the lion guide, the values should come from same raw dataset as the `lgc` values, `dcp_cscl_segment_lgc`. They are derived from stripping the `b5sc` column of the first digit representing borough. Pretty simple logic. 

I spot-checked many records from the query and confirmed that their values are correct in the raw data. The prod values are incorrect. 

Here is example of the first record in the raw data: 

![Image](https://github.com/user-attachments/assets/3ea4cd5f-1671-46c9-8742-af87a9f95d6f)
author_association: CONTRIBUTOR
user: login: sf-dcp
created_at: 2025-05-01T01:57:55Z
body: ### Query 4: `Face Code`

```sql
select distinct
d."Borough",
d."Sequence Number",
d."Segment ID",
d."Borough" || d."5-Digit Street Code (5SC)" ||d."LGC1" as b7sc,
d."Face Code" as dev_facecode,
p."Face Code" as prod_facecode
from  sf_cscl_centerline_street_fields.lion_dat_fields d
inner join production_outputs.citywide_lion p
using("Borough", "Sequence Number", "Segment ID")
where d."Face Code" <> p."Face Code";
```

Returns 115 records. Sample:

|Borough|Sequence Number|Segment ID|b7sc|dev_facecode|prod_facecode|
|-------|---------------|----------|----|------------|-------------|
|1|00010|0037266|11074001|8112|8113|
|1|00015|0195843|11074001|8112|8113|
|1|00020|0195844|11074001|8112|8113|
|1|00110|0071288|19811002|0190|8107|
|1|00789|0295871|11111201|4442|2937|


According to the lion guide, the face code value is derived from `dcp_cscl_streetname` or `dcp_cscl_featurename` based on a `b7sc` code and `principal_flag` = `Y`. We derive `b7sc` as `b5sc` + `lgc1` which means issues with `lgc1` may cause the mismatch in this query. 

I spot-checked several records and the dev values are correct. For reference, here is the raw data for the first record from the query:

![Image](https://github.com/user-attachments/assets/9ec63359-27c2-4e3f-9f75-e5429dd74f31)
author_association: CONTRIBUTOR
user: login: fvankrieken
created_at: 2025-05-01T14:22:24Z
body: Not quite sure how they'd be getting the non-preferred lgc assigned, but for lgc1, are these cases where we have multiple rows in the 'prod" data for these segmentids? As in, there is both a row that matches your output and a row that doesn't.

I've been running this to account for those
```sql
with properly_joined as (
select 
d."Borough",
d."Sequence Number",
d."Segment ID"
from main.lion_dat_fields d
inner join production_outputs.citywide_lion p
using("Borough", "Sequence Number", "Segment ID")
where d."Left Dynamic Block" = p."Left Dynamic Block"
)
select 
d."Borough",
d."Sequence Number",
d."Segment ID",
d."Segment Length in Feet",
d."Left Dynamic Block",
p."Left Dynamic Block"
from main.lion_dat_fields d
left join properly_joined using("Borough", "Sequence Number", "Segment ID")
inner join production_outputs.citywide_lion p using("Borough", "Sequence Number", "Segment ID")
where properly_joined."Borough" is null 
and d."Left Dynamic Block" <> p."Left Dynamic Block";
```

I guess another approach would be select segmentid, lgc1 from your data `EXCEPT` select segmentid, lgc1 from "prod" data
author_association: CONTRIBUTOR
user: login: fvankrieken
created_at: 2025-05-01T16:29:28Z
body: To that point

```sql
with properly_joined as (
select 
d."Borough",
d."Sequence Number",
d."Segment ID"
from sf_cscl_centerline_street_fields.lion_dat_fields d
inner join production_outputs.citywide_lion p
using("Borough", "Sequence Number", "Segment ID")
where d."LGC1" = p."LGC1"
)
select 
d."Borough",
d."Sequence Number",
d."Segment ID",
d."LGC1",
p."LGC1"
from sf_cscl_centerline_street_fields.lion_dat_fields d
left join properly_joined using("Borough", "Sequence Number", "Segment ID")
inner join production_outputs.citywide_lion p using("Borough", "Sequence Number", "Segment ID")
where properly_joined."Borough" is null 
and d."LGC1" <> p."LGC1";
```

returns zero rows!

Same for LGC2
author_association: CONTRIBUTOR
user: login: fvankrieken
created_at: 2025-05-01T16:34:23Z
body: There is one mismatch for lgc3. This is what the row looks like in prod data

<img width="440" alt="Image" src="https://github.com/user-attachments/assets/e9b2d6a2-5227-44ca-abf8-22f5ed04eb83" />

In yours, no lgc3 is assigned. It looks like there are duplicate lgc 03 rows in `dcp_cscl_segment_lgc`.

I guess this is a consequence of using `rank` instead of `row_number`, though I don't know that your solution is "wrong" or if this is more of a bug in the current etl application
author_association: CONTRIBUTOR
user: login: fvankrieken
created_at: 2025-05-01T16:37:25Z
body: How about 'board_of_elections_lgc_pointer'?
author_association: CONTRIBUTOR
user: login: sf-dcp
created_at: 2025-05-01T17:58:58Z
body: Yes, here it is:

## Query 5: `Board of Elections LGC Pointer`

```sql
select 
d."Borough",
d."Sequence Number",
d."Segment ID",
d."Board of Elections LGC Pointer" as boe_dev,
p."Board of Elections LGC Pointer" as boe_prod
from  sf_cscl_centerline_street_fields.lion_dat_fields d
inner join production_outputs.citywide_lion p
using("Borough", "Sequence Number", "Segment ID")
where d."Board of Elections LGC Pointer" <> p."Board of Elections LGC Pointer";
```

This return 567 records. Sample: 

|Borough|Sequence Number|Segment ID|boe_dev|boe_prod|
|-------|---------------|----------|-------|--------|
|4|05100|0092346|2|1|
|4|06780|0120735|2|1|
|4|03500|0074850|2|1|
|4|00590|0107049|2|1|
|4|03190|0174651|2|1|

All 567 records in the table have `1` in the prod column. My guess is that these records can be non-centerline segments as they are expected to have `1`:  

> If the given segment belongs to a non-Centerline feature, the tool will populate the output Board of Elections LGC Pointer field with the value ’1’. If it belongs to a Centerline feature, the tool will populate the output Board of Elections LGC Pointer with an integer corresponding to the output LGC field that has been loaded from the Segment_LGC entry that contains a ‘Y’ in the BOE_PREFERRED_LGC_FLAG. For example, if the output LGC field that has been loaded from that Segment_LGC entry is LGC3, the tool will populate the output Board of Elections LGC Pointer with the value ‘3’.

If these records are, in fact, centerline segments then it doesn't make sense why they are assigned that value in prod. 

I also ran this query which returns exactly same 567 records:

```sql
select d."Segment ID", d."Board of Elections LGC Pointer"
from sf_cscl_centerline_street_fields.lion_dat_fields d
except
select p."Segment ID", p."Board of Elections LGC Pointer" from production_outputs.citywide_lion p
```

@fvankrieken
author_association: CONTRIBUTOR
user: login: fvankrieken
created_at: 2025-05-01T19:01:52Z
body: I don't think they are non-centerline. Grabbing one of them and looking at the prod table

```sql
select "Segment ID", "Board of Elections LGC Pointer" 
from production_outputs.citywide_lion cl 
where "Segment ID" = '0092346';
```

Only returns one row. From my understanding, non-centerline features would never overwrite a centerline one, we'd just see duplicate rows.

Looking at the LGC data for that segid is odd though.

```sql
select * from sf_cscl_centerline_street_fields.dcp_cscl_segment_lgc where segmentid = '0092346'
```

|segmentid|b5sc|preferred_lgc_flag|boe_preferred_lgc_flag|created_by|created_date|modified_by|modified_date|lgc|b7sc|globalid|unique_roadbed_lcg_flag|
|---------|----|------------------|----------------------|----------|------------|-----------|-------------|---|----|--------|-----------------------|
|92346|466290|N|Y|J_Gieble|2023-07-05 04:28:03.000 -0700|J_Gieble|2023-07-07 04:19:57.000 -0700|13|46629013|{0587A4A8-B2FF-405D-9590-95A3BB7E61E4}|N|
|92346|466290|N|N|J_Gieble|2023-07-05 04:28:03.000 -0700|J_Gieble|2023-07-07 04:19:57.000 -0700|17|46629017|{5D3E4694-40DB-4F17-94B5-BD810B2F5180}|N|
|92346|466290|N|N|J_Gieble|2023-07-05 04:28:03.000 -0700|J_Gieble|2023-07-07 04:19:57.000 -0700|22|46629022|{53CE126C-5451-4F95-BE77-5333754CB43B}|N|
|92346|466290|Y|N|J_Gieble|2023-07-05 04:28:03.000 -0700|J_Gieble|2023-07-07 04:19:57.000 -0700|12|46629012|{F9C1807A-9186-459C-8AD5-6E68CE777312}|Y|
|92346|466290|N|N|J_Gieble|2023-07-05 04:28:03.000 -0700|J_Gieble|2023-07-07 04:19:57.000 -0700|28|46629028|{7A85BF13-4B88-427B-B969-3B5DEDF2E1EE}|N|

So we have a row that's the `preferred_lgc`, so it should obviously end up as `LGC1` (which it does). Then we have the next lowest lgc, 13, which ends up as `LGC2`. This is true both in "prod" and in `lion_dat_fields` in your db table

```sql
select "Segment ID", "LGC1", "LGC2", "Board of Elections LGC Pointer" 
from production_outputs.citywide_lion cl 
where "Segment ID" = '0092346';
```

|Segment ID|LGC1|LGC2|LGC3|Board of Elections LGC Pointer|
|----------|----|----|----|----|
|0092346|12|13|17|1|

The row with lgc=13 has the boe_preferred_flag, which ends up in LGC2... so the BOE LGC pointer should be 2 not 1. Which your logic is doing.

Sooooo... I think this looks like a bug in the existing ETL code. I'll read over the docs one more time, but it really seems like you've implemented this correctly
author_association: CONTRIBUTOR

Diff:
diff --git a/ingest_templates/dcp_cscl_featurename.yml b/ingest_templates/dcp_cscl_featurename.yml
new file mode 100644
index 0000000000..2ec7c206e5
--- /dev/null
+++ b/ingest_templates/dcp_cscl_featurename.yml
@@ -0,0 +1,18 @@
+id: dcp_cscl_featurename
+acl: private
+
+attributes:
+  name: CSCL Feature Name
+
+ingestion:
+  source:
+    type: s3
+    bucket: edm-private
+    key: cscl_etl/ETL Working GDB.gdb.zip
+  file_format:
+    type: geodatabase
+    layer: FEATURENAME
+    crs: EPSG:2263
+  processing_steps:
+    - name: clean_column_names
+      args: { lower: True }
diff --git a/ingest_templates/dcp_cscl_segment_lgc.yml b/ingest_templates/dcp_cscl_segment_lgc.yml
new file mode 100644
index 0000000000..36a8486ec3
--- /dev/null
+++ b/ingest_templates/dcp_cscl_segment_lgc.yml
@@ -0,0 +1,18 @@
+id: dcp_cscl_segment_lgc
+acl: private
+
+attributes:
+  name: CSCL Segment Local Group Code (LGC)
+
+ingestion:
+  source:
+    type: s3
+    bucket: edm-private
+    key: cscl_etl/ETL Working GDB.gdb.zip
+  file_format:
+    type: geodatabase
+    layer: SEGMENT_LGC
+    crs: EPSG:2263
+  processing_steps:
+  - name: clean_column_names
+    args: { lower: True }
diff --git a/ingest_templates/dcp_cscl_streetname.yml b/ingest_templates/dcp_cscl_streetname.yml
new file mode 100644
index 0000000000..ec50ee0dbc
--- /dev/null
+++ b/ingest_templates/dcp_cscl_streetname.yml
@@ -0,0 +1,18 @@
+id: dcp_cscl_streetname
+acl: private
+
+attributes:
+  name: CSCL Street Name
+
+ingestion:
+  source:
+    type: s3
+    bucket: edm-private
+    key: cscl_etl/ETL Working GDB.gdb.zip
+  file_format:
+    type: geodatabase
+    layer: STREETNAME
+    crs: EPSG:2263
+  processing_steps:
+    - name: clean_column_names
+      args: { lower: True }
diff --git a/products/lion/models/intermediate/2.2.1/_int_221.yml b/products/lion/models/intermediate/2.2.1/_int_221.yml
new file mode 100644
index 0000000000..754ad17f39
--- /dev/null
+++ b/products/lion/models/intermediate/2.2.1/_int_221.yml
@@ -0,0 +1,88 @@
+models:
+- name: int__preferred_local_group_code
+  description: Segments with preferred (primary) local group. Each segment can have only one local group
+  columns:
+  - name: segmentid
+    tests:
+    - not_null
+    - unique
+  - name: lgc
+  - name: lgc_rank
+    tests:
+      - accepted_values:
+          values: [1]
+  - name: boe_preferred_lgc_flag
+  - name: b5sc
+  - name: b5sc_segmentid_rank
+  - name: b7sc
+  tests:
+    - dbt_utils.unique_combination_of_columns:
+        combination_of_columns:
+          - segmentid
+          - lgc
+
+- name: int__nonpreferred_local_group_code
+  description: Segments with non-preferred local group. A segment can have multiple local non-preferred groups
+  columns:
+  - name: segmentid
+    tests:
+    - not_null
+  - name: lgc
+  - name: lgc_rank
+    tests:
+      - dbt_utils.accepted_range:
+          min_value: 2
+          max_value: 9  # a segment shouldn't have more than 9 lgcs
+          inclusive: true
+  - name: boe_preferred_lgc_flag
+  - name: b5sc
+  - name: b7sc
+
+- name: int__all_local_group_code_ranked
+  description: Segments with all, preferred and non-preferred, local group codes, ranked by local group code.
+  columns:
+  - name: segmentid
+    tests:
+    - not_null
+  - name: lgc
+    tests:
+    - not_null
+  - name: lgc_rank
+    tests:
+      - dbt_utils.accepted_range:
+          min_value: 1
+          max_value: 9  # a segment shouldn't have more than 9 lgcs
+          inclusive: true
+  - name: board_of_elections_lgc_pointer
+
+- name: int__centerline_streetcode_and_facecode
+  description: Centerline segments joined to street codes and face codes
+  columns:
+  - name: segmentid
+    tests:
+    - not_null
+    - unique
+  - name: lgc1
+    tests:
+    - not_null
+  - name: lgc2
+  - name: lgc3
+  - name: lgc4
+  - name: lgc5
+  - name: lgc6
+  - name: lgc7
+  - name: lgc8
+  - name: lgc9
+  - name: board_of_elections_lgc_pointer
+    tests:
+    - not_null
+  - name: five_digit_street_code
+    tests:
+    - not_null
+  - name: face_code
+    tests:
+    - not_null
+  # the test below should be here but dbt errors while building staging tables
+  # tests:
+  # - dbt_expectations.expect_table_row_count_to_equal_other_table:
+  #    compare_model: ref("stg__centerline")
diff --git a/products/lion/models/intermediate/2.2.1/int__all_local_group_code_ranked.sql b/products/lion/models/intermediate/2.2.1/int__all_local_group_code_ranked.sql
new file mode 100644
index 0000000000..0c10bf154a
--- /dev/null
+++ b/products/lion/models/intermediate/2.2.1/int__all_local_group_code_ranked.sql
@@ -0,0 +1,25 @@
+{{ config(
+    materialized = 'table',
+    indexes=[
+      {'columns': ['segmentid']},
+    ]
+) }}
+WITH all_ranked_lgc AS (
+    SELECT preferred_lgc.*
+    FROM {{ ref("int__preferred_local_group_code") }} AS preferred_lgc
+    UNION
+    SELECT nonpreferred_lgc.*
+    FROM {{ ref("int__nonpreferred_local_group_code") }} AS nonpreferred_lgc
+)
+SELECT
+    segmentid,
+    lgc,
+    lgc_rank,
+    CASE WHEN boe_preferred_lgc_flag = 'Y' THEN lgc_rank END AS board_of_elections_lgc_pointer,
+    b5sc,
+    RANK() OVER (
+        PARTITION BY segmentid
+        ORDER BY b5sc
+    ) AS b5sc_segmentid_rank, -- for sanity check. A segment should only have one b5sc (rank = 1)
+    b7sc
+FROM all_ranked_lgc
diff --git a/products/lion/models/intermediate/2.2.1/int__centerline_streetcode_and_facecode.sql b/products/lion/models/intermediate/2.2.1/int__centerline_streetcode_and_facecode.sql
new file mode 100644
index 0000000000..27cb93bead
--- /dev/null
+++ b/products/lion/models/intermediate/2.2.1/int__centerline_streetcode_and_facecode.sql
@@ -0,0 +1,61 @@
+{{ config(
+    materialized = 'table',
+    indexes=[
+      {'columns': ['segmentid']},
+    ]
+) }}
+WITH centerline AS (
+    SELECT * FROM {{ ref("stg__centerline") }}
+),
+all_lgc_pivoted AS (
+    SELECT
+        segmentid,
+        MAX(CASE WHEN lgc_rank = 1 THEN lgc END) AS lgc1,
+        MAX(CASE WHEN lgc_rank = 2 THEN lgc END) AS lgc2,
+        MAX(CASE WHEN lgc_rank = 3 THEN lgc END) AS lgc3,
+        MAX(CASE WHEN lgc_rank = 4 THEN lgc END) AS lgc4,
+        MAX(CASE WHEN lgc_rank = 5 THEN lgc END) AS lgc5,
+        MAX(CASE WHEN lgc_rank = 6 THEN lgc END) AS lgc6,
+        MAX(CASE WHEN lgc_rank = 7 THEN lgc END) AS lgc7,
+        MAX(CASE WHEN lgc_rank = 8 THEN lgc END) AS lgc8,
+        MAX(CASE WHEN lgc_rank = 9 THEN lgc END) AS lgc9,
+        MAX(board_of_elections_lgc_pointer) AS board_of_elections_lgc_pointer,
+        MAX(CASE WHEN lgc_rank = 1 THEN b5sc END) AS b5sc,
+        MAX(CASE WHEN lgc_rank = 1 THEN b7sc END) AS preferred_b7sc
+    FROM {{ ref("int__all_local_group_code_ranked") }}
+    GROUP BY segmentid
+),
+principal_streetnames AS (
+    SELECT
+        b7sc,
+        facecode
+    FROM {{ source("recipe_sources", "dcp_cscl_streetname") }}
+    WHERE principal_flag = 'Y'
+),
+principal_features AS (
+    SELECT
+        b7sc,
+        facecode
+    FROM {{ source("recipe_sources", "dcp_cscl_featurename") }}
+    WHERE principal_flag = 'Y'
+)
+
+SELECT
+    centerline.segmentid,
+    all_lgc_pivoted.lgc1,
+    all_lgc_pivoted.lgc2,
+    all_lgc_pivoted.lgc3,
+    all_lgc_pivoted.lgc4,
+    all_lgc_pivoted.lgc5,
+    all_lgc_pivoted.lgc6,
+    all_lgc_pivoted.lgc7,
+    all_lgc_pivoted.lgc8,
+    all_lgc_pivoted.lgc9,
+    all_lgc_pivoted.board_of_elections_lgc_pointer,
+    RIGHT(all_lgc_pivoted.b5sc, 5) AS five_digit_street_code,
+    COALESCE(principal_streetnames.facecode, principal_features.facecode) AS face_code
+FROM
+    centerline
+LEFT JOIN all_lgc_pivoted ON centerline.segmentid = all_lgc_pivoted.segmentid
+LEFT JOIN principal_streetnames ON all_lgc_pivoted.preferred_b7sc = principal_streetnames.b7sc
+LEFT JOIN principal_features ON all_lgc_pivoted.preferred_b7sc = principal_features.b7sc
diff --git a/products/lion/models/intermediate/2.2.1/int__nonpreferred_local_group_code.sql b/products/lion/models/intermediate/2.2.1/int__nonpreferred_local_group_code.sql
new file mode 100644
index 0000000000..44d22c0f22
--- /dev/null
+++ b/products/lion/models/intermediate/2.2.1/int__nonpreferred_local_group_code.sql
@@ -0,0 +1,28 @@
+WITH nonpreferred AS (
+    SELECT
+        segmentid,
+        lgc,
+        boe_preferred_lgc_flag,
+        b5sc,
+        b7sc
+    FROM
+        {{ source("recipe_sources", "dcp_cscl_segment_lgc") }}
+    WHERE
+        preferred_lgc_flag <> 'Y'
+),
+nonpreferred_ranked_by_lgc AS (
+    SELECT
+        segmentid,
+        lgc,
+        RANK() OVER (
+            PARTITION BY segmentid
+            ORDER BY lgc ASC
+        ) + 1 AS lgc_rank, -- value of 1 is reserved for preferred lgc
+        boe_preferred_lgc_flag,
+        b5sc,
+        b7sc
+    FROM nonpreferred
+)
+
+SELECT *
+FROM nonpreferred_ranked_by_lgc
diff --git a/products/lion/models/intermediate/2.2.1/int__preferred_local_group_code.sql b/products/lion/models/intermediate/2.2.1/int__preferred_local_group_code.sql
new file mode 100644
index 0000000000..2fbd5d75a4
--- /dev/null
+++ b/products/lion/models/intermediate/2.2.1/int__preferred_local_group_code.sql
@@ -0,0 +1,11 @@
+SELECT
+    segmentid,
+    lgc,
+    1 AS lgc_rank,
+    boe_preferred_lgc_flag,
+    b5sc,
+    b7sc
+FROM
+    {{ source("recipe_sources", "dcp_cscl_segment_lgc") }}
+WHERE
+    preferred_lgc_flag = 'Y'
diff --git a/products/lion/models/product/lion.sql b/products/lion/models/product/lion.sql
index 2dc63fed20..f3bb6585f0 100644
--- a/products/lion/models/product/lion.sql
+++ b/products/lion/models/product/lion.sql
@@ -24,19 +24,23 @@ saf AS (
 
 curve AS (
     SELECT * FROM {{ ref("int__centerline_curve") }}
+),
+
+streets AS (
+    SELECT * FROM {{ ref("int__centerline_streetcode_and_facecode") }}
 )
 
 SELECT
     centerline.boroughcode,
-    NULL AS face_code,
+    streets.face_code,
     centerline.segment_seqnum,
     centerline.segmentid,
-    NULL AS five_digit_street_code,
-    NULL AS lgc1,
-    NULL AS lgc2,
-    NULL AS lgc3,
-    NULL AS lgc4,
-    NULL AS board_of_elections_lgc_pointer,
+    streets.five_digit_street_code,
+    streets.lgc1,
+    streets.lgc2,
+    streets.lgc3,
+    streets.lgc4,
+    streets.board_of_elections_lgc_pointer::CHAR(1),
     nodes.from_sectionalmap,
     nodes.from_nodeid,
     nodes.from_x,
@@ -125,11 +129,11 @@ SELECT
     ap.left_2010_census_tract_suffix,
     ap.right_2010_census_tract_basic,
     ap.right_2010_census_tract_suffix,
-    NULL AS lgc5,
-    NULL AS lgc6,
-    NULL AS lgc7,
-    NULL AS lgc8,
-    NULL AS lgc9,
+    streets.lgc5,
+    streets.lgc6,
+    streets.lgc7,
+    streets.lgc8,
+    streets.lgc9,
     centerline.legacy_segmentid,
     ap.left_2000_census_block_basic,
     ap.left_2000_census_block_suffix,
@@ -165,3 +169,4 @@ LEFT JOIN atomic_polygons AS ap ON centerline.segmentid = ap.segmentid
 LEFT JOIN nodes ON centerline.segmentid = nodes.segmentid
 LEFT JOIN saf ON centerline.segmentid = saf.segmentid AND centerline.boroughcode = saf.boroughcode
 LEFT JOIN curve ON centerline.segmentid = curve.segmentid
+LEFT JOIN streets ON centerline.segmentid = streets.segmentid
diff --git a/products/lion/models/sources.yml b/products/lion/models/sources.yml
index 41ae755b33..f27525382e 100644
--- a/products/lion/models/sources.yml
+++ b/products/lion/models/sources.yml
@@ -11,3 +11,6 @@ sources:
   - name: dcp_cscl_altsegmentdata
   - name: dcp_cscl_addresspoints
   - name: dcp_cscl_commonplace_gdb
+  - name: dcp_cscl_streetname
+  - name: dcp_cscl_featurename
+  - name: dcp_cscl_segment_lgc
diff --git a/products/lion/recipe.yml b/products/lion/recipe.yml
index 4c9732d64e..24e5118f2a 100644
--- a/products/lion/recipe.yml
+++ b/products/lion/recipe.yml
@@ -6,6 +6,9 @@ inputs:
   - name: dcp_cscl_centerline
   - name: dcp_cscl_nodes
   - name: dcp_cscl_sectionalmap
+  - name: dcp_cscl_streetname
+  - name: dcp_cscl_featurename
+  - name: dcp_cscl_segment_lgc
   - name: dcp_cscl_atomicpolygons
   - name: dcp_cscl_altsegmentdata
   - name: dcp_cscl_addresspoints


Reviews:
user: login: sf-dcp
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-01T02:11:00Z
user: login: fvankrieken
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-01T14:26:18Z
user: login: fvankrieken
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-01T14:57:42Z
user: login: fvankrieken
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-01T16:27:51Z
user: login: sf-dcp
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-01T16:28:49Z
user: login: fvankrieken
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-01T16:34:57Z
user: login: fvankrieken
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-01T16:38:52Z
user: login: fvankrieken
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-01T19:03:01Z
user: login: fvankrieken
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-01T19:06:04Z
user: login: fvankrieken
body: Approving with a couple github requests
- detail issues around that single LGC3 mismatch in #1597 (i.e. are duplicate LGCs allowed?)
- " for BOE flag. It really seems like this is a bug in the existing ETL
- up to you if you want to get that test working. As I linked in the comment, it's a pretty simple fix (to a very strange bug IMO) to get it working. Happy to punt it towards general pipeline cleanup
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-05-01T19:09:20Z

Review comments:
diff_hunk: @@ -0,0 +1,88 @@
+models:
+- name: int__preferred_local_group_code
+  description: Segments with preferred (primary) local group. Each segment can have only one local group
+  columns:
+  - name: segmentid
+    tests:
+    - not_null
+    - unique
+  - name: lgc
+  - name: lgc_rank
+    tests:
+      - accepted_values:
+          values: [1]
+  - name: boe_preferred_lgc_flag
+  - name: b5sc
+  - name: b5sc_segmentid_rank
+  - name: b7sc
+  tests:
+    - dbt_utils.unique_combination_of_columns:
+        combination_of_columns:
+          - segmentid
+          - lgc
+
+- name: int__nonpreferred_local_group_code
+  description: Segments with non-preferred local group. A segment can have multiple local non-preferred groups
+  columns:
+  - name: segmentid
+    tests:
+    - not_null
+  - name: lgc
+  - name: lgc_rank
+    tests:
+      - dbt_utils.accepted_range:
+          min_value: 2
+          max_value: 9  # a segment shouldn't have more than 9 lgcs
+          inclusive: true
+  - name: boe_preferred_lgc_flag
+  - name: b5sc
+  - name: b7sc
+
+- name: int__all_local_group_code_ranked
+  description: Segments with all, preferred and non-preferred, local group codes, ranked by local group code.
+  columns:
+  - name: segmentid
+    tests:
+    - not_null
+  - name: lgc
+    tests:
+    - not_null
+  - name: lgc_rank
+    tests:
+      - dbt_utils.accepted_range:
+          min_value: 1
+          max_value: 9  # a segment shouldn't have more than 9 lgcs
+          inclusive: true
+  - name: board_of_elections_lgc_pointer
+
+- name: int__centerline_streetcode_and_facecode
+  description: Centerline segments joined to street codes and face codes
+  columns:
+  - name: segmentid
+    tests:
+    - not_null
+    - unique
+  - name: lgc1
+    tests:
+    - not_null
+  - name: lgc2
+  - name: lgc3
+  - name: lgc4
+  - name: lgc5
+  - name: lgc6
+  - name: lgc7
+  - name: lgc8
+  - name: lgc9
+  - name: board_of_elections_lgc_pointer
+    tests:
+    - not_null
+  - name: five_digit_street_code
+    tests:
+    - not_null
+  - name: face_code
+    tests:
+    - not_null
+  # the test below should be here but dbt errors while building staging tables
+  # tests:
+  # - dbt_expectations.expect_table_row_count_to_equal_other_table:
+  #    compare_model: ref("stg__centerline")
path: products/lion/models/intermediate/2.2.1/_int_221.yml
user: login: sf-dcp
body: Would be nice to have this test but `dbt` throws an error when building stage tables. The error is saying that `int__centerline_streetcode_and_facecode` table cannot be found. 
created_at: 2025-05-01T02:11:00Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,28 @@
+WITH nonpreferred AS (
+    SELECT
+        segmentid,
+        lgc,
+        boe_preferred_lgc_flag,
+        b5sc,
+        b7sc
+    FROM
+        {{ source("recipe_sources", "dcp_cscl_segment_lgc") }}
+    WHERE
+        preferred_lgc_flag <> 'Y'
+),
+nonpreferred_ranked_by_lgc AS (
+    SELECT
+        segmentid,
+        lgc,
+        RANK() OVER (
+            PARTITION BY segmentid
+            ORDER BY lgc ASC
path: products/lion/models/intermediate/2.2.1/int__nonpreferred_local_group_code.sql
user: login: fvankrieken
body: I think we maybe should order by `ogc_fid` here - I had another one where it had similar wording like "take the first" or "in order" and the way I got my code to match up with production was by going by row number in the original table
created_at: 2025-05-01T14:26:17Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,28 @@
+WITH nonpreferred AS (
+    SELECT
+        segmentid,
+        lgc,
+        boe_preferred_lgc_flag,
+        b5sc,
+        b7sc
+    FROM
+        {{ source("recipe_sources", "dcp_cscl_segment_lgc") }}
+    WHERE
+        preferred_lgc_flag <> 'Y'
+),
+nonpreferred_ranked_by_lgc AS (
+    SELECT
+        segmentid,
+        lgc,
+        RANK() OVER (
+            PARTITION BY segmentid
+            ORDER BY lgc ASC
path: products/lion/models/intermediate/2.2.1/int__nonpreferred_local_group_code.sql
user: login: fvankrieken
body: How many rows actually get assigned an lgc2?
created_at: 2025-05-01T14:57:42Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,28 @@
+WITH nonpreferred AS (
+    SELECT
+        segmentid,
+        lgc,
+        boe_preferred_lgc_flag,
+        b5sc,
+        b7sc
+    FROM
+        {{ source("recipe_sources", "dcp_cscl_segment_lgc") }}
+    WHERE
+        preferred_lgc_flag <> 'Y'
+),
+nonpreferred_ranked_by_lgc AS (
+    SELECT
+        segmentid,
+        lgc,
+        RANK() OVER (
+            PARTITION BY segmentid
+            ORDER BY lgc ASC
path: products/lion/models/intermediate/2.2.1/int__nonpreferred_local_group_code.sql
user: login: fvankrieken
body: Looks like a lot, it would be surprising if your ranking were not accurate
created_at: 2025-05-01T16:27:51Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,28 @@
+WITH nonpreferred AS (
+    SELECT
+        segmentid,
+        lgc,
+        boe_preferred_lgc_flag,
+        b5sc,
+        b7sc
+    FROM
+        {{ source("recipe_sources", "dcp_cscl_segment_lgc") }}
+    WHERE
+        preferred_lgc_flag <> 'Y'
+),
+nonpreferred_ranked_by_lgc AS (
+    SELECT
+        segmentid,
+        lgc,
+        RANK() OVER (
+            PARTITION BY segmentid
+            ORDER BY lgc ASC
path: products/lion/models/intermediate/2.2.1/int__nonpreferred_local_group_code.sql
user: login: sf-dcp
body: @fvankrieken, I will try with `ogr_uid`. 

Answering your question: there are 49,664  *centerline* records with an assigned `lgc2`. The total number of records from this table with `lgc2` is 72,525 which includes all feature classes.
created_at: 2025-05-01T16:28:49Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,28 @@
+WITH nonpreferred AS (
+    SELECT
+        segmentid,
+        lgc,
+        boe_preferred_lgc_flag,
+        b5sc,
+        b7sc
+    FROM
+        {{ source("recipe_sources", "dcp_cscl_segment_lgc") }}
+    WHERE
+        preferred_lgc_flag <> 'Y'
+),
+nonpreferred_ranked_by_lgc AS (
+    SELECT
+        segmentid,
+        lgc,
+        RANK() OVER (
+            PARTITION BY segmentid
+            ORDER BY lgc ASC
path: products/lion/models/intermediate/2.2.1/int__nonpreferred_local_group_code.sql
user: login: fvankrieken
body: I take it back!
created_at: 2025-05-01T16:34:57Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,88 @@
+models:
+- name: int__preferred_local_group_code
+  description: Segments with preferred (primary) local group. Each segment can have only one local group
+  columns:
+  - name: segmentid
+    tests:
+    - not_null
+    - unique
+  - name: lgc
+  - name: lgc_rank
+    tests:
+      - accepted_values:
+          values: [1]
+  - name: boe_preferred_lgc_flag
+  - name: b5sc
+  - name: b5sc_segmentid_rank
+  - name: b7sc
+  tests:
+    - dbt_utils.unique_combination_of_columns:
+        combination_of_columns:
+          - segmentid
+          - lgc
+
+- name: int__nonpreferred_local_group_code
+  description: Segments with non-preferred local group. A segment can have multiple local non-preferred groups
+  columns:
+  - name: segmentid
+    tests:
+    - not_null
+  - name: lgc
+  - name: lgc_rank
+    tests:
+      - dbt_utils.accepted_range:
+          min_value: 2
+          max_value: 9  # a segment shouldn't have more than 9 lgcs
+          inclusive: true
+  - name: boe_preferred_lgc_flag
+  - name: b5sc
+  - name: b7sc
+
+- name: int__all_local_group_code_ranked
+  description: Segments with all, preferred and non-preferred, local group codes, ranked by local group code.
+  columns:
+  - name: segmentid
+    tests:
+    - not_null
+  - name: lgc
+    tests:
+    - not_null
+  - name: lgc_rank
+    tests:
+      - dbt_utils.accepted_range:
+          min_value: 1
+          max_value: 9  # a segment shouldn't have more than 9 lgcs
+          inclusive: true
+  - name: board_of_elections_lgc_pointer
+
+- name: int__centerline_streetcode_and_facecode
+  description: Centerline segments joined to street codes and face codes
+  columns:
+  - name: segmentid
+    tests:
+    - not_null
+    - unique
+  - name: lgc1
+    tests:
+    - not_null
+  - name: lgc2
+  - name: lgc3
+  - name: lgc4
+  - name: lgc5
+  - name: lgc6
+  - name: lgc7
+  - name: lgc8
+  - name: lgc9
+  - name: board_of_elections_lgc_pointer
+    tests:
+    - not_null
+  - name: five_digit_street_code
+    tests:
+    - not_null
+  - name: face_code
+    tests:
+    - not_null
+  # the test below should be here but dbt errors while building staging tables
+  # tests:
+  # - dbt_expectations.expect_table_row_count_to_equal_other_table:
+  #    compare_model: ref("stg__centerline")
path: products/lion/models/intermediate/2.2.1/_int_221.yml
user: login: fvankrieken
body: Huh that really shouldn't be... I'm gonna poke around
created_at: 2025-05-01T16:38:52Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,88 @@
+models:
+- name: int__preferred_local_group_code
+  description: Segments with preferred (primary) local group. Each segment can have only one local group
+  columns:
+  - name: segmentid
+    tests:
+    - not_null
+    - unique
+  - name: lgc
+  - name: lgc_rank
+    tests:
+      - accepted_values:
+          values: [1]
+  - name: boe_preferred_lgc_flag
+  - name: b5sc
+  - name: b5sc_segmentid_rank
+  - name: b7sc
+  tests:
+    - dbt_utils.unique_combination_of_columns:
+        combination_of_columns:
+          - segmentid
+          - lgc
+
+- name: int__nonpreferred_local_group_code
+  description: Segments with non-preferred local group. A segment can have multiple local non-preferred groups
+  columns:
+  - name: segmentid
+    tests:
+    - not_null
+  - name: lgc
+  - name: lgc_rank
+    tests:
+      - dbt_utils.accepted_range:
+          min_value: 2
+          max_value: 9  # a segment shouldn't have more than 9 lgcs
path: products/lion/models/intermediate/2.2.1/_int_221.yml
user: login: fvankrieken
body: Longer term, we wouldn't want to error here but output to a report (while still generating LION), but that's obviously out of scope of the poc
created_at: 2025-05-01T19:03:01Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,88 @@
+models:
+- name: int__preferred_local_group_code
+  description: Segments with preferred (primary) local group. Each segment can have only one local group
+  columns:
+  - name: segmentid
+    tests:
+    - not_null
+    - unique
+  - name: lgc
+  - name: lgc_rank
+    tests:
+      - accepted_values:
+          values: [1]
+  - name: boe_preferred_lgc_flag
+  - name: b5sc
+  - name: b5sc_segmentid_rank
+  - name: b7sc
+  tests:
+    - dbt_utils.unique_combination_of_columns:
+        combination_of_columns:
+          - segmentid
+          - lgc
+
+- name: int__nonpreferred_local_group_code
+  description: Segments with non-preferred local group. A segment can have multiple local non-preferred groups
+  columns:
+  - name: segmentid
+    tests:
+    - not_null
+  - name: lgc
+  - name: lgc_rank
+    tests:
+      - dbt_utils.accepted_range:
+          min_value: 2
+          max_value: 9  # a segment shouldn't have more than 9 lgcs
+          inclusive: true
+  - name: boe_preferred_lgc_flag
+  - name: b5sc
+  - name: b7sc
+
+- name: int__all_local_group_code_ranked
+  description: Segments with all, preferred and non-preferred, local group codes, ranked by local group code.
+  columns:
+  - name: segmentid
+    tests:
+    - not_null
+  - name: lgc
+    tests:
+    - not_null
+  - name: lgc_rank
+    tests:
+      - dbt_utils.accepted_range:
+          min_value: 1
+          max_value: 9  # a segment shouldn't have more than 9 lgcs
+          inclusive: true
+  - name: board_of_elections_lgc_pointer
+
+- name: int__centerline_streetcode_and_facecode
+  description: Centerline segments joined to street codes and face codes
+  columns:
+  - name: segmentid
+    tests:
+    - not_null
+    - unique
+  - name: lgc1
+    tests:
+    - not_null
+  - name: lgc2
+  - name: lgc3
+  - name: lgc4
+  - name: lgc5
+  - name: lgc6
+  - name: lgc7
+  - name: lgc8
+  - name: lgc9
+  - name: board_of_elections_lgc_pointer
+    tests:
+    - not_null
+  - name: five_digit_street_code
+    tests:
+    - not_null
+  - name: face_code
+    tests:
+    - not_null
+  # the test below should be here but dbt errors while building staging tables
+  # tests:
+  # - dbt_expectations.expect_table_row_count_to_equal_other_table:
+  #    compare_model: ref("stg__centerline")
path: products/lion/models/intermediate/2.2.1/_int_221.yml
user: login: fvankrieken
body: Seems like an odd bug due to the fact that our build step has [multiple commands for the different stages](https://github.com/NYCPlanning/data-engineering/blob/main/.github/workflows/lion_build.yml#L79), and since this test has a reference to a staging table dbt somehow is messing up when it should be run (and running it only after building staging tables rather than building the table the test actually corresponds to)

Fixed action [here](https://github.com/NYCPlanning/data-engineering/actions/runs/14779853699) by [just running `dbt build`](https://github.com/NYCPlanning/data-engineering/blob/fvk-sf-lion/.github/workflows/lion_build.yml#L72), which I think is fine, we don't get that much out of separate bash commands and print statements
created_at: 2025-05-01T19:06:04Z
author_association: CONTRIBUTOR

PR comments:
user: login: fvankrieken
created_at: 2025-05-01T16:40:24Z
body: Looks great. My only notes
- run tests for boe pointer
- decide whether we want to "fix" [this](https://github.com/NYCPlanning/data-engineering/issues/1566#issuecomment-2845203494) or note it in #1597 
- I'm gonna quickly look into the weird failing dbt test
author_association: CONTRIBUTOR
user: login: sf-dcp
created_at: 2025-05-01T18:02:32Z
body: * I ran the tests for the boe pointer - please see the issue. Unfortunately got a big chunk of mismatched records back.

* I added a note to the issue

author_association: CONTRIBUTOR
user: login: fvankrieken
created_at: 2025-05-01T19:10:03Z
body: > Approving with a couple github requests
> 
> * detail issues around that single LGC3 mismatch in [CSCL ETL POC - lingering data questions #1597](https://github.com/NYCPlanning/data-engineering/issues/1597) (i.e. are duplicate LGCs allowed?)
> * " for BOE flag. It really seems like this is a bug in the existing ETL
> * up to you if you want to get that test working. As I linked in the comment, it's a pretty simple fix (to a very strange bug IMO) to get it working. Happy to punt it towards general pipeline cleanup

And nice work - this issue/section feels particularly... demonstrative of the quirks of the ETL logic and jargon.
author_association: CONTRIBUTOR
user: login: fvankrieken
created_at: 2025-05-01T19:10:29Z
body: Also oops merge conflict from curve-related fields that you have to resolve, should be super simple
author_association: CONTRIBUTOR
user: login: sf-dcp
created_at: 2025-05-01T20:17:58Z
body: > Approving with a couple github requests
> 
> * detail issues around that single LGC3 mismatch in [CSCL ETL POC - lingering data questions #1597](https://github.com/NYCPlanning/data-engineering/issues/1597) (i.e. are duplicate LGCs allowed?)
> * " for BOE flag. It really seems like this is a bug in the existing ETL
> * up to you if you want to get that test working. As I linked in the comment, it's a pretty simple fix (to a very strange bug IMO) to get it working. Happy to punt it towards general pipeline cleanup

I did make a note of the first one in the linked GitHub issue. Regarding the test, I would rather keep the current commands separate; it def speeds up development where I only need to rebuild a subset of models while testing queries.
author_association: CONTRIBUTOR