Repository: prefeitura-rio/pipelines_rj_smtr
Pull request #500: Corrige teste de tratamento do gps
Author: login: vtr363
Created: 2025-03-27T15:16:43Z | Merged: 2025-03-27T16:17:55Z



<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit

- **Documentação**
  - Atualizado o changelog para a versão 1.1.4 (27 de março de 2025), reforçando os aprimoramentos registrados.
- **Testes**
  - Refinadas as validações dos dados de GPS para garantir o tratamento eficaz de registros duplicados.
- **Chores**
  - Ajustada a informação de versão para manter a consistência interna.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->


Diff:
diff --git a/pipelines/migration/br_rj_riodejaneiro_onibus_gps/flows.py b/pipelines/migration/br_rj_riodejaneiro_onibus_gps/flows.py
index f241ac7c5..79c0fd078 100644
--- a/pipelines/migration/br_rj_riodejaneiro_onibus_gps/flows.py
+++ b/pipelines/migration/br_rj_riodejaneiro_onibus_gps/flows.py
@@ -2,7 +2,7 @@
 """
 Flows for br_rj_riodejaneiro_onibus_gps
 
-DBT 2024-08-20 2
+DBT 2025-03-27
 """
 
 from copy import deepcopy
diff --git a/pipelines/migration/projeto_subsidio_sppo/CHANGELOG.md b/pipelines/migration/projeto_subsidio_sppo/CHANGELOG.md
index 2a139c649..05431fc91 100644
--- a/pipelines/migration/projeto_subsidio_sppo/CHANGELOG.md
+++ b/pipelines/migration/projeto_subsidio_sppo/CHANGELOG.md
@@ -1,5 +1,11 @@
 # Changelog - projeto_subsidio_sppo
 
+## [1.1.4] - 2025-03-27
+
+### Alterado
+
+- Alterado teste `test_check_gps_treatment.sql` para desconsiderar duplicações assim como o modelo `sppo_aux_registros_filtrada.sql` (https://github.com/prefeitura-rio/pipelines_rj_smtr/pull/500)
+
 ## [1.1.3] - 2025-03-21
 
 ### Adicionado
diff --git a/pipelines/migration/projeto_subsidio_sppo/flows.py b/pipelines/migration/projeto_subsidio_sppo/flows.py
index e1377f9a7..f6f052982 100644
--- a/pipelines/migration/projeto_subsidio_sppo/flows.py
+++ b/pipelines/migration/projeto_subsidio_sppo/flows.py
@@ -3,7 +3,7 @@
 """
 Flows for projeto_subsidio_sppo
 
-DBT: 2025-03-26
+DBT: 2025-03-27
 """
 
 from prefect import Parameter, case, task
diff --git a/queries/macros/test_check_gps_treatment.sql b/queries/macros/test_check_gps_treatment.sql
index 91502ce17..e2ded570c 100644
--- a/queries/macros/test_check_gps_treatment.sql
+++ b/queries/macros/test_check_gps_treatment.sql
@@ -1,104 +1,86 @@
 {% test check_gps_treatment(model) -%}
-WITH
-            data_hora AS (
-                SELECT
-                    EXTRACT(date
-                    FROM
-                    timestamp_array) AS DATA,
-                    EXTRACT(hour
-                    FROM
-                    timestamp_array) AS hora,
-                FROM
-                    UNNEST(GENERATE_TIMESTAMP_ARRAY("{{ var('date_range_start') }}", "{{ var('date_range_end') }}", INTERVAL 1 hour)) AS timestamp_array ),
-            gps_raw AS (
-                SELECT
-                    EXTRACT(date
-                    FROM
-                    timestamp_gps) AS DATA,
-                    EXTRACT(hour
-                    FROM
-                    timestamp_gps) AS hora,
-                    COUNT(*) AS q_gps_raw
-                FROM
-                    -- `rj-smtr.br_rj_riodejaneiro_onibus_gps.sppo_registros`
-                    {{ ref('sppo_registros') }}
-                WHERE
-                    DATA BETWEEN DATE("{{ var('date_range_start') }}")
-                    AND DATE("{{ var('date_range_end') }}")
-                GROUP BY
-                    1,
-                    2 ),
-            gps_filtrada AS (
-                SELECT
-                    EXTRACT(date
-                            FROM
-                            timestamp_gps) AS DATA,
-                    EXTRACT(hour
-                    FROM
-                    timestamp_gps) AS hora,
-                    COUNT(*) AS q_gps_filtrada
-                FROM
-                    -- `rj-smtr.br_rj_riodejaneiro_onibus_gps.sppo_aux_registros_filtrada`
-                    {{ ref('sppo_aux_registros_filtrada') }}
-                WHERE
-                    DATA BETWEEN DATE("{{ var('date_range_start') }}")
-                    AND DATE("{{ var('date_range_end') }}")
-                GROUP BY
-                    1,
-                    2 ),
-            gps_sppo AS (
-                SELECT
-                    DATA,
-                    EXTRACT(hour
-                    FROM
-                    timestamp_gps) AS hora,
-                    COUNT(*) AS q_gps_treated
-                FROM
-                    -- `rj-smtr.br_rj_riodejaneiro_veiculos.gps_sppo`
-                    {{ ref('gps_sppo') }}
-                WHERE
-                    DATA BETWEEN DATE("{{ var('date_range_start') }}")
-                    AND DATE("{{ var('date_range_end') }}")
-                GROUP BY
-                    1,
-                    2),
-            gps_join AS (
-                SELECT
-                    *,
-                    SAFE_DIVIDE(q_gps_filtrada, q_gps_raw) as indice_tratamento_raw,
-                    SAFE_DIVIDE(q_gps_treated, q_gps_filtrada) as indice_tratamento_filtrada,
-                    CASE
-                        WHEN    q_gps_raw = 0 OR q_gps_filtrada = 0 OR q_gps_treated = 0                -- Hipótese de perda de dados no tratamento
-                                OR q_gps_raw IS NULL OR q_gps_filtrada IS NULL OR q_gps_treated IS NULL -- Hipótese de perda de dados no tratamento
-                                OR (q_gps_raw <= q_gps_filtrada) OR (q_gps_filtrada < q_gps_treated)   -- Hipótese de duplicação de dados
-                                OR (COALESCE(SAFE_DIVIDE(q_gps_filtrada, q_gps_raw), 0) < 0.96)         -- Hipótese de perda de dados no tratamento (superior a 3%)
-                                OR (COALESCE(SAFE_DIVIDE(q_gps_treated, q_gps_filtrada), 0) < 0.96)     -- Hipótese de perda de dados no tratamento (superior a 3%)
-                                THEN FALSE
-                    ELSE
-                    TRUE
-                END
-                    AS status
-                FROM
-                    data_hora
-                LEFT JOIN
-                    gps_raw
-                USING
-                    (DATA,
-                    hora)
-                LEFT JOIN
-                    gps_filtrada
-                USING
-                    (DATA,
-                    hora)
-                LEFT JOIN
-                    gps_sppo
-                USING
-                    (DATA,
-                    hora))
-            SELECT
-                *
-            FROM
-                gps_join
-            WHERE
-                status IS FALSE
-{%- endtest %}
\ No newline at end of file
+    with
+        data_hora as (
+            select
+                extract(date from timestamp_array) as data,
+                extract(hour from timestamp_array) as hora,
+            from
+                unnest(
+                    generate_timestamp_array(
+                        "{{ var('date_range_start') }}",
+                        "{{ var('date_range_end') }}",
+                        interval 1 hour
+                    )
+                ) as timestamp_array
+        ),
+        gps as (
+            select data, timestamp_gps, latitude, longitude
+            from {{ ref('sppo_registros') }}
+            where
+                data between date("{{ var('date_range_start') }}") and date(
+                    "{{ var('date_range_end') }}"
+                )
+            qualify row_number() over (partition by ordem, timestamp_gps, linha) = 1
+        ),
+        gps_raw as (
+            select
+                extract(date from timestamp_gps) as data,
+                extract(hour from timestamp_gps) as hora,
+                count(*) as q_gps_raw
+            from gps
+            group by 1, 2
+        ),
+        gps_filtrada as (
+            select
+                extract(date from timestamp_gps) as data,
+                extract(hour from timestamp_gps) as hora,
+                count(*) as q_gps_filtrada
+            from
+                -- `rj-smtr.br_rj_riodejaneiro_onibus_gps.sppo_aux_registros_filtrada`
+                {{ ref("sppo_aux_registros_filtrada") }}
+            where
+                data between date("{{ var('date_range_start') }}") and date(
+                    "{{ var('date_range_end') }}"
+                )
+            group by 1, 2
+        ),
+        gps_sppo as (
+            select
+                data,
+                extract(hour from timestamp_gps) as hora,
+                count(*) as q_gps_treated
+            from
+                -- `rj-smtr.br_rj_riodejaneiro_veiculos.gps_sppo`
+                {{ ref("gps_sppo") }}
+            where
+                data between date("{{ var('date_range_start') }}") and date(
+                    "{{ var('date_range_end') }}"
+                )
+            group by 1, 2
+        ),
+        gps_join as (
+            select
+                *,
+                safe_divide(q_gps_filtrada, q_gps_raw) as indice_tratamento_raw,
+                safe_divide(
+                    q_gps_treated, q_gps_filtrada
+                ) as indice_tratamento_filtrada,
+                case
+                    when
+                        q_gps_raw = 0 or q_gps_filtrada = 0 or q_gps_treated = 0  -- Hipótese de perda de dados no tratamento
+                        or q_gps_raw is null or q_gps_filtrada is null or q_gps_treated is null -- Hipótese de perda de dados no tratamento
+                        or (q_gps_raw <= q_gps_filtrada) or (q_gps_filtrada < q_gps_treated) -- Hipótese de duplicação de dados
+                        or (coalesce(safe_divide(q_gps_filtrada, q_gps_raw), 0) < 0.96) -- Hipótese de perda de dados no tratamento (superior a 3%)
+                        or (coalesce(safe_divide(q_gps_treated, q_gps_filtrada), 0) < 0.96) -- Hipótese de perda de dados no tratamento (superior a 3%)
+                    then false
+                    else true
+                end as status
+            from data_hora
+            left join gps_raw using (data, hora)
+            left join gps_filtrada using (data, hora)
+            left join gps_sppo using (data, hora)
+        )
+    select *
+    from gps_join
+    where status is false
+{%- endtest %}


Reviews:
user: login: eng-rodrigocunha
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-27T15:21:20Z
user: login: eng-rodrigocunha
state: CHANGES_REQUESTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-27T15:26:39Z
user: login: coderabbitai[bot]
body: **Actionable comments posted: 0**

<details>
<summary>🧹 Nitpick comments (2)</summary><blockquote>

<details>
<summary>pipelines/migration/projeto_subsidio_sppo/CHANGELOG.md (1)</summary><blockquote>

`3-8`: **Nova Seção na Changelog e Correção de Redação**  
A seção para a versão [1.1.4] foi adicionada corretamente, documentando a modificação do teste `test_check_gps_treatment.sql`.  
Observação: Na linha 7 há uma duplicação da preposição "para". Recomendo remover um dos "para" para aprimorar a redação.  
Exemplo de diff para correção:  
```diff
- Alterado teste `test_check_gps_treatment.sql` para para desconsiderar duplicações assim como o modelo `sppo_aux_registros_filtrada.sql` (https://github.com/prefeitura-rio/pipelines_rj_smtr/pull/500)
+ Alterado teste `test_check_gps_treatment.sql` para desconsiderar duplicações assim como o modelo `sppo_aux_registros_filtrada.sql` (https://github.com/prefeitura-rio/pipelines_rj_smtr/pull/500)
```

<details>
<summary>🧰 Tools</summary>

<details>
<summary>🪛 LanguageTool</summary>

[duplication] ~7-~7: Possível erro de escrita: você repetiu uma palavra
Context: ...do teste `test_check_gps_treatment.sql` para para desconsiderar duplicações assim como o ...

(PORTUGUESE_WORD_REPEAT_RULE)

</details>
<details>
<summary>🪛 markdownlint-cli2 (0.17.2)</summary>

7-7: Bare URL used
null

(MD034, no-bare-urls)

</details>

</details>

</blockquote></details>
<details>
<summary>queries/macros/test_check_gps_treatment.sql (1)</summary><blockquote>

`68-75`: **Verificação dos Limiar e Condições de Qualidade**  
A lógica definida no bloco `CASE` da CTE `gps_join` está bem estruturada, utilizando funções como `safe_divide` para calcular os índices de tratamento. Recomendo revisar os limiares (por exemplo, o valor 0.96) para confirmar que eles refletem adequadamente os requisitos de qualidade e as hipóteses de perda ou duplicação dos dados.

</blockquote></details>

</blockquote></details>

<details>
<summary>📜 Review details</summary>

**Configuration used: CodeRabbit UI**
**Review profile: CHILL**
**Plan: Pro**

<details>
<summary>📥 Commits</summary>

Reviewing files that changed from the base of the PR and between cd371f21b87718244330f6de8595daee9a2223c6 and 535405810b1d9f81473cabd0bac084f00d6fb994.

</details>

<details>
<summary>📒 Files selected for processing (4)</summary>

* `pipelines/migration/br_rj_riodejaneiro_onibus_gps/flows.py` (1 hunks)
* `pipelines/migration/projeto_subsidio_sppo/CHANGELOG.md` (1 hunks)
* `pipelines/migration/projeto_subsidio_sppo/flows.py` (1 hunks)
* `queries/macros/test_check_gps_treatment.sql` (1 hunks)

</details>

<details>
<summary>🧰 Additional context used</summary>

<details>
<summary>🪛 LanguageTool</summary>

<details>
<summary>pipelines/migration/projeto_subsidio_sppo/CHANGELOG.md</summary>

[duplication] ~7-~7: Possível erro de escrita: você repetiu uma palavra
Context: ...do teste `test_check_gps_treatment.sql` para para desconsiderar duplicações assim como o ...

(PORTUGUESE_WORD_REPEAT_RULE)

</details>

</details>
<details>
<summary>🪛 markdownlint-cli2 (0.17.2)</summary>

<details>
<summary>pipelines/migration/projeto_subsidio_sppo/CHANGELOG.md</summary>

7-7: Bare URL used
null

(MD034, no-bare-urls)

</details>

</details>

</details>

<details>
<summary>⏰ Context from checks skipped due to timeout of 90000ms (2)</summary>

* GitHub Check: Build and publish docs
* GitHub Check: Summary

</details>

<details>
<summary>🔇 Additional comments (4)</summary><blockquote>

<details>
<summary>pipelines/migration/projeto_subsidio_sppo/flows.py (1)</summary>

`6-6`: **Comentário DBT Atualizado**  
A alteração do comentário para "DBT: 2025-03-27" está correta e garante a consistência dos metadados com a nova data do DBT.

</details>
<details>
<summary>pipelines/migration/br_rj_riodejaneiro_onibus_gps/flows.py (1)</summary>

`5-6`: **Atualização da Data do DBT**  
A modificação do comentário para "DBT 2025-03-27" está adequada e alinha este fluxo com as atualizações realizadas em outros pipelines.

</details>
<details>
<summary>queries/macros/test_check_gps_treatment.sql (2)</summary>

`16-24`: **Uso Eficiente do QUALIFY para Eliminação de Duplicatas**  
A CTE `gps` emprega o comando `QUALIFY row_number() over (partition by ordem, timestamp_gps, linha) = 1` para filtrar registros duplicados, o que é uma abordagem eficiente. Verifique se as colunas `ordem`, `timestamp_gps` e `linha` existem na tabela referenciada por `{{ ref('sppo_registros') }}` para assegurar que o filtro funcione conforme o esperado.

---

`83-87`: **Confirmação da Consulta Final**  
A consulta que seleciona os registros com `status is false` está adequada para identificar problemas no tratamento dos dados GPS.

</details>

</blockquote></details>

</details>

<!-- This is an auto-generated comment by CodeRabbit for review status -->
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-27T15:30:55Z
user: login: eng-rodrigocunha
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-27T15:34:56Z
user: login: eng-rodrigocunha
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-27T16:17:48Z

Review comments:
diff_hunk: @@ -1,104 +1,86 @@
 {% test check_gps_treatment(model) -%}
-WITH
-            data_hora AS (
-                SELECT
-                    EXTRACT(date
-                    FROM
-                    timestamp_array) AS DATA,
-                    EXTRACT(hour
-                    FROM
-                    timestamp_array) AS hora,
-                FROM
-                    UNNEST(GENERATE_TIMESTAMP_ARRAY("{{ var('date_range_start') }}", "{{ var('date_range_end') }}", INTERVAL 1 hour)) AS timestamp_array ),
-            gps_raw AS (
-                SELECT
-                    EXTRACT(date
-                    FROM
-                    timestamp_gps) AS DATA,
-                    EXTRACT(hour
-                    FROM
-                    timestamp_gps) AS hora,
-                    COUNT(*) AS q_gps_raw
-                FROM
-                    -- `rj-smtr.br_rj_riodejaneiro_onibus_gps.sppo_registros`
-                    {{ ref('sppo_registros') }}
-                WHERE
-                    DATA BETWEEN DATE("{{ var('date_range_start') }}")
-                    AND DATE("{{ var('date_range_end') }}")
-                GROUP BY
-                    1,
-                    2 ),
-            gps_filtrada AS (
-                SELECT
-                    EXTRACT(date
-                            FROM
-                            timestamp_gps) AS DATA,
-                    EXTRACT(hour
-                    FROM
-                    timestamp_gps) AS hora,
-                    COUNT(*) AS q_gps_filtrada
-                FROM
-                    -- `rj-smtr.br_rj_riodejaneiro_onibus_gps.sppo_aux_registros_filtrada`
-                    {{ ref('sppo_aux_registros_filtrada') }}
-                WHERE
-                    DATA BETWEEN DATE("{{ var('date_range_start') }}")
-                    AND DATE("{{ var('date_range_end') }}")
-                GROUP BY
-                    1,
-                    2 ),
-            gps_sppo AS (
-                SELECT
-                    DATA,
-                    EXTRACT(hour
-                    FROM
-                    timestamp_gps) AS hora,
-                    COUNT(*) AS q_gps_treated
-                FROM
-                    -- `rj-smtr.br_rj_riodejaneiro_veiculos.gps_sppo`
-                    {{ ref('gps_sppo') }}
-                WHERE
-                    DATA BETWEEN DATE("{{ var('date_range_start') }}")
-                    AND DATE("{{ var('date_range_end') }}")
-                GROUP BY
-                    1,
-                    2),
-            gps_join AS (
-                SELECT
-                    *,
-                    SAFE_DIVIDE(q_gps_filtrada, q_gps_raw) as indice_tratamento_raw,
-                    SAFE_DIVIDE(q_gps_treated, q_gps_filtrada) as indice_tratamento_filtrada,
-                    CASE
-                        WHEN    q_gps_raw = 0 OR q_gps_filtrada = 0 OR q_gps_treated = 0                -- Hipótese de perda de dados no tratamento
-                                OR q_gps_raw IS NULL OR q_gps_filtrada IS NULL OR q_gps_treated IS NULL -- Hipótese de perda de dados no tratamento
-                                OR (q_gps_raw <= q_gps_filtrada) OR (q_gps_filtrada < q_gps_treated)   -- Hipótese de duplicação de dados
-                                OR (COALESCE(SAFE_DIVIDE(q_gps_filtrada, q_gps_raw), 0) < 0.96)         -- Hipótese de perda de dados no tratamento (superior a 3%)
-                                OR (COALESCE(SAFE_DIVIDE(q_gps_treated, q_gps_filtrada), 0) < 0.96)     -- Hipótese de perda de dados no tratamento (superior a 3%)
-                                THEN FALSE
-                    ELSE
-                    TRUE
-                END
-                    AS status
-                FROM
-                    data_hora
-                LEFT JOIN
-                    gps_raw
-                USING
-                    (DATA,
-                    hora)
-                LEFT JOIN
-                    gps_filtrada
-                USING
-                    (DATA,
-                    hora)
-                LEFT JOIN
-                    gps_sppo
-                USING
-                    (DATA,
-                    hora))
-            SELECT
-                *
-            FROM
-                gps_join
-            WHERE
-                status IS FALSE
-{%- endtest %}
\ No newline at end of file
+    with
+        data_hora as (
+            select
+                extract(date from timestamp_array) as data,
+                extract(hour from timestamp_array) as hora,
+            from
+                unnest(
+                    generate_timestamp_array(
+                        "{{ var('date_range_start') }}",
+                        "{{ var('date_range_end') }}",
+                        interval 1 hour
+                    )
+                ) as timestamp_array
+        ),
+        gps as (
+            select data, timestamp_gps, latitude, longitude
+            from `rj-smtr.br_rj_riodejaneiro_onibus_gps.sppo_registros`
path: queries/macros/test_check_gps_treatment.sql
user: login: eng-rodrigocunha
body: ```suggestion
            from {{ ref('sppo_registros') }}
```
created_at: 2025-03-27T15:19:53Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1,104 +1,86 @@
 {% test check_gps_treatment(model) -%}
-WITH
-            data_hora AS (
-                SELECT
-                    EXTRACT(date
-                    FROM
-                    timestamp_array) AS DATA,
-                    EXTRACT(hour
-                    FROM
-                    timestamp_array) AS hora,
-                FROM
-                    UNNEST(GENERATE_TIMESTAMP_ARRAY("{{ var('date_range_start') }}", "{{ var('date_range_end') }}", INTERVAL 1 hour)) AS timestamp_array ),
-            gps_raw AS (
-                SELECT
-                    EXTRACT(date
-                    FROM
-                    timestamp_gps) AS DATA,
-                    EXTRACT(hour
-                    FROM
-                    timestamp_gps) AS hora,
-                    COUNT(*) AS q_gps_raw
-                FROM
-                    -- `rj-smtr.br_rj_riodejaneiro_onibus_gps.sppo_registros`
-                    {{ ref('sppo_registros') }}
-                WHERE
-                    DATA BETWEEN DATE("{{ var('date_range_start') }}")
-                    AND DATE("{{ var('date_range_end') }}")
-                GROUP BY
-                    1,
-                    2 ),
-            gps_filtrada AS (
-                SELECT
-                    EXTRACT(date
-                            FROM
-                            timestamp_gps) AS DATA,
-                    EXTRACT(hour
-                    FROM
-                    timestamp_gps) AS hora,
-                    COUNT(*) AS q_gps_filtrada
-                FROM
-                    -- `rj-smtr.br_rj_riodejaneiro_onibus_gps.sppo_aux_registros_filtrada`
-                    {{ ref('sppo_aux_registros_filtrada') }}
-                WHERE
-                    DATA BETWEEN DATE("{{ var('date_range_start') }}")
-                    AND DATE("{{ var('date_range_end') }}")
-                GROUP BY
-                    1,
-                    2 ),
-            gps_sppo AS (
-                SELECT
-                    DATA,
-                    EXTRACT(hour
-                    FROM
-                    timestamp_gps) AS hora,
-                    COUNT(*) AS q_gps_treated
-                FROM
-                    -- `rj-smtr.br_rj_riodejaneiro_veiculos.gps_sppo`
-                    {{ ref('gps_sppo') }}
-                WHERE
-                    DATA BETWEEN DATE("{{ var('date_range_start') }}")
-                    AND DATE("{{ var('date_range_end') }}")
-                GROUP BY
-                    1,
-                    2),
-            gps_join AS (
-                SELECT
-                    *,
-                    SAFE_DIVIDE(q_gps_filtrada, q_gps_raw) as indice_tratamento_raw,
-                    SAFE_DIVIDE(q_gps_treated, q_gps_filtrada) as indice_tratamento_filtrada,
-                    CASE
-                        WHEN    q_gps_raw = 0 OR q_gps_filtrada = 0 OR q_gps_treated = 0                -- Hipótese de perda de dados no tratamento
-                                OR q_gps_raw IS NULL OR q_gps_filtrada IS NULL OR q_gps_treated IS NULL -- Hipótese de perda de dados no tratamento
-                                OR (q_gps_raw <= q_gps_filtrada) OR (q_gps_filtrada < q_gps_treated)   -- Hipótese de duplicação de dados
-                                OR (COALESCE(SAFE_DIVIDE(q_gps_filtrada, q_gps_raw), 0) < 0.96)         -- Hipótese de perda de dados no tratamento (superior a 3%)
-                                OR (COALESCE(SAFE_DIVIDE(q_gps_treated, q_gps_filtrada), 0) < 0.96)     -- Hipótese de perda de dados no tratamento (superior a 3%)
-                                THEN FALSE
-                    ELSE
-                    TRUE
-                END
-                    AS status
-                FROM
-                    data_hora
-                LEFT JOIN
-                    gps_raw
-                USING
-                    (DATA,
-                    hora)
-                LEFT JOIN
-                    gps_filtrada
-                USING
-                    (DATA,
-                    hora)
-                LEFT JOIN
-                    gps_sppo
-                USING
-                    (DATA,
-                    hora))
-            SELECT
-                *
-            FROM
-                gps_join
-            WHERE
-                status IS FALSE
-{%- endtest %}
\ No newline at end of file
+    with
+        data_hora as (
+            select
+                extract(date from timestamp_array) as data,
+                extract(hour from timestamp_array) as hora,
+            from
+                unnest(
+                    generate_timestamp_array(
+                        "{{ var('date_range_start') }}",
+                        "{{ var('date_range_end') }}",
+                        interval 1 hour
+                    )
+                ) as timestamp_array
+        ),
+        gps as (
+            select data, timestamp_gps, latitude, longitude
+            from {{ ref('sppo_registros') }}
+            where
+                data between date("{{ var('date_range_start') }}") and date(
+                    "{{ var('date_range_end') }}"
+                )
+            qualify row_number() over (partition by ordem, timestamp_gps, linha) = 1
+        ),
+        gps_raw as (
+            select
+                extract(date from timestamp_gps) as data,
+                extract(hour from timestamp_gps) as hora,
+                count(*) as q_gps_raw
+            from gps
+            group by 1, 2
+        ),
path: queries/macros/test_check_gps_treatment.sql
user: login: eng-rodrigocunha
body: Avalie usar group by ao inves do qualify para reduzir consumo
created_at: 2025-03-27T15:26:31Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1,5 +1,11 @@
 # Changelog - projeto_subsidio_sppo
 
+## [1.1.4] - 2025-03-27
+
+### Alterado
+
+- Alterado teste `test_check_gps_treatment.sql` para para desconsiderar duplicações assim como o modelo `sppo_aux_registros_filtrada.sql` (https://github.com/prefeitura-rio/pipelines_rj_smtr/pull/500)
path: pipelines/migration/projeto_subsidio_sppo/CHANGELOG.md
user: login: eng-rodrigocunha
body: ```suggestion
- Alterado teste `test_check_gps_treatment.sql` para desconsiderar duplicações assim como o modelo `sppo_aux_registros_filtrada.sql` (https://github.com/prefeitura-rio/pipelines_rj_smtr/pull/500)
```
created_at: 2025-03-27T15:34:55Z
author_association: CONTRIBUTOR

PR comments:
user: login: coderabbitai[bot]
created_at: 2025-03-27T15:16:53Z
body: <!-- This is an auto-generated comment: summarize by coderabbit.ai -->
<!-- walkthrough_start -->

## Walkthrough

As alterações abrangem atualizações em comentários e na documentação dos pipelines, bem como uma reestruturação no teste SQL de tratamento de GPS. Dois arquivos `flows.py` tiveram seus comentários atualizados para refletir a nova data do DBT (2025-03-27). Além disso, foi adicionada uma nova seção na changelog para a versão 1.1.4 e o teste SQL (`test_check_gps_treatment.sql`) foi modificado, com alterações significativas nos CTEs para tratar duplicidades e otimizar a extração e agregação dos dados.

## Changes

| Arquivos                                                          | Resumo das Alterações |
|-------------------------------------------------------------------|-----------------------|
| `pipelines/migration/.../flows.py` (br_rj_riodejaneiro_onibus_gps e projeto_subsidio_sppo) | Atualização em comentários para alterar a data do DBT para 2025-03-27. |
| `pipelines/migration/.../CHANGELOG.md`                              | Adição de nova seção para a versão 1.1.4, documentando a modificação do teste SQL. |
| `queries/macros/test_check_gps_treatment.sql`                       | Reestruturação dos CTEs: inclusão do CTE `gps` com cláusula QUALIFY e ajustes nos CTEs `gps_raw`, `gps_filtrada` e `gps_join` para lidar com duplicidades. |

## Sequence Diagram(s)

```mermaid
sequenceDiagram
    participant TR as Runner de Teste
    participant S as Tabela sppo_registros
    participant G as CTE gps (filtro ROW_NUMBER)
    participant GR as CTE gps_raw (agregação)
    participant GF as CTE gps_filtrada
    participant GJ as CTE gps_join

    TR->>+G: Executa consulta de dados GPS
    G->>+S: Seleciona dados (timestamp, latitude, longitude)
    S-->>-G: Retorna registros
    G->>+GR: Agrega dados filtrados
    GR-->>-G: Retorna agregação
    G->>+GF: Filtra registros usando sppo_aux_registros_filtrada
    GF-->>-G: Retorna dados filtrados
    G->>+GJ: Junta dados para tratamento e índices
    GJ-->>-TR: Retorna resultado final
```

## Possibly related PRs

- **prefeitura-rio/pipelines_rj_smtr#492**: Alterações em comentários dos arquivos `flows.py` com atualização de datas, conectando alterações de forma direta.

## Suggested reviewers

- akaBotelho
- pixuimpou

## Poem

> Sou um coelhinho saltitante,  
> Vejo mudanças com olhar brilhante.  
> Datas novas pulam no comentário,  
> SQL se transforma com caixa de encanto.  
> Com o CodeRabbit, sigo a saltitar,  
> Celebrando o código que vem a brilhar! 🐇

<!-- walkthrough_end -->


---

<details>
<summary>📜 Recent review details</summary>

**Configuration used: CodeRabbit UI**
**Review profile: CHILL**
**Plan: Pro**

<details>
<summary>📥 Commits</summary>

Reviewing files that changed from the base of the PR and between 535405810b1d9f81473cabd0bac084f00d6fb994 and 498e93f0d9920dd9e8e082b4c1518ded8fa03a2d.

</details>

<details>
<summary>📒 Files selected for processing (1)</summary>

* `pipelines/migration/projeto_subsidio_sppo/CHANGELOG.md` (1 hunks)

</details>

<details>
<summary>🧰 Additional context used</summary>

<details>
<summary>🪛 LanguageTool</summary>

<details>
<summary>pipelines/migration/projeto_subsidio_sppo/CHANGELOG.md</summary>

[uncategorized] ~7-~7: Encontrada possível ausência de vírgula.
Context: ...est_check_gps_treatment.sql` para desconsiderar duplicações assim como o modelo `sppo_aux_r...

(AI_PT_HYDRA_LEO_MISSING_COMMA)

</details>

</details>
<details>
<summary>🪛 markdownlint-cli2 (0.17.2)</summary>

<details>
<summary>pipelines/migration/projeto_subsidio_sppo/CHANGELOG.md</summary>

7-7: Bare URL used
null

(MD034, no-bare-urls)

</details>

</details>

</details>

<details>
<summary>⏰ Context from checks skipped due to timeout of 90000ms (2)</summary>

* GitHub Check: Build and publish docs
* GitHub Check: Summary

</details>

<details>
<summary>🔇 Additional comments (2)</summary><blockquote>

<details>
<summary>pipelines/migration/projeto_subsidio_sppo/CHANGELOG.md (2)</summary>

`3-8`: **Nova seção para a versão 1.1.4 adicionada corretamente**  
A nova seção, datada de 2025-03-27 e com o rótulo "Alterado", está de acordo com os objetivos do PR, documentando explicitamente a alteração que ajusta o teste de tratamento de GPS.

<details>
<summary>🧰 Tools</summary>

<details>
<summary>🪛 LanguageTool</summary>

[uncategorized] ~7-~7: Encontrada possível ausência de vírgula.
Context: ...est_check_gps_treatment.sql` para desconsiderar duplicações assim como o modelo `sppo_aux_r...

(AI_PT_HYDRA_LEO_MISSING_COMMA)

</details>
<details>
<summary>🪛 markdownlint-cli2 (0.17.2)</summary>

7-7: Bare URL used
null

(MD034, no-bare-urls)

</details>

</details>

---

`7-7`: **Sugestão de melhoria na formatação da URL e pontuação**  
A linha contém uma URL em formato "bare", o que aciona o aviso do markdownlint (MD034). Para melhorar a conformidade e a legibilidade, considere formatar a URL como um link Markdown, por exemplo:  
```
- Alterado teste `test_check_gps_treatment.sql` para desconsiderar duplicações assim como o modelo `sppo_aux_registros_filtrada.sql` ([#500](https://github.com/prefeitura-rio/pipelines_rj_smtr/pull/500))
```  
Além disso, verifique se é apropriado inserir uma vírgula para melhorar a clareza da frase.

<details>
<summary>🧰 Tools</summary>

<details>
<summary>🪛 LanguageTool</summary>

[uncategorized] ~7-~7: Encontrada possível ausência de vírgula.
Context: ...est_check_gps_treatment.sql` para desconsiderar duplicações assim como o modelo `sppo_aux_r...

(AI_PT_HYDRA_LEO_MISSING_COMMA)

</details>
<details>
<summary>🪛 markdownlint-cli2 (0.17.2)</summary>

7-7: Bare URL used
null

(MD034, no-bare-urls)

</details>

</details>

</details>

</blockquote></details>

</details>
<!-- internal state start -->


<!-- DwQgtGAEAqAWCWBnSTIEMB26CuAXA9mAOYCmGJATmriQCaQDG+Ats2bgFyQAOFk+AIwBWJBrngA3EsgEBPRvlqU0AgfFwA6NPEgQAfACgjoCEejqANiS4BhfBQrxSkGohqQlLqrjRsMBD3xIIm5EAwA5bGYBSi4AVgAGBIMAVURYyAlcCgBmADYco31jcCgyenwAMxwCYjJlGnomVnYuXn5hUXEpGXkmJSpVdS0dYpMoOFRUTBrCUnJvOgUW/y4qAHdIRCjmNAp5OQUBlTVNbV0wQxLTA254bhILeHJEAHpmJ294fAxXgQoAPoUIRA75KISYEjwCj4AE/eACbCIAEhN6VCz4daIDTcWQcAwAIiJBgAxCTIABBACSdQW1CW21Ye3kVUYsEwpDCBjgJA88Eq1UQsExyDQkGYin58AY1G+WACYuaflwKHlsF5AAN0SKcbINZBKvArAAaLYPBhSmUWCx9dkYIjPIgudUeenoRCIfAW+n0dbqWDO3kAEQAQtBMpREHKDTDmJACQAmBIJgAsYASAA4wEnIAmCS4gonk3F0zlswB2AkaGAIZAMO3ObDcWj05C4F1SChR+H21WVey7cQ/SCIlW0fDSSAYfAqtAWGh8TDySrYDBiOVzo4kU0Yh0MU32BT+GEWA0YzZ+9vPQMGo0kas8o80AAeKtZ7d5hqskAoJF2z2QVd6w5OhTUwehp3QWhaHUDdTxXNchwwOd1BZPhKBhPg7VoJ5e3ZGQSDIVVskUbAGCWQ9fwlKRaA0IwjDJSl5waOU2yCD8PFECw9llH5kFZEhn24exGn4PhuGwAQngYSB2Fg6QjHCH4SEJYkDAgMAjDuB5cOkd5Pl435eHwEQCABbYBCjGDYUQbhhNeGwAAkKXCABxABRAAZAB5VyNGYWh8SJAlSXJalaQaBkdmZfhqmA+0FIMCkpxITZ0nXYd8JHQisDQaClgCDj4tIXcDUPTtuywABGDQapTU0W1EgBZPZ61zctTSTBM4gfWsti6aNngYCxsCUUVxUlQ0ZSQgsb1cFUNXmgF61EABrFFQgBbISGoZUNEQABHCwNVNdYEDarKYiIucFwKoIYMQX8iD2ehaCbaTDMQMCniIDBHRQFVLwDDiJSUU8NVs4SATQbBnyBEgHTcGFkS/bI8rQfajo1XreWK3lUHHBgonYJYgfQH8SEqSgyHI2axVs0RLR4bBrQpg7sGkN8sFc9RHMkujQqYhdPtmjilGGnikP46pBOEihRMPCSpOlWT/HkrkoCU9BmK+PjRZdcXuN1jBpdkoSRIo8TJOk1XxHESd1mp8U8vxtUCf5So6OCoobm0x5nj0j4iGN15jNMmzJKs75zLs/BXm1LFdSC4lGPC+ZIvoRldn2WK2RArlHxggUKfRLpxtBy1DLp5ZlVVG8tXPbFcX1L9tzNRmprnG1ICbRr/rFt00A9L14B9SAyY40NwwACiDagxRDbAjXoaB8HwCwAEoYxYeMupLBIywTPJ8wCItutLCsqxrVA8cCSdp1nAUunQDBlyApDN36Nvd2lA8+CYY868zyYnHv6a8HFEAMEcNwTQNZcY/BfG+aoRUWC1yotoE2Pc1wNlAi/egXd+Afj4LpU2X90C/iwWgJ+Yg6ACwYmFHWItCoGy4pLViudZYWwqFbZWMk5L2y5EpcgqkQrqRuOzSg8BA5oCgfgN4S0VoMHWqiLav5drsExhYZOIVU40nTosTO0Uc6sjxgXF0ABlAAip5FwnNIAakUcoza211H+H1FlVcAwiDKS2E4P6nd/AUyRmRXA2BHD2hxmJJwzxNx2FYMOaAKhvzuSEr+D07Dp42GgO5RAW92RSGykRCuhpcETxdFRfAEhNzvhdBqFRGwTr2JUajKgLZGngSaZtSG+B9RZJydlf6v5uDcXIvQQ4YpyCbDqaEXp2TnTUH6lYMQyBGpoFNOINgbhfDcB3LKUJSgwIYHoBie06hRqfljPXbp8NEYkUQPqX8VNfxrnvNfZAkzIB9NVMNc540NSWJSBSTyVIABiABNfUEskS8gCKjSgP4RQjiHhRXK9iABK3kADqAJwgpCaiGdyaLp4b31NwPYds5T/UOBqewShmCNMWvATZPhmDcA2vcw59ANS4XZI0sg2xwlOh+N3Vc8AJG20cJOPYvJfw+ADrRMwtT6loHWAy5pRo0ZtM5Z05E3TZn9PybyK6WBZUYKWKOUB7Zta3WOfgPckTpnIgaZ8uZ05NhoCIMHBGrZXQ+G3nGDiHzHX6tOggb8HFHUAhaejfVT5ngczYiXamLz7HXJhnDJ6SA7lRo1a0tALdDxwt/PQVyAAFcxvqMZwJ1THYSsb0EASwXjeg151DIAkhQYS6QRx4BQO8kBEoKFIx2swXSLasBtsrYsgaPwBaPkNMhcGKihD4GeLGy6OUe7NjHgER5SyVQfm7aY00/5/AYP7i6RAvheQ/xkv2f+c4ibcXEL2FxuBa7PBguRUURytg+FCXWdUSjsTVsqcoVmwSxBhN5BuoiUZWVPBKbQU6/pyb9iJvxLAw0dq/j4Kss2aMMq5V/R6r1z0ZrGW/ekbE9ChYsT1sw3khs2F6wEubeWltma8NturIoMA7HFKkYFexABvAApLYtwbI1rstUTtd97Bp6g0eFvMTABffU14NQSMlW8XYsj5Gc2WkBpxyI317UOsdERPsrjjFVhUaoMNaj6LHkqVoP4VVbCMQcPoihlBDDOKMS4Bg7NKnUACeAtAnUkAkFIx2tBzI+HlhcWzpRIBxByHEFMCQ4gZiqgkAQVVaAAE5Kh5ZTOWHIMoBC0AKzIzMKZKhJFoHkSoAhivFZTClkLaWUzFYzCQYrOQmsleK0maCxWSADczAmAQKYGBVTiFVDMShaAZkqGgA+aAEz0GKD1iANdwuRei7F1KdAATlBS0AA= -->

<!-- internal state end -->
<!-- tips_start -->

---



<details>
<summary>🪧 Tips</summary>

### Chat

There are 3 ways to chat with [CodeRabbit](https://coderabbit.ai?utm_source=oss&utm_medium=github&utm_campaign=prefeitura-rio/pipelines_rj_smtr&utm_content=500):

- Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  - `I pushed a fix in commit <commit_id>, please review it.`
  - `Generate unit testing code for this file.`
  - `Open a follow-up GitHub issue for this discussion.`
- Files and specific lines of code (under the "Files changed" tab): Tag `@coderabbitai` in a new review comment at the desired location with your query. Examples:
  - `@coderabbitai generate unit testing code for this file.`
  -	`@coderabbitai modularize this function.`
- PR comments: Tag `@coderabbitai` in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  - `@coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.`
  - `@coderabbitai read src/utils.ts and generate unit testing code.`
  - `@coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.`
  - `@coderabbitai help me debug CodeRabbit configuration file.`

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

### CodeRabbit Commands (Invoked using PR comments)

- `@coderabbitai pause` to pause the reviews on a PR.
- `@coderabbitai resume` to resume the paused reviews.
- `@coderabbitai review` to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
- `@coderabbitai full review` to do a full review from scratch and review all the files again.
- `@coderabbitai summary` to regenerate the summary of the PR.
- `@coderabbitai generate docstrings` to [generate docstrings](https://docs.coderabbit.ai/finishing-touches/docstrings) for this PR.
- `@coderabbitai resolve` resolve all the CodeRabbit review comments.
- `@coderabbitai plan` to trigger planning for file edits and PR creation.
- `@coderabbitai configuration` to show the current CodeRabbit configuration for the repository.
- `@coderabbitai help` to get help.

### Other keywords and placeholders

- Add `@coderabbitai ignore` anywhere in the PR description to prevent this PR from being reviewed.
- Add `@coderabbitai summary` to generate the high-level summary at a specific location in the PR description.
- Add `@coderabbitai` anywhere in the PR title to generate the title automatically.

### CodeRabbit Configuration File (`.coderabbit.yaml`)

- You can programmatically configure CodeRabbit by adding a `.coderabbit.yaml` file to the root of your repository.
- Please see the [configuration documentation](https://docs.coderabbit.ai/guides/configure-coderabbit) for more information.
- If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: `# yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json`

### Documentation and Community

- Visit our [Documentation](https://docs.coderabbit.ai) for detailed information on how to use CodeRabbit.
- Join our [Discord Community](http://discord.gg/coderabbit) to get help, request features, and share feedback.
- Follow us on [X/Twitter](https://twitter.com/coderabbitai) for updates and announcements.

</details>

<!-- tips_end -->
author_association: CONTRIBUTOR
user: login: coderabbitai[bot]
created_at: 2025-03-27T15:34:14Z
body: > [!CAUTION]
> No docstrings were generated.
author_association: CONTRIBUTOR
user: login: github-actions[bot]
created_at: 2025-03-27T15:54:39Z
body: ### Análise da árvore de código

**Os seguintes arquivos são afetados diretamente por alterações realizadas nesse pull request:**
	- `pipelines/migration/projeto_subsidio_sppo/flows.py`
	- `pipelines/migration/br_rj_riodejaneiro_onibus_gps/flows.py`
author_association: NONE