# Sync Code Review review-diff from ODS to DWV Layer by Day (PySpark)

## Prompt
I need you to generate a PySpark script that synchronizes the "code review diff file details" from the ODS layer to the DWV layer by day for the code review scenario.

**Business Background and Objective**: The upstream databus produces a daily review diff detail (one row = one review comment + corresponding diff text) that lands in the ODS table. The downstream DWV layer wants to organize the data with a three-level date partition (year/month/day) for convenient date-based retrieval, and to deduplicate rows with the same id and diff within the same day. This task has the same structure as case5 (note diff), except the source/target tables are the review series.

**Input Table**:
- `internal_platform_db.case6_ods_code_v2_review_diff_files_diff_v3`

**Data Range and Filter Conditions (business description)**:
  - Only take data where the business date equals the current run date `ds`.
  - `ds` is fixed as `'20260513'` (sandboxed date; the original task passes this via `sys.argv[1]`; TaskType=63 does not pass command-line arguments, so it is hardcoded to avoid IndexError).

**Table Join Relationships**: (No joins, single-table extraction.)

**Aggregation and Computation Rules (must be reflected in the SQL)**:
  - Directly SELECT `id`, `diff` columns, and apply `DISTINCT` deduplication (only one row is retained for the same id and diff within the same day).
  - Prepend three partition columns at the beginning of the SELECT, derived by splitting `ds`: `year='2026'`, `month='05'`, `day='13'`.
  - Use `INSERT OVERWRITE TABLE` to overwrite the entire table.

**Output Requirements**:
  - Target table: `internal_platform_db.case6_dwv_code_v2_review_diff_files_diff_v3`
  - Output table schema (write in this order, types and business semantics as follows):
    1. `year` STRING: year, originally a partition column
    2. `month` STRING: month, originally a partition column
    3. `day` STRING: day, originally a partition column
    4. `id` BIGINT: review id
    5. `diff` STRING: diff text content
  - Write strategy: `INSERT OVERWRITE` full table overwrite, this write targets `year='2026'`, `month='05'`, `day='13'`.
  - Sandbox note: The original task writes to a `PARTITION(year, month, day)` partitioned table in production; in the sandbox version, the three partition columns have been sandboxed as regular columns.
  - If the target table does not exist, first create it using standard Hive format (ORC storage), then write the data.

## Reference Solution
# data-mocker rewrite-gt
# source: pyspark_0428_desensitized.jsonl idx=43924
# task_name: dwv_code_v2_review_diff_files_diff_d_f
# rewrite rules:
#   1) dept_t_yg_code.ods_code_v2_review_diff_files_diff_d_f
#       -> internal_platform_db.case6_ods_code_v2_review_diff_files_diff_v3
#   2) code.dwv_code_v2_review_diff_files_diff_d_f
#       -> internal_platform_db.case6_dwv_code_v2_review_diff_files_diff_v3
#   3) INSERT OVERWRITE TABLE ... PARTITION(year=...,month=...,day=...) SELECT id,diff
#       -> INSERT OVERWRITE TABLE ... SELECT '$year' AS year, '$month' AS month, '$day' AS day, id, diff
#      (沙箱用无分区表，分区列改为普通列)
#   4) sys.argv[1] -> 硬编码 '20260513'
#      (datawd PySpark TaskType=63 不直传命令行参数，硬编码避免 IndexError)
# ---
#!/usr/bin/env bash
from __future__ import print_function

import re
import sys
import os
import json

import datetime

from pyspark.sql import SparkSession

if __name__ == "__main__":

    print('PythonSQL start')
    #yyyyMMdd
    ds = '20260513'  # was: sys.argv[1]
    match = re.match(r"([\d]{4})([\d]{2})([\d]{2})", ds)
    daySub = match.group(1, 2, 3)

    ds_year = daySub[0]
    ds_month = daySub[1]
    ds_day = daySub[2]

    last_day = datetime.datetime(
        int(daySub[0]), int(daySub[1]), int(daySub[2]))
    last_day_str = last_day.strftime('%Y%m%d')

    print('last_day_str=%s' % last_day_str)
    print('ds_year=%s' % ds_year)
    print('ds_month=%s' % ds_month)
    print('ds_day=%s' % ds_day)

    spark = SparkSession.builder.appName('case6_dwv_code_v2_review_diff_files_diff_d_f').enableHiveSupport().getOrCreate()

    # 确保输出表以正确 schema 存在（agent 可能未建表或建了错误 schema）
    spark.sql("DROP TABLE IF EXISTS internal_platform_db.case6_dwv_code_v2_review_diff_files_diff_v3")
    spark.sql('''
CREATE TABLE internal_platform_db.case6_dwv_code_v2_review_diff_files_diff_v3 (
    `year` STRING COMMENT '年，原为分区列',
    `month` STRING COMMENT '月，原为分区列',
    `day` STRING COMMENT '日，原为分区列',
    `id` BIGINT COMMENT '-',
    `diff` STRING COMMENT '-'
) STORED AS ORC
''')

    sql = '''
            insert overwrite table internal_platform_db.case6_dwv_code_v2_review_diff_files_diff_v3
            SELECT
                    DISTINCT
                    '$year' AS year, '$month' AS month, '$day' AS day, id ,diff 
            FROM internal_platform_db.case6_ods_code_v2_review_diff_files_diff_v3 where databus_imp_date='$ds'
        '''

    sql = sql.replace('$ds', last_day_str)
    sql = sql.replace('$year', ds_year)
    sql = sql.replace('$month', ds_month)
    sql = sql.replace('$day', ds_day)

    print(sql)
    result = spark.sql(sql)
    print(result.show())
    spark.stop()