# Cleanse API Cloud-Check Logs (PySpark)

## Prompt
I need you to generate a PySpark script that cleanses the "API cloud-check log" hourly table from ODS and synchronizes it to the DWD layer.

**Business Background and Objective**: API-SDK cloud-check writes a raw log to ODS every hour (containing the caller's phone number `phonenum`/`phonenum_reg`, device IMEI/GUID/IMSI, calling IP (integer), query type, etc.). The downstream DWD layer wants: (1) filter out dirty data (empty phone number or IP, out-of-range IP); (2) use an IP library UDF to reverse-lookup country/province/city/district; (3) rename several fields and CAST them to the downstream schema.

**Input Table**:
- `internal_platform_db.case16_ods_comm_security_user_api_query_mgr_hi_v3`

**Data Range and Filter Conditions (business description)**:
  - `ds` = current day partition; for this run, `ds` is fixed as `'20260513'` (original `sys.argv[1]` sandboxed and hardcoded).
  - `phonenum_reg IS NOT NULL`
  - `phonenum IS NOT NULL`
  - `ip IS NOT NULL`
  - `ip BETWEEN 0 AND 4294967295` (valid uint32 range, to avoid dirty data)

**Table Join Relationships**: (No joins, single-table cleansing.)

**Aggregation and Computation Rules (must be reflected in the SQL)**:
  1. Column renaming / CAST:
     - `phonenum_reg` → `phone_num`
     - `phonenum` → `phone_num_original`
     - `imei` → `imei_reg` (retain the original `imei` as well)
     - `time` → `log_time`
     - `query_type` / `calltype` / `phonetype` / `src` all CAST AS STRING
  2. Derived IP reverse-lookup columns `ip_country` / `ip_province` / `ip_city` / `ip_district`:
     - The original task reverse-looks up via the `get_ip_info(ip)[0..3]` UDF; this UDF's jar (`hdfs://cluster-alpha/...`) is unavailable in the sandbox, so the sandbox version replaces it with constants `lit('CN')` / `lit('default_province')` / `lit('default_city')` / `lit('default_district')`.
     - The original task also has a pure Python `int_to_ip_string` UDF (using only socket+struct, available in the sandbox), but since the output columns have been replaced with `lit`, it is no longer referenced and is removed to keep the GT clean.

**Output Requirements**:
  - Target table: `internal_platform_db.case16_dwd_comm_security_user_api_query_mgr_hi_v3`
  - Output table schema (17 columns, in this order, with the following semantics):
    1. `phone_num` STRING: normalized phone number
    2. `phone_num_original` STRING: original phone number
    3. `imei_reg` STRING: normalized IMEI (i.e., `imei`)
    4. `imei` STRING: original IMEI
    5. `guid` STRING, 6. `imsi` STRING: device identifiers
    7. `log_time` STRING: log time (i.e., `time`)
    8. `ip` BIGINT: query IP (integer)
    9. `query_type` STRING, 10. `calltype` STRING, 11. `phonetype` STRING, 12. `src` STRING: query categories
    13. `ip_country` STRING: country (sandbox constant `'CN'`)
    14. `ip_province` STRING: province (sandbox constant `'default_province'`)
    15. `ip_city` STRING: city (sandbox constant `'default_city'`)
    16. `ip_district` STRING: district (sandbox constant `'default_district'`)
    17. `ds` BIGINT: business date (originally a partition column, sandboxed as a regular column)
  - Write strategy: `INSERT OVERWRITE TABLE`, using an explicit column name list (17 columns).
  - Sandbox note: `PARTITION(ds={ds})` is changed to a regular column + `WHERE` filter; the `get_ip_info` UDF is sandboxed as `lit` constants; `addJar` + `registerJavaFunction("get_ip_info", ...)` are removed; `ArrayType` / `StringType` imports are removed (no longer needed).
  - If the target table does not exist, first create it using standard Hive format (ORC storage), then write the data.

## Reference Solution
# data-mocker rewrite-gt
# source: v7 xlsx row 17 (case16)
# task_name: t_dwd_comm_security_user_api_query_mgr_hi_copilot
# rewrite rules:
#   1) internal_platform_db.t_ods_comm_security_user_api_query_mgr_hi_copilot
#       -> internal_platform_db.case16_ods_comm_security_user_api_query_mgr_hi_v3
#   2) internal_platform_db.t_dwd_comm_security_user_api_query_mgr_hi_copilot
#       -> internal_platform_db.case16_dwd_comm_security_user_api_query_mgr_hi_v3
#   3) PARTITION(ds={ds}) 删除；output 表 ds 改为普通列，SELECT 末尾补 ds
#      WHERE ds = {ds} 保留（普通列过滤）
#      INSERT 用显式列名避免列序错位
#   4) sys.argv[1] -> 硬编码 '20260513'
#   5) get_ip_info(...)[0..3] UDF -> lit('CN'/'default'/'default'/'default')
#      (原 jar hdfs://cluster-alpha/... 在沙箱不可用，同 case15 OpenidCrypt 处理)
#   6) 移除 addJar + registerJavaFunction("get_ip_info", ...)
#   7) int_to_ip_string Python UDF 保留（纯 socket+struct 沙箱可用），
#      但因为输出列已 lit 替换，实际不再被引用 → 一并删除以保持 GT 干净
#   8) 删除 ArrayType/StringType import（已不再需要）
# ---
import sys
from pyspark.sql import SparkSession

"""

脚本说明：API-SDK云查（lit 替换 IP 反查 UDF 版）

"""


# 生成和配置spark实例
def get_spark_cli():
    spark = SparkSession.builder \
        .enableHiveSupport() \
        .config("spark.sql.storeAssignmentPolicy", "LEGACY") \
        .getOrCreate()
    return spark


# 生成spark sql语句
def get_sql_str():
    ds = '20260513'  # was: sys.argv[1]
    sql = f"""
INSERT OVERWRITE TABLE internal_platform_db.case16_dwd_comm_security_user_api_query_mgr_hi_v3
(phone_num, phone_num_original, imei_reg, imei, guid, imsi, log_time, ip, query_type, calltype, phonetype, src, ip_country, ip_province, ip_city, ip_district, ds)
SELECT
    phonenum_reg AS phone_num,
    phonenum AS phone_num_original,
    imei AS imei_reg,
    imei,
    guid,
    imsi,
    time AS log_time,
    ip,
    CAST(query_type AS STRING) AS query_type,
    CAST(calltype AS STRING) AS calltype,
    CAST(phonetype AS STRING) AS phonetype,
    CAST(src AS STRING) AS src,
    'CN' AS ip_country,
    'default_province' AS ip_province,
    'default_city' AS ip_city,
    'default_district' AS ip_district,
    ds
FROM internal_platform_db.case16_ods_comm_security_user_api_query_mgr_hi_v3
WHERE ds = {ds}
  AND phonenum_reg IS NOT NULL
  AND phonenum IS NOT NULL
  AND ip IS NOT NULL
  AND ip BETWEEN 0 AND 4294967295
"""
    print(f"sql: {sql}")
    return sql


# 执行
if __name__ == "__main__":
    spark = get_spark_cli()

    # 确保输出表以正确 schema 存在
    spark.sql("DROP TABLE IF EXISTS internal_platform_db.case16_dwd_comm_security_user_api_query_mgr_hi_v3")
    spark.sql('''
CREATE TABLE internal_platform_db.case16_dwd_comm_security_user_api_query_mgr_hi_v3 (
    `phone_num` STRING,
    `phone_num_original` STRING,
    `imei_reg` STRING,
    `imei` STRING,
    `guid` STRING,
    `imsi` STRING,
    `log_time` STRING,
    `ip` BIGINT,
    `query_type` STRING,
    `calltype` STRING,
    `phonetype` STRING,
    `src` STRING,
    `ip_country` STRING,
    `ip_province` STRING,
    `ip_city` STRING,
    `ip_district` STRING,
    `ds` BIGINT
) STORED AS ORCFILE
''')

    SQL = get_sql_str()
    spark.sql(SQL)