# AI Search Parse Quality Daily Detail (PySpark)

## Prompt
I need you to generate a PySpark script that produces the "AI Search Parse Quality Daily Detail".

Business Background: Compute daily details of AI search web page parse quality by site + directory dimensions. Take data from the past few days, aggregate success rate, index page ratio, short content ratio, and failure step counts by the (date_key, host, fld) dimension, and join with the Top500 site priority labels.

**Input Tables**:
- `internal_platform_db.case10_ai_engine_classify_parse_result_daily_v3`
- `internal_platform_db.case10_sec_app_hy_top_500_sites_tag_v1_v3`

  Data Range:

  - Fact table filter: `date_key >= '20260508' AND date_key <= '20260513'` (hardcoded, today='20260513', looking back 5 days)
  - Dimension table participates in full, no pre-filtering

  Processing Logic:

  1. Field preparation: Select the following columns from the fact table:

  - `date_key` as-is

  - `host` → renamed `source_host`

  - `when(col("is_error") == 0, 1).otherwise(0)` → `success` (`is_error` is STRING; relies on Spark implicit conversion to compare with integer 0)

  - `when(col("is_error") == 1, 1).otherwise(0)` → `failure`

  - `web_type` as-is

  - `from_json(col("parse_html"), schema_of_json('{"content": "string"}')).getField("content")` → `content`

  - `error_step` as-is

  - `fld` → renamed `source_fld`

  - Then `withColumn`: `content_length = length(content)`
  2. Four independent aggregations (all grouped by `date_key`, `source_host`, `source_fld`):

  - summary: `sum(success)` → `successful_parses`, `sum(failure)` → `failed_parses`, `count(*)` → `total_parses`

  - index_page_ratio: `sum(when(web_type=='索引页', 1).otherwise(0))` → `index_page_count`

  - content_length_ratio: first `filter(success==1)`, then `sum(when(content_length<50, 1).otherwise(0))` → `short_content_count`

  - error_reasons: first `filter(failure==1)`, then:

- `sum(when(error_step==1, 1).otherwise(0))` → `error_step_1_count`

- `sum(when(error_step==2, 1).otherwise(0))` → `error_step_2_count`

- `sum(when(error_step==3, 1).otherwise(0))` → `error_step_3_count`

- `sum(when(error_step.isNotNull(), 1).otherwise(0))` → `total_error_steps` (this field is not used or output subsequently)
  3. Merge: Left join the four aggregation results sequentially on `[date_key, source_host, source_fld]`

  4. Compute ratios:

  - `index_page_count_rate = index_page_count / total_parses`


  - `short_content_count_rate = short_content_count / successful_parses` (no divide-by-zero protection; produces null when `successful_parses=0`)
  5. Hardcoded empty fields:

  - `withColumn("error_info_list", lit(""))` — fixed empty string, no `collect_list`

  - `withColumn("is_host", lit(""))` — fixed empty string, not obtained from dimension table
  6. Join dimension table (OR condition):
  `final_result.join(`

`priority_info`, — contains only `host`, `fld`, `priority`

`(source_fld == priority_info.fld) OR (source_host == priority_info.host),`

`"left"`

  `)`
  6. Key: The condition is OR (matching on either fld or host), not AND. One main table record may match multiple dimension table rows, causing row inflation.
  7. Final select (16 columns, in this order):
  `date_key`, `source_host`→`host`, `source_fld`→`fld`, `is_host` (step5's `lit("")`), `priority` (from dimension table join), `successful_parses`, `failed_parses`, `total_parses`, `index_page_count`, `index_page_count_rate`, `short_content_count`, `short_content_count_rate`,
  `error_step_1_count`, `error_step_2_count`, `error_step_3_count`, `error_info_list` (step5's `lit("")`)
  8. Deduplication: `dropDuplicates(["date_key", "host", "fld", "priority", "successful_parses", "total_parses", "index_page_count"])` (to converge inflated rows from the OR join)
  9. Write: Write via `createOrReplaceTempView` + `INSERT OVERWRITE TABLE`, explicitly specifying column name order (the output table DDL column order differs from the DataFrame column order):

  - Output table column order: `host`, `fld`, `is_host`, `priority`, `successful_parses`, `failed_parses`, `total_parses`, `index_page_count`, `index_page_count_rate`, `short_content_count`, `short_content_count_rate`, `error_step_1_count`, `error_step_2_count`,
  `error_step_3_count`, `error_info_list`, `date_key`

  - All fields are written with `CAST AS STRING`

  Output table: `internal_platform_db.case10_ai_engine_classify_parse_result_daily_detail_v3`
  - Output table DDL column order (note that `date_key` is last): `host`, `fld`, `is_host`, `priority`, `successful_parses`, `failed_parses`, `total_parses`, `index_page_count`, `index_page_count_rate`, `short_content_count`, `short_content_count_rate`, `error_step_1_count`,
  `error_step_2_count`, `error_step_3_count`, `error_info_list`, `date_key`
  - All column types are STRING

## Reference Solution
# case10 rewrite-gt
# source: data/v7_gt_codes/case_row10_case_0043.py (v7 row 9, manifest_row 10)
# task_name: ai_engine_classify_parse_result_daily_detail
# rewrite rules:
#   1) internal_platform_db.ai_engine_classify_parse_result_daily_copilot
#       -> internal_platform_db.case10_ai_engine_classify_parse_result_daily_v3
#   2) internal_platform_db.sec_app_hy_top_500_sites_tag_v1_copilot
#       -> internal_platform_db.case10_sec_app_hy_top_500_sites_tag_v1_v3
#   3) internal_platform_db.ai_engine_classify_parse_result_daily_detail_copilot
#       -> internal_platform_db.case10_ai_engine_classify_parse_result_daily_detail_v3
#   4) today = datetime.now() -> 硬编码 today_str = '20260513'
#      ten_days_ago_str = '20260508'（today - 5 天，与原 timedelta(days=5) 一致）
#   5) 沙箱主表去 date_key 分区（thive UI 建分区表 INSERT VALUES 全 silent no-op）
#      -> 改为普通 STRING 列；GT 仍按 date_key 范围过滤
#   6) 末尾 final_result.write.mode("overwrite").insertInto(...) -> 改为
#      createOrReplaceTempView + spark.sql('INSERT OVERWRITE TABLE ... SELECT ...')
#      显式列名清单（16 列）避免 DataFrame select 列序与 output DDL 列序错位
# ---
import os
from pyspark.sql import SparkSession
from pyspark.sql.functions import col, count, when, length, sum as _sum, from_json, schema_of_json
from pyspark.sql.types import StringType
from datetime import datetime, timedelta
from pyspark.sql.functions import lit
from pyspark.sql import functions as F

spark = (
    SparkSession.builder
        .appName("case10_ai_engine_classify_parse_result_daily_detail")
        .enableHiveSupport()
        .config("spark.driver.memory", "6g")
        .config("spark.executor.cores", 6)
        .config("spark.executor.memory", "12g")
        .getOrCreate()
)

# 确保输出表以正确 schema 存在
spark.sql("DROP TABLE IF EXISTS internal_platform_db.case10_ai_engine_classify_parse_result_daily_detail_v3")
spark.sql('''
CREATE TABLE internal_platform_db.case10_ai_engine_classify_parse_result_daily_detail_v3 (
    `host` STRING,
    `fld` STRING,
    `is_host` STRING,
    `priority` STRING,
    `successful_parses` STRING,
    `failed_parses` STRING,
    `total_parses` STRING,
    `index_page_count` STRING,
    `index_page_count_rate` STRING,
    `short_content_count` STRING,
    `short_content_count_rate` STRING,
    `error_step_1_count` STRING,
    `error_step_2_count` STRING,
    `error_step_3_count` STRING,
    `error_info_list` STRING,
    `date_key` STRING
) STORED AS ORCFILE
''')

# 沙箱化运行日期（原 today=datetime.now()，ten_days_ago=today-timedelta(days=5)）
today_str = '20260513'
ten_days_ago_str = '20260508'
print(today_str, ten_days_ago_str)

# 读取 priority 信息
priority_info = spark.table("internal_platform_db.case10_sec_app_hy_top_500_sites_tag_v1_v3").select(
    "host", "fld", "priority"
)

# 读取数据，从今天往前 5 天的数据
df = spark.table("internal_platform_db.case10_ai_engine_classify_parse_result_daily_v3").filter(
    (col("date_key") >= ten_days_ago_str) & (col("date_key") <= today_str)
)

# 解析数据
parsed_data = df.select(
    "date_key",
    col("host").alias("source_host"),
    when(col("is_error") == 0, 1).otherwise(0).alias("success"),
    when(col("is_error") == 1, 1).otherwise(0).alias("failure"),
    "web_type",
    from_json(col("parse_html"), schema_of_json('{"content": "string"}')).getField("content").alias("content"),
    "error_step",
    col("fld").alias("source_fld"),
)

# 计算内容长度
parsed_data = parsed_data.withColumn("content_length", length(col("content")))

summary = parsed_data.groupBy("date_key", "source_host", "source_fld").agg(
    _sum("success").alias("successful_parses"),
    _sum("failure").alias("failed_parses"),
    count("*").alias("total_parses"),
)

index_page_ratio = parsed_data.groupBy("date_key", "source_host", "source_fld").agg(
    _sum(when(col("web_type") == "索引页", 1).otherwise(0)).alias("index_page_count")
)

content_length_ratio = parsed_data.filter(col("success") == 1).groupBy("date_key", "source_host", "source_fld").agg(
    _sum(when(col("content_length") < 50, 1).otherwise(0)).alias("short_content_count")
)

error_reasons = parsed_data.filter(col("failure") == 1).groupBy("date_key", "source_host", "source_fld").agg(
    _sum(when(col("error_step") == 1, 1).otherwise(0)).alias("error_step_1_count"),
    _sum(when(col("error_step") == 2, 1).otherwise(0)).alias("error_step_2_count"),
    _sum(when(col("error_step") == 3, 1).otherwise(0)).alias("error_step_3_count"),
    _sum(when(col("error_step").isNotNull(), 1).otherwise(0)).alias("total_error_steps"),
)

final_result = summary.join(index_page_ratio, ["date_key", "source_host", "source_fld"], "left") \
    .join(content_length_ratio, ["date_key", "source_host", "source_fld"], "left") \
    .join(error_reasons, ["date_key", "source_host", "source_fld"], "left")

final_result = final_result.withColumn(
    "index_page_count_rate",
    col("index_page_count") / col("total_parses"),
).withColumn(
    "short_content_count_rate",
    col("short_content_count") / col("successful_parses"),
)

final_result = final_result.withColumn("error_info_list", lit(""))
final_result = final_result.withColumn("is_host", lit(""))

# fld 存在 500tag 则扩散 host 组合 / host 存在则直接匹配 host
final_result = final_result.join(
    priority_info,
    (final_result.source_fld == priority_info.fld) |
    (final_result.source_host == priority_info.host),
    "left",
)

final_result = final_result.select(
    "date_key",
    F.col("source_host").alias("host"),
    F.col("source_fld").alias("fld"),
    "is_host",
    "priority",
    "successful_parses",
    "failed_parses",
    "total_parses",
    "index_page_count",
    "index_page_count_rate",
    "short_content_count",
    "short_content_count_rate",
    "error_step_1_count",
    "error_step_2_count",
    "error_step_3_count",
    "error_info_list",
)

final_result = final_result.dropDuplicates([
    "date_key", "host", "fld", "priority",
    "successful_parses", "total_parses", "index_page_count",
])

# INSERT OVERWRITE 显式列名（16 列）；output DDL 列序：host, fld, is_host, priority, ..., date_key
final_result.createOrReplaceTempView("case10_final_result_view")
insert_sql = """
INSERT OVERWRITE TABLE internal_platform_db.case10_ai_engine_classify_parse_result_daily_detail_v3 (
    host, fld, is_host, priority,
    successful_parses, failed_parses, total_parses,
    index_page_count, index_page_count_rate,
    short_content_count, short_content_count_rate,
    error_step_1_count, error_step_2_count, error_step_3_count,
    error_info_list, date_key
)
SELECT
    CAST(host AS STRING),
    CAST(fld AS STRING),
    CAST(is_host AS STRING),
    CAST(priority AS STRING),
    CAST(successful_parses AS STRING),
    CAST(failed_parses AS STRING),
    CAST(total_parses AS STRING),
    CAST(index_page_count AS STRING),
    CAST(index_page_count_rate AS STRING),
    CAST(short_content_count AS STRING),
    CAST(short_content_count_rate AS STRING),
    CAST(error_step_1_count AS STRING),
    CAST(error_step_2_count AS STRING),
    CAST(error_step_3_count AS STRING),
    CAST(error_info_list AS STRING),
    CAST(date_key AS STRING)
FROM case10_final_result_view
"""
print("insert_sql:")
print(insert_sql)
spark.sql(insert_sql)
print("saved")

spark.stop()