# AI Search Parse Quality Daily Detail (case9 version) (PySpark)

## Prompt
Business Background: Compute daily details of AI search web page parse quality by site + directory dimensions. Take data from the past few days, aggregate success rate, index page ratio, short content ratio, and failure step counts by the (date_key, host, fld) dimension, and join with the Top500 site priority labels.

**Input Tables**:
- `internal_platform_db.ai_engine_classify_parse_result_daily_copilot_v3`
- `internal_platform_db.sec_app_hy_top_500_sites_tag_v1_copilot_v3`

  Data Range:
  - Fact table filter: `date_key >= '20260508' AND date_key <= '20260513'` (hardcoded, today='20260513', looking back 5 days)
  - Dimension table participates in full

  Processing Logic:

  1. Field preparation: Select the following columns from the fact table:

  - `date_key` retained as-is

  - `host` → renamed to `source_host`

  - `fld` → renamed to `source_fld`

  - `when(col("is_error") == 0, 1).otherwise(0)` → `success` (note: `is_error` is STRING; comparing with `== 0` triggers implicit type conversion in Spark)

  - `when(col("is_error") == 1, 1).otherwise(0)` → `failure`

  - `web_type` retained as-is

  - `from_json(col("parse_html"), schema_of_json('{"content": "string"}')).getField("content")` → `content`

  - `error_step` retained as-is

  - Then `withColumn("content_length", length(col("content")))`
  2. Four aggregations (all grouped by `date_key`, `source_host`, `source_fld`):


  - summary: `sum(success)` → `successful_parses`, `sum(failure)` → `failed_parses`, `count(*)` → `total_parses`

  - index_page_ratio: `sum(when(web_type=='索引页', 1).otherwise(0))` → `index_page_count`


  - content_length_ratio: first `filter(success==1)`, then `sum(when(content_length<50, 1).otherwise(0))` → `short_content_count`

  - error_reasons: first `filter(failure==1)`, then separately `sum(when(error_step==1/2/3, 1).otherwise(0))` → `error_step_1/2/3_count`; additionally `sum(when(error_step.isNotNull(), 1).otherwise(0))` → `total_error_steps` (this field is not used subsequently)
  3. Merge: Left join the four aggregation results on `[date_key, source_host, source_fld]` to form `final_result`
  4. Compute ratios:

  - `index_page_count_rate = index_page_count / total_parses`

  - `short_content_count_rate = short_content_count / successful_parses` (no divide-by-zero protection; produces null when `successful_parses=0`)
  5. Hardcoded empty fields:


  - `withColumn("error_info_list", lit(""))` — no `collect_list`, directly fixed as empty string

  - `withColumn("is_host", lit(""))` — not taken from the dimension table, directly fixed as empty string

  6. Join dimension table (key: OR condition):
  `final_result.join(`

`priority_info`, — contains only `host`, `fld`, `priority` columns

`(final_result.source_fld == priority_info.fld) OR`

`(final_result.source_host == priority_info.host),`

`"left"`

  `)`
  6. Note: The join condition is OR (matching on either fld or host), not AND. This may cause one main table record to match multiple rows in the dimension table.
  7. Final select (16 columns, in this order):

  - `date_key`, `source_host`→`host`, `source_fld`→`fld`, `is_host` (from step 5's `lit("")`), `priority` (from dimension table join), `successful_parses`, `failed_parses`, `total_parses`, `index_page_count`, `index_page_count_rate`, `short_content_count`,
  `short_content_count_rate`, `error_step_1_count`, `error_step_2_count`, `error_step_3_count`, `error_info_list` (from step 5's `lit("")`)
  8. Deduplication: `dropDuplicates(["date_key", "host", "fld", "priority", "successful_parses", "total_parses", "index_page_count"])`

  - Note: Because the OR join may produce multiple rows (one main table row matching multiple dimension table rows), deduplication is used to converge the inflated rows
  9. Write: `final_result.write.mode("overwrite").insertInto(target table)`

  Output table: `internal_platform_db.ai_engine_classify_parse_result_daily_detail_copilot_v3`

## Reference Solution
# data-mocker rewrite-gt
# source: 离线计算pyspark例子_v7.xlsx 工作表1 row=10 col=3
# row_idx: 9
# intent_short: 帮我做个AI搜索引擎解析质量的每日明细统计，按站点和目录维度算一下成功率、索引页占比、短内容占比，还有各环节的失败分布。另外需要把Top500站点的优先级标签关联上
# v3 tables: internal_platform_db.ai_engine_classify_parse_result_daily_detail_copilot_v3, internal_platform_db.ai_engine_classify_parse_result_daily_copilot_v3, internal_platform_db.sec_app_hy_top_500_sites_tag_v1_copilot_v3
# extra fqns scanned from GT (not in v7_rows.tables): internal_platform_db.ai_engine_classify_parse_result_daily_detail_copilot
# rewrite rule: <fqn> -> <fqn>_v3 (covers bare / `fqn` / db.`tbl`)
# ---
#!/usr/bin/env python
# coding: utf-8

import os
from pyspark.sql import SparkSession
from pyspark.sql.functions import col, count, when, length, sum as _sum, from_json, schema_of_json
from pyspark.sql.types import StringType
from datetime import datetime, timedelta
from pyspark.sql.functions import lit
from pyspark.sql import functions as F
# 创建SparkSession
spark = (
   
   SparkSession.builder.enableHiveSupport()
       
   .config("spark.driver.memory", "6g")
        .config("spark.executor.cores", 6)
       
   .config("spark.executor.memory", "12g")
        .getOrCreate()
)

# 确保输出表以正确 schema 存在
spark.sql("DROP TABLE IF EXISTS internal_platform_db.ai_engine_classify_parse_result_daily_detail_copilot_v3")
spark.sql('''
CREATE TABLE internal_platform_db.ai_engine_classify_parse_result_daily_detail_copilot_v3 (
`date_key` STRING COMMENT '-',
`host` STRING COMMENT '-',
`fld` STRING COMMENT '-',
`is_host` STRING COMMENT '-',
`priority` STRING COMMENT '-',
`successful_parses` BIGINT COMMENT '-',
`failed_parses` BIGINT COMMENT '-',
`total_parses` BIGINT COMMENT '-',
`index_page_count` BIGINT COMMENT '-',
`index_page_count_rate` DOUBLE COMMENT '-',
`short_content_count` BIGINT COMMENT '-',
`short_content_count_rate` DOUBLE COMMENT '-',
`error_step_1_count` BIGINT COMMENT '-',
`error_step_2_count` BIGINT COMMENT '-',
`error_step_3_count` BIGINT COMMENT '-',
`error_info_list` STRING COMMENT '-'
) STORED AS ORCFILE
''')

 

# # 获取今天的日期，并计算10天前的日期
# today = datetime.now()
# ten_days_ago = today - timedelta(days=5)

# # 将日期格式化为 YYYYMMDD
# today_str = today.strftime('%Y%m%d')
# ten_days_ago_str = ten_days_ago.strftime('%Y%m%d')
today_str = '20260513'
ten_days_ago_str = '20260508'
print(today_str, ten_days_ago_str)

# 读取 priority 信息
priority_info = spark.table("internal_platform_db.sec_app_hy_top_500_sites_tag_v1_copilot_v3").select(
    "host", "fld", 
   "priority"
)

# 读取数据，从今天往前10天的数据
df = spark.table("internal_platform_db.ai_engine_classify_parse_result_daily_copilot_v3").filter(
    (col("date_key") >= ten_days_ago_str) & (col("date_key") <= today_str)
)

# 解析数据
parsed_data = df.select(
    "date_key",
   
   col("host").alias("source_host") ,
    when(col("is_error") == 0, 1).otherwise(0).alias("success"),
    when(col("is_error") == 1, 1).otherwise(0).alias("failure"),
    "web_type",
    # 从 parse_html 中提取 content 字段
   
   from_json(col("parse_html"), schema_of_json('{"content": "string"}')).getField("content").alias("content"),
    "error_step",
    col("fld").alias("source_fld")  # 使用别名避免歧义
)

# 计算内容长度
parsed_data = parsed_data.withColumn("content_length", length(col("content")))

# 在 summary_copilot 中保留 source_fld 字段
summary = parsed_data.groupBy("date_key", "source_host", "source_fld").agg(  # 使用别名
    _sum("success").alias("successful_parses"),
   
   _sum("failure").alias("failed_parses"),
    count("*").alias("total_parses")
)

# 计算索引页比例
index_page_ratio = parsed_data.groupBy("date_key","source_host", "source_fld").agg(  # 使用别名
   
   _sum(when(col("web_type") == "索引页", 1).otherwise(0)).alias("index_page_count")
)

# 计算内容长度比例
content_length_ratio = parsed_data.filter(col("success") == 1).groupBy("date_key","source_host", "source_fld").agg(  # 使用别名
   
   _sum(when(col("content_length") < 50, 1).otherwise(0)).alias("short_content_count")
)

# 计算错误原因
error_reasons = parsed_data.filter(col("failure") == 1).groupBy("date_key", "source_host", "source_fld").agg(  # 使用别名
   
   _sum(when(col("error_step") == 1, 1).otherwise(0)).alias("error_step_1_count"),
    _sum(when(col("error_step") == 2, 1).otherwise(0)).alias("error_step_2_count"),
   
   _sum(when(col("error_step") == 3, 1).otherwise(0)).alias("error_step_3_count"),
   
   _sum(when(col("error_step").isNotNull(), 1).otherwise(0)).alias("total_error_steps")
)

# 最终合并
final_result = summary.join(index_page_ratio, ["date_key", "source_host", "source_fld"], "left") \
    .join(content_length_ratio, ["date_key","source_host", "source_fld"], "left") \
    .join(error_reasons, ["date_key","source_host", "source_fld"], "left")

# 计算比例
final_result = final_result.withColumn(
   
   "index_page_count_rate",
    col("index_page_count") / col("total_parses")
).withColumn(
   
   "short_content_count_rate",
   
   col("short_content_count") / col("successful_parses")
)


# 添加 error_info_list 字段
final_result = final_result.withColumn("error_info_list", lit(""))
final_result = final_result.withColumn("is_host", lit(""))



####fld存在500tag则扩散对应的ccdata的host组合---host存在则直接匹配host
 

final_result = final_result.join(
    priority_info,
    (final_result.source_fld == priority_info.fld) |
    (final_result.source_host == priority_info.host),
    "left"
)
# 选择所需的列
final_result = final_result.select(
    "date_key",
   
   F.col("source_host").alias("host"),    
   
   F.col("source_fld").alias("fld"),  # 使用 source_fld
    "is_host",
    "priority",
    "successful_parses",
    "failed_parses",
    "total_parses",
    "index_page_count",
   
   "index_page_count_rate",
   
   "short_content_count",
   
   "short_content_count_rate",
    "error_step_1_count",
   
   "error_step_2_count",
   
   "error_step_3_count",
    "error_info_list"
)

final_result = final_result.dropDuplicates(["date_key", "host", "fld","priority","successful_parses","total_parses","index_page_count"])
# 将结果写入目标表
final_result.write.mode("overwrite").insertInto("internal_platform_db.ai_engine_classify_parse_result_daily_detail_copilot_v3")