# 过去半年在线过的实验计划信息查询 (PySpark)

## Prompt
我需要你生成一段 PySpark 代码，查询过去半年内上线过的实验计划信息，取每个实验分组的最新一条记录。

**业务背景与目标**：实验平台按小时维度存储了实验计划的上线记录（fdate, exp_group_id, trigger_list）。运营需要从过去 180 天（半年）的历史数据中，按实验分组 exp_group_id 维度取最新的一条记录，输出到结果表中，用于后续的实验分析。

**输入表**：
- `internal_platform_db.exp_hours_input_pyspark_102`

**数据范围与过滤条件**：
- 运行日期 cur_date 固定为 '20260513'
- 历史天数 history_days 固定为 180
- 读取的分区范围：从 p_2026051300 往前推 180 * 24 = 4320 个小时的所有分区
- 分区格式：p_YYYYMMDDHH

**计算规则**：
1. 生成历史分区列表：使用 Python time 模块，从 cur_date 当天 00:00:00 开始，依次减去 1 小时，共生成 history_days * 24 个分区名（如 p_2026051300, p_2026051223, p_2026051222, ...）
2. 使用 WHERE p_hour IN (...) 过滤读取所有历史分区数据
3. 使用窗口函数 row_number() OVER (PARTITION BY exp_group_id ORDER BY fdate DESC) 计算 date_rank_no
4. 过滤 date_rank_no = 1，取每个实验分组的最新记录
5. 添加 fdate = lit(cur_date) 列

**输出要求**：
- 目标表：`internal_platform_db.exp_hours_output_pyspark_102`
- 写入策略：先建表（ORC 存储，5 列），再 INSERT OVERWRITE TABLE 写入
- 输出列：exp_date, exp_group_id, trigger_list, date_rank_no, fdate

## Reference Solution
#!/usr/bin/env python
# -*- coding: utf-8 -*-
"""
pyspark_014: Experiment plans past half year query
- 1 input table: exp_hours_input_pyspark_102 (fdate, exp_group_id, trigger_list, partitioned by p_hour)
- Read all partitions from p_2026051300 back to p_2026051300 - 24*180 hours (half year)
- Hardcode: cur_date='20260513', history_days=180
- Generate list of hour partitions to read
- Window function: rank() over (partition by exp_group_id order by fdate desc), keep rank=1
- Add fdate = lit(cur_date)
- Output table: exp_hours_output_pyspark_102 via INSERT OVERWRITE TABLE
"""

import time
from pyspark.sql import SparkSession
from pyspark.sql import functions as F
from pyspark.sql.window import Window

cur_date = '20260513'
history_days = 180

def get_history_hours(cur_date, history_days):
    cur_hour = cur_date + " 00:00:00"
    cur_ts = int(time.mktime(time.strptime(cur_hour, "%Y%m%d %H:%M:%S")))
    history_hours = []
    history_hour_num = history_days * 24
    for i in range(history_hour_num):
        history_hour = time.strftime("%Y%m%d%H", time.localtime(cur_ts - i * 3600))
        history_hours.append("p_" + history_hour)
    return history_hours

partition_list = get_history_hours(cur_date, history_days)
print("cur_date: " + cur_date)
print("history_days: " + str(history_days))
print("partition_list length: " + str(len(partition_list)))

spark = SparkSession.builder \
    .appName('pyspark_014_exp_plans') \
    .enableHiveSupport() \
    .getOrCreate()

# Read all history partitions using a single SQL with OR conditions on partition values
# Build partition filter string
partition_filters = ", ".join(["'%s'" % p for p in partition_list])
spark.sql("""
    SELECT fdate, exp_group_id, trigger_list
    FROM internal_platform_db.exp_hours_input_pyspark_102
    WHERE p_hour IN ({0})
""".format(partition_filters)).createOrReplaceTempView("t_exp_hours")

sql = '''
    select
        fdate as exp_date,
        exp_group_id,
        trigger_list
    from
        t_exp_hours
'''
print("sql: " + sql)
exp_hours_df = spark.sql(sql)
exp_hours_df.printSchema()

# Window function: rank by exp_date desc per exp_group_id
exp_partition_window = Window.partitionBy(["exp_group_id"]).orderBy(exp_hours_df["exp_date"].desc())
exp_hours_df = exp_hours_df.withColumn("date_rank_no", F.row_number().over(exp_partition_window))
exp_hours_df = exp_hours_df.filter(exp_hours_df.date_rank_no == 1)
exp_hours_df = exp_hours_df.withColumn("fdate", F.lit(cur_date))

# Output table DDL
spark.sql('DROP TABLE IF EXISTS internal_platform_db.exp_hours_output_pyspark_102')
spark.sql('''
CREATE TABLE internal_platform_db.exp_hours_output_pyspark_102 (
    `exp_date` STRING,
    `exp_group_id` STRING,
    `trigger_list` STRING,
    `date_rank_no` INT,
    `fdate` STRING
)
STORED AS ORC
''')

# Write output via INSERT OVERWRITE TABLE
exp_hours_df.createOrReplaceTempView("tmp_result")
spark.sql("""
INSERT OVERWRITE TABLE internal_platform_db.exp_hours_output_pyspark_102
SELECT exp_date, exp_group_id, trigger_list, date_rank_no, fdate
FROM tmp_result
""")

print("pyspark_014 done")
spark.stop()