# Phone Number Encrypted Prefix Quality Score Statistics (PySpark)

## Prompt
I need you to generate a PySpark script that computes a phone number encrypted prefix quality score statistics table.

**Business Background and Objective**: The phone number quality score table stores the quality score (`maochi` field) for each phone number encrypted string. Operations needs to perform aggregation analysis on the first 7 characters of the encrypted prefix for each phone number, computing the average quality score, malicious ratio, and whitelist ratio under that prefix. The aggregation results are then backfilled onto each detail record to produce a wide table with prefix-level statistical dimensions.

**Input Table**:
- `internal_platform_db.t_dwd_phone_enc_quality_score_df_pyspark_216`

**Data Range and Filter Conditions**:
- Run date `ds` is fixed as `'20260513'`
- Filter out invalid records where `maochi = -9`

**Computation Rules**:
1. Use `substr(phone_enc, 1, 7)` to extract the phone number encrypted prefix `phone_enc_p7`
2. Group by `phone_enc_p7` and aggregate:
   - `prefix_1_7_score = floor(avg(maochi) * 10000)`: prefix average quality score (scaled by 10000)
   - `prefix_1_7_evil_score = floor(count(if(maochi >= -4 and maochi <= -2, 1, NULL)) * 10000 / count(1))`: prefix malicious ratio (in parts per ten thousand)
   - `prefix_1_7_white_score = floor(count(if(maochi >= 2 and maochi <= 4, 1, NULL)) * 10000 / count(1))`: prefix whitelist ratio (in parts per ten thousand)
3. LEFT JOIN to associate the prefix aggregation results back to the original detail records

**Output Requirements**:
- Target table: `internal_platform_db.t_dwd_phone_enc_prefix_quality_score_df_pyspark_216`
- Partition field: `p_date='20260513'`
- Output columns: `ds`, `phone_enc`, `quality_score`, `prefix_1_7_score`, `prefix_1_7_evil_score`, `prefix_1_7_white_score`
- Write strategy: First create the table (ORC storage), then use `INSERT OVERWRITE TABLE PARTITION(p_date=...)` to write

## Reference Solution
#!/usr/bin/env python
# -*- coding: utf-8 -*-
"""
pyspark_011: Phone enc prefix quality score
- Read t_dwd_phone_enc_quality_score_df, filter maochi != -9
- Use substr() to extract phone_enc prefix (first 7 chars) instead of Java UDF IdSubstr
- CTE aggregation per prefix: avg quality_score, evil ratio, white ratio
- LEFT JOIN back to original, output with ds='20260513'
"""

from pyspark.sql import SparkSession

spark = (
    SparkSession.builder
    .appName('pyspark_011_phone_enc_prefix_score')
    .enableHiveSupport()
    .getOrCreate()
)

TODAY = '20260513'

query = """
    with t_phone_enc_score as (
        select
            phone_enc,
            maochi,
            substr(phone_enc, 1, 7) as phone_enc_p7
        from internal_platform_db.t_dwd_phone_enc_quality_score_df_pyspark_216
        where ds = {today} and maochi != -9
    ), t_phone_enc_p7_score as (
        select
            phone_enc_p7,
            floor(avg(maochi) * 10000) as prefix_1_7_score,
            floor(count(if(maochi >= -4 and maochi <= -2, 1, NULL)) * 10000 / count(1)) as prefix_1_7_evil_score,
            floor(count(if(maochi >= 2 and maochi <= 4, 1, NULL)) * 10000 / count(1)) as prefix_1_7_white_score
        from
            t_phone_enc_score
        group by
            phone_enc_p7
    ), t_merge as (
        select
            phone_enc,
            maochi as quality_score,
            prefix_1_7_score,
            prefix_1_7_evil_score,
            prefix_1_7_white_score
        from
            t_phone_enc_score t1
        left join
            t_phone_enc_p7_score t2
        on
            t1.phone_enc_p7 = t2.phone_enc_p7
    )
    select {today} as ds, phone_enc, quality_score, prefix_1_7_score, prefix_1_7_evil_score, prefix_1_7_white_score from t_merge
""".format(today=TODAY)
print(f"query_sql: {query}")
df = spark.sql(query)
df.createOrReplaceTempView("tmp_result")

# Output table DDL
spark.sql('DROP TABLE IF EXISTS internal_platform_db.t_dwd_phone_enc_prefix_quality_score_df_pyspark_216')
spark.sql('''
CREATE TABLE internal_platform_db.t_dwd_phone_enc_prefix_quality_score_df_pyspark_216 (
    `ds` STRING,
    `phone_enc` STRING,
    `quality_score` INT,
    `prefix_1_7_score` BIGINT,
    `prefix_1_7_evil_score` BIGINT,
    `prefix_1_7_white_score` BIGINT
) PARTITIONED BY (`p_date` STRING)
STORED AS ORC
''')

# Write output via INSERT OVERWRITE with explicit partition
spark.sql("""
INSERT OVERWRITE TABLE internal_platform_db.t_dwd_phone_enc_prefix_quality_score_df_pyspark_216
PARTITION (p_date='{0}')
SELECT ds, phone_enc, quality_score, prefix_1_7_score, prefix_1_7_evil_score, prefix_1_7_white_score
FROM tmp_result
""".format(TODAY))

print("pyspark_011 done")

spark.stop()