# SOA Dependency Analysis (PySpark)

## Prompt
I need you to generate a PySpark script that implements strong dependency relationship analysis for high-impact modules of use cases, based on the Payment Platform W SOA architecture (SOA Dependency Analysis).

**Business Background and Objective**: In the Payment Platform W SOA architecture, it is necessary to analyze the call relationships of high-quality network use cases (`qualitynetwork_case=1` or `importance=1`). Through upstream and downstream traversal of strong dependency (`dependent_intensity="strong"`) call chains, identify modules directly and indirectly affected by use cases, record impact chain evidence, and output a strong dependency module relationship table for high-impact use cases, for use in premium network evaluation and architecture governance.

**Input Tables**:
- `internal_platform_db.t_dm_callrelation_extended_info_hour_pyspark_209`
- `internal_platform_db.t_dm_usecase_and_asset_callrelation_hour_pyspark_209`
- `internal_platform_db.t_dwd_module_info_hour_pyspark_209`
- `internal_platform_db.t_app_callrelation_for_quality_analyse_hour_pyspark_209`

**Data Range and Filter Conditions**:
- Run date `ds` is fixed as `'p_2026051300'` (hourly partition)
- From `t_dm_usecase_and_asset_callrelation_hour`, filter use cases where `qualitynetwork_case=1` or `importance=1`
- Exclude modules not entering the premium network where `no_entry_premium_network_flag=1`
- From `t_dm_callrelation_extended_info_hour`, filter strong dependency call relationships where `is_involved_to_high_usecase=1` and `dependent_intensity="strong"`
- Filter out invalid call relationships `ossid:0` and `0:0`

**Processing Logic**:
1. From the use case and call relationship table, obtain directly associated modules (mapping of `enter_module_name` to `case_id`), mark `is_dir=1`, with evidence "direct association"
2. Build `case_id_relation_dict`: using `"callee_module_name***callee_interface_name"` as key, and the associated `case_id` set as value
3. Build `relation_dict` from the call relationship table: using `"callee_module_name***callee_interface_name"` as key, and the `"called_module_name***called_interface_name"` set as value
4. Use a DFS algorithm to traverse downstream call chains, add traversed modules to the results, mark `is_dir=0`, with evidence as the downstream analysis chain
5. For use case entries where `analyse_type="cgi"`, build a reverse call relationship dictionary `reverse_relation_dict`
6. Use a DFS algorithm to traverse upstream call chains, add traversed modules to the results, mark `is_dir=0`, with evidence as the upstream analysis chain
7. Merge the results of direct association, downstream analysis, and upstream analysis, and write to the target table

**Output Requirements**:
- Target table: `internal_platform_db.t_dwd_architecture_module_high_case_relation_strong_hour_pyspark_209`
- Write strategy: `INSERT OVERWRITE TABLE`
- Output columns: `databus_imp_date` (STRING, date), `case_id` (STRING, use case ID), `module_name` (STRING, module name), `is_dir` (BIGINT, 1=direct association, 0=indirect association), `evidence` (STRING, impact evidence chain)

## Reference Solution
#!/usr/bin/env python3
"""pyspark_018 ground truth: SOA dependency analysis"""
import json
from pyspark.sql import SparkSession

spark = SparkSession.builder \
    .appName('dataclaw_eval_gt_pyspark_018') \
    .enableHiveSupport() \
    .config('spark.sql.warehouse.dir', '/tmp/hive_warehouse') \
    .getOrCreate()

ds = 'p_2026051300'
today = '2026051300'

# Output table DDL
spark.sql('''
DROP TABLE IF EXISTS internal_platform_db.t_dwd_architecture_module_high_case_relation_strong_hour_pyspark_209
''')
spark.sql('''
CREATE TABLE IF NOT EXISTS internal_platform_db.t_dwd_architecture_module_high_case_relation_strong_hour_pyspark_209 (
    databus_imp_date STRING,
    case_id STRING,
    module_name STRING,
    is_dir BIGINT,
    evidence STRING
)
STORED AS ORC
''')

# Create temp views from input tables
spark.sql(f'''
CREATE OR REPLACE TEMP VIEW t_dm_callrelation_extended_info_hour AS
SELECT * FROM internal_platform_db.t_dm_callrelation_extended_info_hour_pyspark_209
WHERE ds = '{ds}'
''')

spark.sql(f'''
CREATE OR REPLACE TEMP VIEW t_dm_usecase_and_asset_callrelation_hour AS
SELECT * FROM internal_platform_db.t_dm_usecase_and_asset_callrelation_hour_pyspark_209
WHERE ds = '{ds}'
''')

spark.sql(f'''
CREATE OR REPLACE TEMP VIEW t_dwd_module_info_hour AS
SELECT * FROM internal_platform_db.t_dwd_module_info_hour_pyspark_209
WHERE ds = '{ds}'
''')

spark.sql(f'''
CREATE OR REPLACE TEMP VIEW t_app_callrelation_for_quality_analyse_hour AS
SELECT * FROM internal_platform_db.t_app_callrelation_for_quality_analyse_hour_pyspark_209
WHERE ds = '{ds}'
''')

print("step 1")
print("获取直接相关的模块")

T_SCHEMA = ('databus_imp_date', 'case_id', 'module_name', 'is_dir', 'evidence')

# Get direct relations - modules directly associated with cases
dir_sql = """
select distinct
    relation.enter_module_name,
    relation.case_id
from t_dm_usecase_and_asset_callrelation_hour relation
left join (
    select module_name
    from t_dwd_module_info_hour
    where no_entry_premium_network_flag = 1
) modules_info on relation.enter_module_name = modules_info.module_name
where (qualitynetwork_case = 1 or importance = 1)
    and (enter_module_name is not null and enter_module_name <> '')
    and modules_info.module_name is null
"""

service_datas = spark.sql(dir_sql).toPandas()

module_evidence = dict()

class Evidence:
    def __init__(self, module_name):
        self.module_name = module_name

    def set_case(self, case_id):
        self.case_id = case_id
        return self

    def set_evidence(self, evidence):
        self.evidence = evidence
        return self

    def to_string(self, comment):
        return "[用例ID:{}--{}--证据链路:{}]".format(self.case_id, comment, self.evidence)

result_list = []
for data in service_datas.values:
    enter_module_name = data[0]
    case_id = data[1]
    is_dir = 1

    if enter_module_name not in module_evidence:
        module_evidence[enter_module_name] = Evidence(enter_module_name).set_case(case_id).set_evidence("直接关联")
    result_list.append((today, case_id, enter_module_name, is_dir, module_evidence[enter_module_name].to_string("直接关联")))

print("直接影响的个数")
origin_length = len(result_list)
print(origin_length)

print("step 2")
print("获取用例和调用关系入口模块-接口映射")

case_id_relation_dict_sql = """
select
    enter_module_name, case_id, callee_module_name, callee_interface_name
from t_dm_usecase_and_asset_callrelation_hour
where (qualitynetwork_case = 1 or importance = 1)
    and (enter_module_name is not null and enter_module_name <> '')
    and (callee_module_name is not null and callee_module_name <> '')
    and (callee_interface_name is not null and callee_interface_name <> '')
    and (called_module_name is not null and called_module_name <> '')
    and (called_interface_name is not null and called_interface_name <> '')
    and enter_module_name = callee_module_name
"""

case_id_relation_dict = dict()
service_datas = spark.sql(case_id_relation_dict_sql).toPandas()
for data in service_datas.values:
    enter_module_name = data[0]
    case_id = data[1]
    callee_module_name = data[2]
    callee_interface_name = data[3]

    relation = "{}***{}".format(callee_module_name, callee_interface_name)
    if relation in case_id_relation_dict:
        case_id_relation_dict[relation].add(case_id)
    else:
        case_id_relation_dict[relation] = set()
        case_id_relation_dict[relation].add(case_id)

print("step 3")
print("获取剔除了不进入精品网模块的强依赖调用关系")

relation_dict_sql = """
select
    relation.callee_module_name,
    relation.callee_interface_name,
    relation.called_module_name,
    relation.called_interface_name
from
    (select
        callee_module_name, callee_interface_name, called_module_name, called_interface_name
    from t_dm_callrelation_extended_info_hour
    where is_involved_to_high_usecase = 1
        and dependent_intensity = 'strong'
    ) relation
left join (
    select module_name
    from t_dwd_module_info_hour
    where no_entry_premium_network_flag = 1
) modules_callee on relation.callee_module_name = modules_callee.module_name
left join (
    select module_name
    from t_dwd_module_info_hour
    where no_entry_premium_network_flag = 1
) modules_called on relation.called_module_name = modules_called.module_name
where modules_callee.module_name is null
    and modules_called.module_name is null
"""

relation_dict = dict()
service_datas = spark.sql(relation_dict_sql).toPandas()
for data in service_datas.values:
    callee_module_name = data[0]
    callee_interface_name = data[1]
    called_module_name = data[2]
    called_interface_name = data[3]

    relation_callee = "{}***{}".format(callee_module_name, callee_interface_name)
    relation_called = "{}***{}".format(called_module_name, called_interface_name)
    if relation_callee in relation_dict:
        relation_dict[relation_callee].add(relation_called)
    else:
        relation_dict[relation_callee] = set()
        relation_dict[relation_callee].add(relation_called)


def dfs(relation_dict, called_info, record_list, chain_list, space, case_id_now):
    if called_info not in relation_dict:
        return
    called_infos = relation_dict[called_info]
    for called_info_item in called_infos:
        if called_info_item in record_list:
            continue
        record_list.add(called_info_item)

        chain_list.append(called_info_item)

        module_name_now = called_info_item.split("***")[0]
        if module_name_now != "" and module_name_now not in module_evidence:
            module_evidence[module_name_now] = Evidence(module_name_now).set_evidence(space.join(chain_list)).set_case(case_id_now)

        dfs(relation_dict, called_info_item, record_list, chain_list, space, case_id_now)

        chain_list.pop()


def dfs_fist(relation_dict, key, reverse, case_id_now):
    space = "<-" if reverse else "->"
    record_list = set()
    chain_list = list()

    if key not in relation_dict:
        return None

    chain_list.append(key)

    module_name_now = key.split("***")[0]
    if module_name_now != "" and module_name_now not in module_evidence:
        module_evidence[module_name_now] = Evidence(module_name_now).set_evidence(space.join(chain_list)).set_case(case_id_now)

    called_infos = relation_dict[key]
    for called_info in called_infos:
        if called_info in record_list:
            continue
        record_list.add(called_info)

        module_name_now = called_info.split("***")[0]
        chain_list.append(called_info)
        if module_name_now != "" and module_name_now not in module_evidence:
            module_evidence[module_name_now] = Evidence(module_name_now).set_evidence(space.join(chain_list)).set_case(case_id_now)

        dfs(relation_dict, called_info, record_list, chain_list, space, case_id_now)
        chain_list.pop()
    return record_list


# Downstream analysis
print("step 4")
print("开始下游分析")

is_dir = 0
module_evidence = dict()
for key in case_id_relation_dict:
    case_id_now = list(case_id_relation_dict[key])[0]
    modules_list = dfs_fist(relation_dict, key, 0, case_id_now)
    if modules_list is None:
        continue
    cases = case_id_relation_dict[key]
    for case_id in cases:
        for module in modules_list:
            module_name = module.split("***")[0]
            if module_name == "":
                continue
            result_list.append((today, case_id, module_name, is_dir, module_evidence[module_name].to_string("下游分析")))

print("间接影响的个数")
print(len(result_list) - origin_length)

# Upstream analysis (CGI)
print("step 5")
print("获取cgi用例和调用关系入口模块-接口映射")

case_id_relation_dict_sql = """
select
    enter_module_name, case_id, callee_module_name, callee_interface_name
from t_dm_usecase_and_asset_callrelation_hour
where (qualitynetwork_case = 1 or importance = 1)
    and (enter_module_name is not null and enter_module_name <> '')
    and (callee_module_name is not null and callee_module_name <> '')
    and (callee_interface_name is not null and callee_interface_name <> '')
    and (called_module_name is not null and called_module_name <> '')
    and (called_interface_name is not null and called_interface_name <> '')
    and enter_module_name = callee_module_name
    and analyse_type = 'cgi'
"""

case_id_relation_dict = dict()
service_datas = spark.sql(case_id_relation_dict_sql).toPandas()
for data in service_datas.values:
    enter_module_name = data[0]
    case_id = data[1]
    callee_module_name = data[2]
    callee_interface_name = data[3]

    relation = "{}***{}".format(callee_module_name, callee_interface_name)
    if relation in case_id_relation_dict:
        case_id_relation_dict[relation].add(case_id)
    else:
        case_id_relation_dict[relation] = set()
        case_id_relation_dict[relation].add(case_id)

# Build reverse relation dict
all_relation_dict_sql = """
select distinct
    callee_module_name,
    callee_interface_name,
    called_module_name,
    called_interface_name
from t_app_callrelation_for_quality_analyse_hour
where callee_module_name <> 'ossid:0'
    and called_module_name <> 'ossid:0'
    and callee_module_name <> '0:0'
    and called_module_name <> '0:0'
"""

reverse_relation_dict = dict()
service_datas = spark.sql(all_relation_dict_sql).toPandas()
for data in service_datas.values:
    callee_module_name = data[0]
    callee_interface_name = data[1]
    called_module_name = data[2]
    called_interface_name = data[3]

    relation_callee = "{}***{}".format(callee_module_name, callee_interface_name)
    relation_called = "{}***{}".format(called_module_name, called_interface_name)
    if relation_called in reverse_relation_dict:
        reverse_relation_dict[relation_called].add(relation_callee)
    else:
        reverse_relation_dict[relation_called] = set()
        reverse_relation_dict[relation_called].add(relation_callee)

module_evidence = dict()
is_dir = 0
reverse_result_list = list()
for key in case_id_relation_dict:
    case_id_now = list(case_id_relation_dict[key])[0]
    modules_list = dfs_fist(reverse_relation_dict, key, 1, case_id_now)
    if modules_list is None:
        continue
    cases = case_id_relation_dict[key]
    for case_id in cases:
        for module in modules_list:
            module_name = module.split("***")[0]
            if module_name == "":
                continue
            reverse_result_list.append((today, case_id, module_name, is_dir, module_evidence[module_name].to_string("上游分析")))

print("反向影响的个数")
print(len(reverse_result_list))
result_list = result_list + reverse_result_list

# Write output
t_df = spark.createDataFrame(result_list, T_SCHEMA)
t_df.write.mode('overwrite').format('orc').saveAsTable(
    'internal_platform_db.t_dwd_architecture_module_high_case_relation_strong_hour_pyspark_209'
)

print('Ground truth computation complete')
spark.stop()