Domain: Business Operations & Analytics

You are a Senior Data Scientist at **TrustGuard Analytics**, a social media threat-intelligence firm. Your platform's data pipeline has suffered a catastrophic schema migration failure, producing a severely corrupted extract of social media post data. Columns have inconsistent formatting — follower counts are recorded with mixed notations, region names contain misspellings and irregular casing, boolean and categorical fields use incompatible representations, and duplicate records from a double-ingestion event have leaked into the dataset.

Management has commissioned an urgent **Content Engagement Forensics** audit after detecting a spike in coordinated inauthentic behavior across multiple regions. You must rigorously clean the dataset, compute a sequence of risk metrics, and deliver a final **Composite Threat Assessment Score (CTAS)**.

### Dataset

The file `social_media_forensics.csv` is located at `/tmp/files/`. 

### Data Sanitization 

The raw data contains multiple quality issues that must be resolved before any analysis. The data pipeline corruption introduced the following known problems:

- **Follower counts** are recorded inconsistently 
- **Region names** contain typos, inconsistent casing, extra whitespace, and punctuation artifacts. There are exactly 6 canonical regions: North America, Europe, Asia Pacific, Latin America, Middle East, and Africa. 
- The **`is_verified`** field uses mixed boolean representations (Yes/No, TRUE/FALSE, 1/0, Y/N, etc.) and must be standardized to boolean.
- The **`content_flag`** field uses a mix of numeric (0/1) and text ("clean"/"flagged") representations with various casings, plus some missing entries. 
- **Impressions** contain non-numeric entries ("N/A", empty strings, negative values). 
- **Likes** may contain negative values (data error);
- **`account_age_days`** may contain non-positive values; 
- The **`toxicity_score`** field has out-of-range values (some > 1.0, some negative). 
- The **`sentiment_confidence`** field contains percentage signs embedded in some entries and out-of-range values.
- **Duplicate posts** exist from a double-ingestion event — records sharing the same `user_id` AND `content` are duplicates.


### Analysis Tasks

After cleaning, perform the following sequential analyses. Each subsequent task builds on the cleaned dataset and previously computed metrics.

---

**Task 1 – Weighted Engagement Rate (WER)**

Compute a per-post Weighted Engagement Rate using the industry-standard amplification model: shares and retweets carry **double** the weight of a standard interaction (like), while replies carry **half** the weight. Normalize by impressions and express as a percentage.

---

**Task 2 – Account Credibility Index (ACI)**

Compute a per-post Account Credibility Index defined as:

$$ACI = \log_2(\text{followers}) \times \frac{\text{account\_age\_days}}{365} \times \frac{\text{sentiment\_confidence}}{100}$$

For records where `sentiment_confidence` is missing after cleaning, impute with the **global median** of available (non-missing) `sentiment_confidence` values before computing ACI.

---

**Task 3 – Regional Verified-Account Engagement Profile**

Restrict the analysis to **verified accounts only**. For each of the 6 regions, compute:
- **Median WER**
- **75th percentile WER (Q3)**

Identify the region with the **highest median WER** among verified accounts and report its name and value.

---

**Task 4 – Anomalous Post Detection**

Using the **entire cleaned dataset** (not just verified accounts), compute the global Q1, Q3, and IQR of WER. A post is classified as **"anomalous"** if it satisfies **ALL** of the following five conditions simultaneously:

1. WER exceeds the upper fence (Q3 + 1.5 × IQR)
2. The post is flagged (`content_flag` = 1)
3. `toxicity_score` is strictly greater than 0.65
4. The account is younger than one year (`account_age_days` < 365)
5. The post's ACI is strictly below the global median ACI

Report the **total count** of anomalous posts across all regions.

---

**Task 5 – Regional Threat Index (RTI)**

For each of the 6 canonical regions, compute:

$$RTI = \frac{\text{anomalous posts in region}}{\text{total posts in region}} \times \overline{\text{toxicity}}_{\text{flagged}} \times 1000$$

where $\overline{\text{toxicity}}_{\text{flagged}}$ is the **mean toxicity_score** of all posts with `content_flag = 1` within that specific region.

Sort regions by RTI in descending order.

---

**Task 6 – Platform Risk Score (PRS)**

For each of the 4 platforms, compute:

$$PRS = \overline{WER}_{\text{flagged}} \times CV_{WER} \times \frac{\%_{\text{flagged}}}{100}$$

where:
- $\overline{WER}_{\text{flagged}}$ = mean WER of flagged posts (`content_flag = 1`) on that platform
- $CV_{WER}$ = coefficient of variation of WER across **all** posts on that platform (sample standard deviation ÷ mean)
- $\%_{\text{flagged}}$ = percentage of posts on that platform that are flagged

Sort platforms by PRS in descending order.

---

**Task 7 – Composite Threat Assessment Score (CTAS)**

Combine the regional and platform risk signals:

$$CTAS = \sum_{i=1}^{3} RTI_{\text{top-}i} + \sum_{j=1}^{2} PRS_{\text{top-}j}$$

That is, sum the RTI values of the **top 3 regions** (by RTI descending) and the PRS values of the **top 2 platforms** (by PRS descending).

---

### Output Requirements

Report the following values in a single formatted summary. Do not include code, only the final results:

1. **Total number of records** after all cleaning steps
2. **Region with the highest median WER** among verified accounts, and its median WER value (rounded to 4 decimal places)
3. **Total count of anomalous posts**
4. **RTI for each of the 6 regions**, sorted descending, each rounded to 4 decimal places
5. **PRS for each of the 4 platforms**, sorted descending, each rounded to 4 decimal places
6. **Final CTAS value**, rounded to **2 decimal places**


Grading rubric:
criterion: Does the solution report the total number of cleaned records as exactly 1137?
weight: 10
criterion: Does the solution correctly identify 'Latin America' as the region with the median WER value rounding to 11.4226?
weight: 10
criterion: Does the solution report the total count of anomalous posts as  9?
weight: 10
criterion: Does the solution report the RTI for Africa  with a value of 14.2082?
weight: 10
criterion: Does the solution report the RTI for Asia Pacific as exactly 0.0000, reflecting that zero anomalous posts were detected in that region?
weight: 10
criterion: Does the solution correctly compute and report the PRS for iOS as the highest among all platforms, with a value that rounds to 31.76?
weight: 10
criterion: Does the solution identify the top 3 RTI regions in the correct descending order as Africa, Latin America, and Middle East?
weight: 10
criterion: Does the solution report the RTI for Middle East with a value that rounds to 7.8874?
weight: 10
criterion: Does the solution report the final Composite Threat Assessment Score (CTAS) as 84.53?",
weight: 20

Golden answer:
EXPERT SOLUTION
===============

============================================================
STEP 1: DATA SANITIZATION
============================================================

Raw dataset: 2060 rows × 17 columns

1a. Parse `followers` column:
    - Values like "1.5K" → 1500, "2.3M" → 2300000
    - Remove commas (e.g., "15,000" → 15000)
    - "N/A" and empty strings → NaN
    - Negative values remain as-is (removed in next step)
    - Result: 94 NaN values after parsing

1b. Standardize `region` names:
    - Map all variants to 6 canonical regions:
      "north america", "NORTH AMERICA", "Noth America", "North  America", "N. America" → "North America"
      "europe", "EUROPE", "Euorpe", "Europe " → "Europe"
      "asia pacific", "ASIA PACIFIC", "Asia-Pacific", "Asia  Pacific", "Asi Pacific" → "Asia Pacific"
      "latin america", "LATIN AMERICA", "Latin  America", "Latim America", "latin_america" → "Latin America"
      "middle east", "MIDDLE EAST", "Middle  East", "Middel East", "middle_east" → "Middle East"
      "africa", "AFRICA", "Afrca" → "Africa"
    - Strip whitespace, replace underscores with spaces, collapse multiple spaces
    - Result: 0 unmapped regions

1c. Standardize `is_verified` to boolean:
    - YES/TRUE/1/Y/yes/True → True
    - NO/FALSE/0/N/no/False → False
    - Result: True=370, False=1690

1d. Standardize `content_flag` to numeric 0/1:
    - "1", "flagged", "Flagged", "FLAGGED" → 1
    - "0", "clean", "Clean", "CLEAN" → 0
    - Empty string → NaN
    - Result: 0=1459, 1=374, NaN=227

1e. Parse `impressions` to numeric:
    - "N/A", empty strings → NaN
    - Negative strings → negative floats
    - Result: mixture of valid and invalid entries

1f. Parse `sentiment_confidence`:
    - Strip "%" characters
    - Convert to numeric
    - Empty strings → NaN

1g. Remove records with invalid followers (NaN or ≤ 0):
    - Removed: 239 rows
    - Remaining: 1821

1h. Remove records with negative likes (< 0):
    - Removed: 72 rows
    - Remaining: 1749

1i. Remove records with non-positive account_age_days (≤ 0):
    - Removed: 123 rows
    - Remaining: 1626

1j. Remove records with invalid impressions (NaN or ≤ 0):
    - Removed: 303 rows
    - Remaining: 1323

1k. Clip toxicity_score to [0.0, 1.0]:
    - Values < 0 → 0.0
    - Values > 1.0 → 1.0

1l. Clip sentiment_confidence to [0, 100]:
    - Values > 100 → 100

1m. Drop duplicate posts (same user_id AND content, keep first):
    - Removed: 41 duplicate rows
    - Remaining: 1282

1n. Drop rows with missing content_flag (NaN after standardization):
    - Removed: 145 rows
    - Remaining: 1137

1o. Standardize platform (strip whitespace, normalize casing):
    - iOS, Android, Web, Desktop

**RESULT: 1137 clean records**

Post-cleaning distributions:
    Platform: Android=403, iOS=337, Web=268, Desktop=129
    Region: North America=350, Europe=280, Asia Pacific=217, Latin America=152, Middle East=88, Africa=50
    Verified: False=934, True=203
    Content Flag: 0=907, 1=230


============================================================
STEP 2: WEIGHTED ENGAGEMENT RATE (WER)
============================================================

Formula: WER = (likes + 2 × retweets + 0.5 × replies) / impressions × 100

Interpretation of "industry-standard amplification model":
    - Likes: weight = 1 (base interaction)
    - Retweets/shares: weight = 2 (double weight — amplification)
    - Replies: weight = 0.5 (half weight — conversational)

Applied to all 1137 cleaned records.

WER Statistics:
    Mean:   40.8087
    Median:  9.5912
    Std:   133.6223
    Q1:      3.1924
    Q3:     28.2241


============================================================
STEP 3: ACCOUNT CREDIBILITY INDEX (ACI)
============================================================

Formula: ACI = log₂(followers) × (account_age_days / 365) × (sentiment_confidence / 100)

Missing sentiment_confidence handling:
    - Global median of available (non-NaN) sentiment_confidence values = 70.41
    - Impute NaN entries with 70.41 before computing ACI

ACI Statistics:
    Mean:    16.2941
    Median:   9.0594
    Std:     26.7043


============================================================
STEP 4: REGIONAL VERIFIED-ACCOUNT ENGAGEMENT PROFILE
============================================================

Filtered to verified accounts only: 203 records

Regional breakdown (verified accounts):
    Region           | Count | Median WER | Q3 WER
    -----------------+-------+------------+---------
    Latin America    |    28 |    11.4226 | 24.0205
    Europe           |    48 |     8.8883 | 31.2709
    Middle East      |    19 |     7.7428 | 29.6740
    North America    |    56 |     6.8625 | 22.5119
    Asia Pacific     |    44 |     6.6615 | 14.3754
    Africa           |     8 |     6.3874 | 10.9003

**RESULT: Region with highest median WER (verified) = Latin America (11.4226)**


============================================================
STEP 5: ANOMALOUS POST DETECTION
============================================================

Global WER statistics (entire cleaned dataset, all 1137 records):
    Q1 (25th percentile) = 3.1924
    Q3 (75th percentile) = 28.2241
    IQR = Q3 - Q1 = 25.0317
    Upper fence = Q3 + 1.5 × IQR = 28.2241 + 37.5475 = 65.7716

Global median ACI = 9.0594

Five conditions for anomalous post (ALL must hold):
    1. WER > 65.7716 (upper fence)
    2. content_flag = 1 (flagged)
    3. toxicity_score > 0.65 (strictly)
    4. account_age_days < 365 (younger than one year)
    5. ACI < 9.0594 (below global median ACI)

**RESULT: Total anomalous posts = 9**

Anomalous posts by region:
    North America: 3
    Europe:        2
    Latin America: 2
    Middle East:   1
    Africa:        1
    Asia Pacific:  0


============================================================
STEP 6: REGIONAL THREAT INDEX (RTI)
============================================================

Formula: RTI = (anomalous_in_region / total_in_region) × mean_toxicity_of_flagged_in_region × 1000

"Flagged" = content_flag = 1 within that region.

Region-by-region calculation:

Africa:
    anomalous_in_region = 1
    total_in_region = 50
    flagged_in_region = 19
    mean_toxicity_flagged = 0.710411
    RTI = (1/50) × 0.710411 × 1000 = 14.2082

Latin America:
    anomalous_in_region = 2
    total_in_region = 152
    flagged_in_region = 30
    mean_toxicity_flagged = 0.650597
    RTI = (2/152) × 0.650597 × 1000 = 8.5605

Middle East:
    anomalous_in_region = 1
    total_in_region = 88
    flagged_in_region = 18
    mean_toxicity_flagged = 0.694089
    RTI = (1/88) × 0.694089 × 1000 = 7.8874

North America:
    anomalous_in_region = 3
    total_in_region = 350
    flagged_in_region = 59
    mean_toxicity_flagged = 0.676488
    RTI = (3/350) × 0.676488 × 1000 = 5.7985

Europe:
    anomalous_in_region = 2
    total_in_region = 280
    flagged_in_region = 59
    mean_toxicity_flagged = 0.631486
    RTI = (2/280) × 0.631486 × 1000 = 4.5106

Asia Pacific:
    anomalous_in_region = 0
    total_in_region = 217
    flagged_in_region = 45
    mean_toxicity_flagged = 0.650027
    RTI = (0/217) × 0.650027 × 1000 = 0.0000

**RTI Rankings (descending):**
    1. Africa:        14.2082
    2. Latin America:  8.5605
    3. Middle East:    7.8874
    4. North America:  5.7985
    5. Europe:         4.5106
    6. Asia Pacific:   0.0000


============================================================
STEP 7: PLATFORM RISK SCORE (PRS)
============================================================

Formula: PRS = mean_WER_flagged × CV_WER × (pct_flagged / 100)

Where:
    mean_WER_flagged = mean WER of posts with content_flag=1 on that platform
    CV_WER = sample std(WER) / mean(WER) across ALL posts on that platform
    pct_flagged = (flagged_count / total_count) × 100

Platform-by-platform calculation:

iOS:
    total_posts = 337
    flagged_posts = 67
    mean_WER_flagged = 40.6810
    CV_WER = 3.9270
    pct_flagged = 19.8813%
    PRS = 40.6810 × 3.9270 × (19.8813 / 100) = 31.7613

Android:
    total_posts = 403
    flagged_posts = 75
    mean_WER_flagged = 39.7114
    CV_WER = 2.9924
    pct_flagged = 18.6104%
    PRS = 39.7114 × 2.9924 × (18.6104 / 100) = 22.1156

Web:
    total_posts = 268
    flagged_posts = 62
    mean_WER_flagged = 36.6971
    CV_WER = 2.3531
    pct_flagged = 23.1343%
    PRS = 36.6971 × 2.3531 × (23.1343 / 100) = 19.9771

Desktop:
    total_posts = 129
    flagged_posts = 26
    mean_WER_flagged = 29.7784
    CV_WER = 2.4637
    pct_flagged = 20.1550%
    PRS = 29.7784 × 2.4637 × (20.1550 / 100) = 14.7867

**PRS Rankings (descending):**
    1. iOS:      31.7613
    2. Android:  22.1156
    3. Web:      19.9771
    4. Desktop:  14.7867


============================================================
STEP 8: COMPOSITE THREAT ASSESSMENT SCORE (CTAS)
============================================================

Formula: CTAS = sum(top-3 RTI) + sum(top-2 PRS)

Top 3 RTI regions:
    1. Africa:        14.2082
    2. Latin America:  8.5605
    3. Middle East:    7.8874
    Sum = 30.6561

Top 2 PRS platforms:
    1. iOS:     31.7613
    2. Android: 22.1156
    Sum = 53.8769

CTAS = 30.6561 + 53.8769 = 84.5330

**FINAL ANSWER: CTAS = 84.53**


============================================================
COMPLETE OUTPUT SUMMARY
============================================================

1. Total records after cleaning: 1137
2. Region with highest median WER (verified): Latin America = 11.4226
3. Total anomalous posts: 9
4. RTI by region (descending):
    Africa:         14.2082
    Latin America:   8.5605
    Middle East:     7.8874
    North America:   5.7985
    Europe:          4.5106
    Asia Pacific:    0.0000
5. PRS by platform (descending):
    iOS:      31.7613
    Android:  22.1156
    Web:      19.9771
    Desktop:  14.7867
6. CTAS: 84.53