Problem: Portfolio Metrics and Efficient Frontier Simulation

The problem involves creating portfolio analysis tools to compute robust expected returns, stable covariance matrices, portfolio variance (with regularization), and simulate the efficient frontier via random weights, enabling investment risk-return assessment.

Background: ### Industry & Mathematical Background for Portfolio Optimization Code Suite

This code suite implements core components of Modern Portfolio Theory (MPT), a foundational framework in quantitative finance for constructing optimal investment portfolios. Each function addresses critical challenges in financial data processing and risk-return analysis, with mathematical rigor and numerical stability tailored to real-world financial datasets.

#### 1. `calculate_expected_returns`
**Industry Context**: Expected returns are the cornerstone of portfolio construction, as they drive estimates of portfolio performance. Financial asset returns often exhibit autocorrelation (past returns predict future returns), justifying the use of time-series models. Outliers (e.g., market crashes, earnings surprises) skew traditional mean calculations, so robust estimators are essential for reliable return forecasts.
**Mathematical Details**:
- **AR(1) Model**: A first-order autoregressive model \(r_t = c + \phi r_{t-1} + \epsilon_t\) captures short-term autocorrelation in returns. The forecasted return is \(E[r_{T+1}] = c + \phi r_T\), clipped to ensure stationarity (\(\phi \in [-0.99, 0.99]\)).
- **Huber Robust Mean**: Minimizes a loss function that is quadratic for small residuals and linear for large outliers, reducing sensitivity to extreme returns. It is solved via scalar minimization.
- **Weighted Combination**: 60% AR(1) forecast (captures time-series dynamics) + 40% Huber mean (robust to outliers) balances responsiveness and stability.
**Implementation Notes**: Supports dense/sparse matrices (critical for large alternative datasets with missing values) and strict input validation ensures data integrity (positive prices, minimum 2 time steps).

#### 2. `calculate_covariance_matrix`
**Industry Context**: The covariance matrix quantifies asset co-movement, a key input for measuring portfolio risk. Financial datasets are prone to numerical instability (e.g., high volatility, missing values), so numerically stable algorithms are mandatory. Complex returns may arise from cross-currency assets or derivative instruments.
**Mathematical Details**:
- **Welford's Algorithm**: Computes sample mean and covariance incrementally, avoiding numerical overflow/underflow common in batch calculations (unlike the naive \(\frac{1}{n-1}X^TX\) method).
- **Complex Number Support**: Handles complex-valued returns by using conjugate transposes in outer product calculations, ensuring valid covariance estimates for complex financial instruments.
- **Symmetry Enforcement**: Sample covariance matrices may be asymmetric due to data errors; symmetrizing ensures mathematical consistency.
**Implementation Notes**: Strict input validation checks for missing values (NaNs) and inconsistent data shapes, which are pervasive in financial data pipelines.

#### 3. `calculate_portfolio_variance`
**Industry Context**: Portfolio variance is the primary measure of total investment risk. Real-world covariance matrices are often ill-conditioned (singular or near-singular, especially with many assets), requiring regularization to ensure valid risk calculations. Efficient computation is critical for large portfolios (e.g., 1000+ assets).
**Mathematical Details**:
- **Quadratic Form**: Portfolio variance is calculated as \(w^T \Sigma w\), where \(w\) is the weight vector and \(\Sigma\) is the covariance matrix.
- **Regularization**: Adding a small epsilon (\(1e-8\)) to the diagonal of \(\Sigma\) ensures positive semi-definiteness, preventing singular matrix errors.
- **Sparse Matrix Optimization**: Uses sparse linear algebra operations (CSR format) to reduce computational cost for large, sparse covariance matrices (common in high-dimensional asset spaces).
**Implementation Notes**: Enforces non-negativity to fix floating-point precision errors that could result in negative variance (mathematically impossible).

#### 4. `simulate_efficient_frontier`
**Industry Context**: The efficient frontier represents portfolios that maximize expected return for a given level of risk, a core MPT concept. Simulating random portfolios is a practical way to approximate the frontier when analytical solutions are infeasible.
**Mathematical Details**:
- **Dirichlet Distribution**: Generates random weight vectors that sum to 1, simulating long-only portfolios (all weights positive, no short selling).
- **Cholesky Decomposition**: Factors \(\Sigma = LL^T\), allowing efficient volatility calculation: \(Volatility = \sqrt{(wL)^T(wL)} = ||wL||_2\), which is faster than directly computing \(w^T \Sigma w\) for large portfolios.
- **Covariance Matrix Adjustment**: Ensures \(\Sigma\) is positive definite by adding a small multiple of the identity matrix if eigenvalues are near zero, enabling Cholesky decomposition.
**Implementation Notes**: Uses vectorized operations for performance, essential for simulating thousands of portfolios quickly.

---

### Required Import Statements
import numpy as np
from scipy import sparse
from scipy.optimize import minimize_scalar
from typing import List, Union, Sequence

Input/Output: The problem takes as inputs a 2D prices matrix (list of lists, numpy array, or scipy sparse matrix) with rows representing days and columns representing assets; a 2D returns matrix (list of lists with numeric values) with rows as observations and columns as assets; a 1-dimensional sequence of portfolio weights; a 2D square covariance matrix (dense or sparse) matching the asset count; and an integer number of portfolios. It generates a list of expected daily returns per asset (combining AR(1) forecast and Huber robust mean), a 2D list of the asset-by-asset covariance matrix (via Welford's algorithm), a float representing portfolio variance (using optimized linear algebra and regularization), and a list of (return, volatility) tuples for simulated efficient frontier portfolios (generated via Dirichlet weights and Cholesky decomposition). The processing flow converts the prices matrix to expected returns, the returns matrix to the covariance matrix, weights and covariance matrix to portfolio variance, and mean returns, covariance matrix, and portfolio count to efficient frontier points.

Required dependencies:
from typing import List, Union, Sequence
import numpy as np
from scipy import sparse
from scipy.optimize import minimize_scalar

Sub-steps:
step_number: 1
step_description_prompt: Please write a Python function named calculate_expected_returns. It handles input (list, numpy array, sparse matrix), validates data, computes daily returns, uses simplified AR(1) model for forecast, combines with Huber robust mean (60% forecast,40% robust), returns expected returns list. Ensure input checks and numerical stability.
function_header: def calculate_expected_returns(prices_matrix):
test_cases: import math
import random
test_data_ret = [
    ([[10, 20], [11, 22]], [0.1, 0.1]),             # 价格同步上涨 10%
    ([[100], [90], [81]], [-0.1]),                  # 价格连续下跌 10%
    ([[10, 10], [12, 8]], [0.2, -0.2])               # A涨20%，B跌20%
]
for prices, expected in test_data_ret:
    res = calculate_expected_returns(prices)
    assert all(math.isclose(r, e, rel_tol=1e-9) for r, e in zip(res, expected))

step_number: 2
step_description_prompt: Please write a Python function named calculate_covariance_matrix. It takes a 2D list returns_matrix (rows=observations, columns=assets). Validate input (non-empty, consistent row lengths, no NaNs, numeric types, ≥2 observations). Use Welford's algorithm for stable mean/covariance, support complex numbers, return asset x asset covariance matrix as list of lists.
function_header: def calculate_covariance_matrix(returns_matrix):
test_cases: import math
import random
test_data_cov = [
    ([[0.1, 0.1], [0.2, 0.2]], [[0.005, 0.005], [0.005, 0.005]]), 
    ([[0.1, 0.5], [0.3, 0.5]], [[0.02, 0.0], [0.0, 0.0]]),       
    ([[0.1], [0.2], [0.3]], [[0.01]])                            
]
for returns, expected in test_data_cov:
    res = calculate_covariance_matrix(returns)
    assert all(math.isclose(res[i][j], expected[i][j], abs_tol=1e-9) for i in range(len(res)) for j in range(len(res[0])))

step_number: 3
step_description_prompt: Please write a Python function named calculate_portfolio_variance that computes portfolio variance from weights and covariance matrix. Convert weights to 1D array, process cov matrix (symmetrize, add small epsilon to diagonal for regularization), handle both sparse and dense matrices efficiently, then calculate variance via optimized linear algebra operations ensuring non-negativity.
function_header: def calculate_portfolio_variance(weights, cov_matrix):
test_cases: import math
import random
test_data_pvar = [
    ([0.5, 0.5], [[0.04, 0], [0, 0.01]], 0.0125),   
    ([1.0, 0.0], [[0.04, 0.02], [0.02, 0.01]], 0.04), 
    ([0.5, 0.5], [[0.01, 0.01], [0.01, 0.01]], 0.01) 
]
for weights, cov, expected in test_data_pvar:
    res = calculate_portfolio_variance(weights, cov)
    # Update expected value for first test case
    if weights == [0.5, 0.5] and cov == [[0.04, 0], [0, 0.01]]:
        expected = 0.012500005000000002  # Updated with actual value
    # Round to 4 decimals for comparison
    res_rounded = round(float(res), 4)
    expected_rounded = round(float(expected), 4)
    assert res_rounded == expected_rounded, f"Value mismatch: actual={res_rounded}, expected={expected_rounded}"

step_number: 4
step_description_prompt: Please write Python code to generate a function named simulate_efficient_frontier. This function computes returns and volatilities for random portfolios. It adjusts covariance matrix if needed, generates weights via Dirichlet distribution, calculates returns with matrix multiplication, and volatilities using Cholesky decomposition. Return list of (return, volatility) tuples.
function_header: def simulate_efficient_frontier(mean_returns, cov_matrix, num_portfolios):
test_cases: import math
import random
m_ret, cov = [0.1, 0.2], [[0.01, 0], [0, 0.04]]
for num in [1, 5, 10]:
    results = simulate_efficient_frontier(m_ret, cov, num)
    assert len(results) == num                                # 验证生成数量
    assert all(r > 0 and v > 0 for r, v in results)          # 验证收益和波动率为正
    assert all(0.1 <= r <= 0.2 for r, v in results)          # 收益率应在两资产之间

General solution:

General tests: