2024-11-09 13:54:37,659 INFO __main__ Thread-2 (process_request_thread) : Portfolio loaded: portfolio_20241109_132032.json
2024-11-09 13:54:37,660 INFO __main__ Thread-2 (process_request_thread) : Portfolios listed successfully
2024-11-09 13:54:37,663 INFO werkzeug Thread-2 (process_request_thread) : 127.0.0.1 - - [09/Nov/2024 13:54:37] "GET /api/portfolio/load HTTP/1.1" 200 -
2024-11-09 13:54:41,543 INFO werkzeug Thread-3 (process_request_thread) : 127.0.0.1 - - [09/Nov/2024 13:54:41] "OPTIONS /api/backtest HTTP/1.1" 200 -
2024-11-09 13:54:41,848 INFO werkzeug Thread-4 (process_request_thread) : 127.0.0.1 - - [09/Nov/2024 13:54:41] "OPTIONS /api/attribution HTTP/1.1" 200 -
2024-11-09 13:54:41,852 INFO werkzeug Thread-5 (process_request_thread) : 127.0.0.1 - - [09/Nov/2024 13:54:41] "OPTIONS /api/rebalance HTTP/1.1" 200 -
2024-11-09 13:54:41,856 INFO werkzeug Thread-6 (process_request_thread) : 127.0.0.1 - - [09/Nov/2024 13:54:41] "OPTIONS /api/correlation HTTP/1.1" 200 -
2024-11-09 13:54:41,859 INFO __main__ Thread-7 (process_request_thread) : === Starting Backtest ===
2024-11-09 13:54:41,859 INFO __main__ Thread-7 (process_request_thread) : Request data: {'portfolio': [{'ticker': 'AAPL', 'weight': 40}, {'ticker': 'TSLA', 'weight': 60}], 'benchmark': 'SPY', 'startDate': '2021-11-10'}
2024-11-09 13:54:41,859 INFO __main__ Thread-7 (process_request_thread) : Processing portfolio: [{'ticker': 'AAPL', 'weight': 40}, {'ticker': 'TSLA', 'weight': 60}]
2024-11-09 13:54:41,859 INFO __main__ Thread-7 (process_request_thread) : Start date: 2021-11-10
2024-11-09 13:54:41,859 INFO __main__ Thread-7 (process_request_thread) : Benchmark: SPY
C:\Users\calvi\Downloads\react\backend.py:144: FutureWarning: 'M' is deprecated and will be removed in a future version, please use 'ME' instead.
  monthly_returns = portfolio_returns.resample('M').apply(
C:\Users\calvi\Downloads\react\backend.py:144: FutureWarning: 'M' is deprecated and will be removed in a future version, please use 'ME' instead.
  monthly_returns = portfolio_returns.resample('M').apply(
2024-11-09 13:54:41,934 INFO __main__ Thread-7 (process_request_thread) : Response data prepared successfully
2024-11-09 13:54:41,936 INFO werkzeug Thread-7 (process_request_thread) : 127.0.0.1 - - [09/Nov/2024 13:54:41] "POST /api/backtest HTTP/1.1" 200 -
2024-11-09 13:54:42,178 INFO __main__ Thread-8 (process_request_thread) : Calculating null strategy...
2024-11-09 13:54:42,186 INFO __main__ Thread-10 (process_request_thread) : Starting attribution analysis...
2024-11-09 13:54:42,256 INFO __main__ Thread-10 (process_request_thread) : Fetching benchmark data...
2024-11-09 13:54:42,304 INFO __main__ Thread-9 (process_request_thread) : Correlation calculation completed.
2024-11-09 13:54:42,306 INFO __main__ Thread-10 (process_request_thread) : Finding common dates...
2024-11-09 13:54:42,309 INFO werkzeug Thread-9 (process_request_thread) : 127.0.0.1 - - [09/Nov/2024 13:54:42] "POST /api/correlation HTTP/1.1" 200 -
2024-11-09 13:54:42,311 INFO __main__ Thread-8 (process_request_thread) : Strategy: null
2024-11-09 13:54:42,312 INFO __main__ Thread-10 (process_request_thread) : Calculating attribution...
2024-11-09 13:54:42,313 INFO __main__ Thread-8 (process_request_thread) : Final Value: $10,654.99
2024-11-09 13:54:42,313 INFO __main__ Thread-8 (process_request_thread) : Total Transaction Costs: $0.00
2024-11-09 13:54:42,313 INFO __main__ Thread-10 (process_request_thread) : Starting attribution calculation...
2024-11-09 13:54:42,314 INFO __main__ Thread-8 (process_request_thread) : Annualized Return: 2.15%
2024-11-09 13:54:42,316 INFO __main__ Thread-8 (process_request_thread) : Volatility: 38.34%
2024-11-09 13:54:42,318 INFO __main__ Thread-8 (process_request_thread) : Max Drawdown: -57.59%
2024-11-09 13:54:42,318 INFO __main__ Thread-8 (process_request_thread) : Calculating M strategy...
2024-11-09 13:54:42,446 INFO __main__ Thread-8 (process_request_thread) : Strategy: M
2024-11-09 13:54:42,446 INFO __main__ Thread-8 (process_request_thread) : Final Value: $11,033.87
2024-11-09 13:54:42,447 INFO __main__ Thread-8 (process_request_thread) : Total Transaction Costs: $12.68
2024-11-09 13:54:42,447 INFO __main__ Thread-8 (process_request_thread) : Annualized Return: 3.35%
2024-11-09 13:54:42,447 INFO __main__ Thread-8 (process_request_thread) : Volatility: 41.91%
2024-11-09 13:54:42,447 INFO __main__ Thread-8 (process_request_thread) : Max Drawdown: -59.12%
2024-11-09 13:54:42,448 INFO __main__ Thread-8 (process_request_thread) : Calculating Q strategy...
2024-11-09 13:54:42,566 INFO __main__ Thread-8 (process_request_thread) : Strategy: Q
2024-11-09 13:54:42,566 INFO __main__ Thread-8 (process_request_thread) : Final Value: $11,043.55
2024-11-09 13:54:42,567 INFO __main__ Thread-8 (process_request_thread) : Total Transaction Costs: $5.60
2024-11-09 13:54:42,567 INFO __main__ Thread-8 (process_request_thread) : Annualized Return: 3.38%
2024-11-09 13:54:42,567 INFO __main__ Thread-8 (process_request_thread) : Volatility: 40.10%
2024-11-09 13:54:42,567 INFO __main__ Thread-8 (process_request_thread) : Max Drawdown: -56.50%
2024-11-09 13:54:42,568 INFO __main__ Thread-8 (process_request_thread) : Calculating Y strategy...
2024-11-09 13:54:42,684 INFO __main__ Thread-8 (process_request_thread) : Strategy: Y
2024-11-09 13:54:42,684 INFO __main__ Thread-8 (process_request_thread) : Final Value: $10,618.30
2024-11-09 13:54:42,686 INFO __main__ Thread-8 (process_request_thread) : Total Transaction Costs: $0.14
2024-11-09 13:54:42,687 INFO __main__ Thread-8 (process_request_thread) : Annualized Return: 2.03%
2024-11-09 13:54:42,687 INFO __main__ Thread-8 (process_request_thread) : Volatility: 38.59%
2024-11-09 13:54:42,687 INFO __main__ Thread-8 (process_request_thread) : Max Drawdown: -57.86%
2024-11-09 13:54:42,691 INFO werkzeug Thread-8 (process_request_thread) : 127.0.0.1 - - [09/Nov/2024 13:54:42] "POST /api/rebalance HTTP/1.1" 200 -
2024-11-09 13:54:43,927 ERROR __main__ Thread-10 (process_request_thread) : Error in calculate_attribution: all the input array dimensions except for the concatenation axis must match exactly, but along dimension 1, the array at index 0 has size 1 and the array at index 1 has size 754
Traceback (most recent call last):
  File "C:\Users\calvi\Downloads\react\backend.py", line 638, in calculate_attribution
    model = stats.linregress(benchmark_returns, weighted_returns.sum(axis=1))
  File "C:\Users\calvi\Downloads\react\venv\lib\site-packages\scipy\stats\_stats_py.py", line 10904, in linregress
    ssxm, ssxym, _, ssym = np.cov(x, y, bias=1).flat
  File "C:\Users\calvi\Downloads\react\venv\lib\site-packages\numpy\lib\_function_base_impl.py", line 2783, in cov
    X = np.concatenate((X, y), axis=0)
ValueError: all the input array dimensions except for the concatenation axis must match exactly, but along dimension 1, the array at index 0 has size 1 and the array at index 1 has size 754
2024-11-09 13:54:43,931 ERROR __main__ Thread-10 (process_request_thread) : Error in get_attribution: all the input array dimensions except for the concatenation axis must match exactly, but along dimension 1, the array at index 0 has size 1 and the array at index 1 has size 754
Traceback (most recent call last):
  File "C:\Users\calvi\Downloads\react\backend.py", line 987, in get_attribution
    attribution = calculate_attribution(
  File "C:\Users\calvi\Downloads\react\backend.py", line 638, in calculate_attribution
    model = stats.linregress(benchmark_returns, weighted_returns.sum(axis=1))
  File "C:\Users\calvi\Downloads\react\venv\lib\site-packages\scipy\stats\_stats_py.py", line 10904, in linregress
    ssxm, ssxym, _, ssym = np.cov(x, y, bias=1).flat
  File "C:\Users\calvi\Downloads\react\venv\lib\site-packages\numpy\lib\_function_base_impl.py", line 2783, in cov
    X = np.concatenate((X, y), axis=0)
ValueError: all the input array dimensions except for the concatenation axis must match exactly, but along dimension 1, the array at index 0 has size 1 and the array at index 1 has size 754
2024-11-09 13:54:43,932 INFO werkzeug Thread-10 (process_request_thread) : 127.0.0.1 - - [09/Nov/2024 13:54:43] "[31m[1mPOST /api/attribution HTTP/1.1[0m" 400 -

from flask import Flask, request, jsonify
from flask_cors import CORS
import yfinance as yf
import pandas as pd
import numpy as np
from datetime import datetime, timedelta
import json
import os
from scipy import stats
from functools import lru_cache
import hashlib
import pickle
import empyrical as ep
import traceback
import io
import csv
import logging

app = Flask(__name__)
CORS(app)

# Configure Logging
logging.basicConfig(
    level=logging.INFO,
    format='%(asctime)s %(levelname)s %(name)s %(threadName)s : %(message)s'
)
logger = logging.getLogger(__name__)

CACHE_DIR = 'cache'
PORTFOLIO_DIR = 'portfolios'
os.makedirs(CACHE_DIR, exist_ok=True)
os.makedirs(PORTFOLIO_DIR, exist_ok=True)

# Cache functions
def get_cache_key(ticker, start_date):
    return hashlib.md5(f"{ticker}_{start_date}".encode()).hexdigest()

def save_to_cache(key, data):
    cache_file = os.path.join(CACHE_DIR, f"{key}.pkl")
    with open(cache_file, 'wb') as f:
        pickle.dump({'data': data, 'timestamp': datetime.now()}, f)
    logger.debug(f"Data saved to cache: {cache_file}")

def load_from_cache(key, max_age_hours=24):
    cache_file = os.path.join(CACHE_DIR, f"{key}.pkl")
    if os.path.exists(cache_file):
        with open(cache_file, 'rb') as f:
            cached = pickle.load(f)
            if datetime.now() - cached['timestamp'] < timedelta(hours=max_age_hours):
                logger.debug(f"Cache hit: {cache_file}")
                return cached['data']
            else:
                logger.debug(f"Cache expired: {cache_file}")
    logger.debug(f"Cache miss: {cache_file}")
    return None

# Utility functions
@lru_cache(maxsize=100)
def get_ticker_info(ticker):
    try:
        stock = yf.Ticker(ticker)
        info = stock.info
        return {
            'sector': info.get('sector', 'Unknown'),
            'industry': info.get('industry', 'Unknown'),
            'name': info.get('longName', ticker)
        }
    except Exception as e:
        logger.error(f"Failed to fetch ticker info for {ticker}: {str(e)}")
        return {'sector': 'Unknown', 'industry': 'Unknown', 'name': ticker}

def validate_ticker(ticker):
    try:
        ticker = ticker.strip().upper()
        if not ticker:
            return False
        stock = yf.Ticker(ticker)
        info = stock.info
        return bool(info)
    except Exception as e:
        logger.error(f"Ticker validation failed for {ticker}: {str(e)}")
        return False

def fetch_stock_data(ticker, start_date):
    cache_key = get_cache_key(ticker, start_date)
    cached_data = load_from_cache(cache_key)

    if cached_data is not None:
        return cached_data

    try:
        stock_data = yf.download(ticker, start=start_date, progress=False)
        if not stock_data.empty:
            save_to_cache(cache_key, stock_data)
            logger.info(f"Data fetched and cached for {ticker} from {start_date}")
        else:
            logger.warning(f"No data fetched for {ticker} from {start_date}")
        return stock_data
    except Exception as e:
        logger.error(f"Failed to fetch data for {ticker}: {str(e)}")
        return pd.DataFrame()

def clean_for_json(obj):
    if isinstance(obj, (np.integer, np.floating)):
        return float(obj) if not (np.isnan(obj) or np.isinf(obj)) else None
    elif isinstance(obj, dict):
        return {k: clean_for_json(v) for k, v in obj.items()}
    elif isinstance(obj, (list, tuple)):
        return [clean_for_json(x) for x in obj]
    elif isinstance(obj, pd.Series):
        return [clean_for_json(x) for x in obj.values]
    elif isinstance(obj, pd.DataFrame):
        return {col: clean_for_json(obj[col]) for col in obj.columns}
    elif isinstance(obj, np.ndarray):
        return [clean_for_json(x) for x in obj.flatten()]
    elif pd.isna(obj):
        return None
    return obj

# Metric calculation functions
def calculate_portfolio_metrics(portfolio_data, weights, benchmark_returns=None):
    """Calculate portfolio performance metrics with improved accuracy"""
    try:
        # Constants
        TRADING_DAYS = 252
        RISK_FREE_RATE = 0.02  # 2% annual risk-free rate
        INITIAL_INVESTMENT = 10000

        # Calculate daily returns
        returns = portfolio_data.pct_change().fillna(0)
        portfolio_returns = returns.dot(weights)

        # Calculate CAGR and end balance
        start_value = portfolio_data.iloc[0].dot(weights)
        end_value = portfolio_data.iloc[-1].dot(weights)
        years = len(portfolio_returns) / TRADING_DAYS

        # Improved CAGR calculation
        total_return = end_value / start_value - 1
        cagr = (1 + total_return) ** (1 / years) - 1
        end_balance = INITIAL_INVESTMENT * (1 + total_return)

        # Calculate monthly returns properly
        monthly_returns = portfolio_returns.resample('M').apply(
            lambda x: (1 + x).prod() - 1
        ).dropna()

        # Improved mean calculations
        arithmetic_mean_monthly = monthly_returns.mean()
        arithmetic_mean_annual = (1 + arithmetic_mean_monthly) ** 12 - 1

        # Proper geometric mean calculation
        geometric_mean_monthly = (1 + monthly_returns).prod() ** (1 / len(monthly_returns)) - 1
        geometric_mean_annual = (1 + geometric_mean_monthly) ** 12 - 1

        # Improved volatility calculations
        daily_std = portfolio_returns.std()
        annual_volatility = daily_std * np.sqrt(TRADING_DAYS)

        # Better downside deviation calculation
        downside_returns = monthly_returns[monthly_returns < 0]
        downside_deviation = np.sqrt(np.mean(downside_returns**2)) if len(downside_returns) > 0 else 0

        # Improved drawdown calculation
        cumulative_returns = (1 + portfolio_returns).cumprod()
        rolling_max = cumulative_returns.expanding().max()
        drawdowns = cumulative_returns / rolling_max - 1
        max_drawdown = drawdowns.min()

        # Improved risk-adjusted return calculations
        excess_return = cagr - RISK_FREE_RATE
        daily_excess_returns = portfolio_returns - RISK_FREE_RATE / TRADING_DAYS

        # Better Sharpe ratio calculation
        sharpe_ratio = np.sqrt(TRADING_DAYS) * daily_excess_returns.mean() / daily_std if daily_std != 0 else 0

        # Improved Sortino ratio calculation
        negative_excess_returns = daily_excess_returns[daily_excess_returns < 0]
        downside_std = np.sqrt(np.mean(negative_excess_returns**2)) * np.sqrt(TRADING_DAYS)
        sortino_ratio = excess_return / downside_std if downside_std != 0 else 0

        # Better Calmar ratio calculation
        calmar_ratio = cagr / abs(max_drawdown) if max_drawdown != 0 else np.inf

        # Improved VaR calculations
        var_historical = np.percentile(monthly_returns, 5)
        cvar = monthly_returns[monthly_returns <= var_historical].mean()

        metrics = {
            'start_balance': INITIAL_INVESTMENT,
            'end_balance': float(end_balance),
            'cagr': float(cagr),
            'annual_volatility': float(annual_volatility),
            'max_drawdown': float(max_drawdown),
            'sharpe_ratio': float(sharpe_ratio),
            'sortino_ratio': float(sortino_ratio),
            'arithmetic_mean_monthly': float(arithmetic_mean_monthly),
            'arithmetic_mean_annual': float(arithmetic_mean_annual),
            'geometric_mean_monthly': float(geometric_mean_monthly),
            'geometric_mean_annual': float(geometric_mean_annual),
            'downside_deviation': float(downside_deviation),
            'var_historical': float(var_historical),
            'cvar': float(cvar),
            'calmar_ratio': float(calmar_ratio)
        }

        # Calculate benchmark-relative metrics if benchmark provided
        if benchmark_returns is not None:
            # Ensure benchmark_returns is a Series
            if isinstance(benchmark_returns, pd.DataFrame):
                if len(benchmark_returns.columns) == 1:
                    benchmark_returns = benchmark_returns.iloc[:, 0]
                else:
                    raise ValueError('benchmark_returns has multiple columns')
            elif not isinstance(benchmark_returns, pd.Series):
                benchmark_returns = pd.Series(benchmark_returns)

            # Align data properly
            common_index = portfolio_returns.index.intersection(benchmark_returns.index)
            portfolio_returns_aligned = portfolio_returns.loc[common_index]
            benchmark_returns_aligned = benchmark_returns.loc[common_index]

            # Confirm both are Series
            if not isinstance(portfolio_returns_aligned, pd.Series):
                portfolio_returns_aligned = pd.Series(portfolio_returns_aligned)
            if not isinstance(benchmark_returns_aligned, pd.Series):
                benchmark_returns_aligned = pd.Series(benchmark_returns_aligned)

            # Improved correlation calculation
            correlation = portfolio_returns_aligned.corr(benchmark_returns_aligned)

            # Better beta calculation using covariance method
            covariance = np.cov(portfolio_returns_aligned, benchmark_returns_aligned)[0, 1]
            benchmark_variance = np.var(benchmark_returns_aligned)
            beta = covariance / benchmark_variance if benchmark_variance != 0 else 1.0

            # Calculate benchmark CAGR properly
            benchmark_cumulative = (1 + benchmark_returns_aligned).cumprod()
            benchmark_end_value = benchmark_cumulative.iloc[-1]
            benchmark_cagr = (benchmark_end_value) ** (1 / years) - 1

            # Improved alpha calculation using CAPM
            alpha = cagr - (RISK_FREE_RATE + beta * (benchmark_cagr - RISK_FREE_RATE))

            # Better R-squared calculation
            r_squared = correlation ** 2

            # Improved tracking error calculation
            tracking_diff = portfolio_returns_aligned - benchmark_returns_aligned
            tracking_error = tracking_diff.std() * np.sqrt(TRADING_DAYS)

            # Better active return calculation
            active_return = cagr - benchmark_cagr

            # Improved information ratio
            information_ratio = active_return / tracking_error if tracking_error != 0 else 0

            metrics.update({
                'benchmark_correlation': float(correlation),
                'beta': float(beta),
                'alpha': float(alpha),
                'r_squared': float(r_squared),
                'tracking_error': float(tracking_error),
                'information_ratio': float(information_ratio),
                'active_return': float(active_return),
                'benchmark_cagr': float(benchmark_cagr),
                'benchmark_end_balance': float(INITIAL_INVESTMENT * (benchmark_end_value - 1))
            })

        return metrics

    except Exception as e:
        logger.error(f"Error calculating portfolio metrics: {str(e)}", exc_info=True)
        raise ValueError(f"Error calculating portfolio metrics: {str(e)}")

def calculate_rebalancing_impact(portfolio_data, initial_weights, transaction_cost=0.001, rebalancing_frequency='M'):
    """Calculate the impact of different rebalancing strategies with transaction costs"""
    try:
        initial_investment = 10000
        results = {}

        # Define rebalancing frequencies
        frequencies = {
            'null': None,  # Buy and hold
            'M': 'M',      # Monthly
            'Q': 'Q',      # Quarterly
            'Y': 'Y'       # Yearly
        }

        # Calculate daily returns
        daily_returns = portfolio_data.pct_change().fillna(0)

        for freq_key, freq in frequencies.items():
            logger.info(f"Calculating {freq_key} strategy...")

            # Initialize portfolio
            portfolio_value = initial_investment
            current_weights = np.array(initial_weights, dtype=float)
            values = [initial_investment]
            total_cost = 0  # Track total transaction costs

            # Initialize positions (number of shares)
            initial_prices = portfolio_data.iloc[0]
            positions = current_weights * initial_investment / initial_prices

            # Track daily values
            for i in range(1, len(portfolio_data)):
                # Get current prices
                current_prices = portfolio_data.iloc[i]

                # Update positions values
                current_values = positions * current_prices
                portfolio_value = current_values.sum()

                # Calculate current weights
                current_weights = current_values / portfolio_value

                # Check if rebalancing is needed
                should_rebalance = False

                current_date = portfolio_data.index[i]
                if freq == 'M' and current_date.is_month_end:
                    should_rebalance = True
                elif freq == 'Q' and current_date.is_quarter_end:
                    should_rebalance = True
                elif freq == 'Y' and current_date.is_year_end:
                    should_rebalance = True

                if freq is not None and should_rebalance:
                    # Calculate rebalancing trades
                    target_values = portfolio_value * initial_weights
                    trades = target_values - current_values

                    # Calculate transaction costs
                    total_trades = np.abs(trades).sum()
                    cost = total_trades * transaction_cost
                    total_cost += cost

                    # Apply transaction costs
                    portfolio_value -= cost

                    # Update positions after costs
                    positions = (target_values / current_prices) * (1 - transaction_cost)

                values.append(float(portfolio_value))

            # Calculate metrics
            returns_series = pd.Series(values).pct_change().dropna()
            total_return = values[-1] / values[0] - 1
            years = len(returns_series) / 252

            # Calculate annualized return
            annualized_return = (1 + total_return) ** (1 / years) - 1 if years > 0 else 0

            # Calculate volatility
            daily_std = returns_series.std()
            annualized_volatility = daily_std * np.sqrt(252)

            # Calculate Sharpe ratio
            excess_returns = returns_series - 0.02 / 252  # Daily excess returns over 2% risk-free rate
            sharpe = np.sqrt(252) * excess_returns.mean() / daily_std if daily_std != 0 else 0

            # Calculate maximum drawdown
            cumulative = (1 + returns_series).cumprod()
            rolling_max = cumulative.expanding().max()
            drawdowns = (cumulative - rolling_max) / rolling_max
            max_drawdown = drawdowns.min()

            logger.info(f"Strategy: {freq_key}")
            logger.info(f"Final Value: ${portfolio_value:,.2f}")
            logger.info(f"Total Transaction Costs: ${total_cost:,.2f}")
            logger.info(f"Annualized Return: {annualized_return*100:.2f}%")
            logger.info(f"Volatility: {annualized_volatility*100:.2f}%")
            logger.info(f"Max Drawdown: {max_drawdown*100:.2f}%")

            results[freq_key] = {
                'final_value': float(values[-1]),
                'annualized_return': float(annualized_return),
                'volatility': float(annualized_volatility),
                'sharpe_ratio': float(sharpe),
                'max_drawdown': float(max_drawdown),
                'values': values,
                'total_cost': float(total_cost),
                'cost_percentage': float(total_cost / initial_investment * 100)
            }
        return results
    except Exception as e:
        logger.error(f"Error calculating rebalancing impact: {str(e)}")
        raise

def save_portfolio(portfolio_data):
    timestamp = datetime.now().strftime('%Y%m%d_%H%M%S')
    filename = f"portfolio_{timestamp}.json"

    filepath = os.path.join(PORTFOLIO_DIR, filename)
    with open(filepath, 'w') as f:
        json.dump(portfolio_data, f)

    logger.info(f"Portfolio saved: {filename}")
    return filename

def load_portfolio(filename):
    filepath = os.path.join(PORTFOLIO_DIR, filename)
    try:
        with open(filepath, 'r') as f:
            portfolio = json.load(f)
        logger.info(f"Portfolio loaded: {filename}")
        return portfolio
    except Exception as e:
        logger.error(f"Failed to load portfolio {filename}: {str(e)}")
        return None

def generate_csv_report(results):
    output = io.StringIO()
    writer = csv.writer(output)

    writer.writerow(['Portfolio Metrics'])
    writer.writerow(['Metric', 'Portfolio', 'Benchmark'])
    for key in results.get('portfolio_metrics', {}):
        writer.writerow([
            key,
            results['portfolio_metrics'][key],
            results['benchmark_metrics'].get(key, 'N/A')
        ])

    writer.writerow([])
    writer.writerow(['Historical Performance'])
    writer.writerow(['Date', 'Portfolio Value', 'Benchmark Value'])
    for i, date in enumerate(results['historical_performance']['dates']):
        writer.writerow([
            date,
            results['historical_performance']['portfolio_values'][i],
            results['historical_performance']['benchmark_values'][i]
        ])

    csv_content = output.getvalue()
    output.close()
    logger.info("CSV report generated successfully")
    return csv_content

def get_sector_info(ticker):
    try:
        info = get_ticker_info(ticker)
        return info.get('sector', 'Unknown')
    except Exception as e:
        logger.error(f"Failed to get sector info for {ticker}: {str(e)}")
        return 'Unknown'

def calculate_attribution(portfolio_data, weights, benchmark_data):
    try:
        logger.info("Starting attribution calculation...")
        logger.debug(f"Portfolio data shape: {portfolio_data.shape}")
        logger.debug(f"Weights length: {len(weights)}")
        logger.debug(f"Benchmark data shape: {benchmark_data.shape}")

        # Align dates
        common_dates = portfolio_data.index.intersection(benchmark_data.index)
        portfolio_data = portfolio_data.reindex(common_dates)
        benchmark_data = benchmark_data.reindex(common_dates)

        # Calculate returns
        portfolio_returns = portfolio_data.pct_change().fillna(0)
        benchmark_returns = benchmark_data.pct_change().fillna(0)

        # Calculate weighted returns
        weighted_returns = portfolio_returns.multiply(weights, axis=1)

        # Sector Attribution
        sectors = {ticker: get_sector_info(ticker) for ticker in portfolio_data.columns}

        sector_attribution = {}
        total_return = 0

        for sector in set(sectors.values()):
            sector_tickers = [t for t, s in sectors.items() if s == sector]
            if sector_tickers:
                # Calculate sector weight
                sector_weights = [weights[list(portfolio_data.columns).index(t)] for t in sector_tickers]
                sector_weight = sum(sector_weights)

                # Calculate sector returns
                sector_returns = weighted_returns[sector_tickers].sum(axis=1)
                annual_return = (1 + sector_returns).prod() ** (252 / len(sector_returns)) - 1
                contribution = annual_return * sector_weight
                total_return += contribution

                sector_attribution[sector] = {
                    'weight': float(sector_weight),
                    'return': float(annual_return),
                    'contribution': float(contribution)
                }

        # Factor Attribution
        portfolio_cumulative = (1 + weighted_returns.sum(axis=1)).cumprod()
        benchmark_cumulative = (1 + benchmark_returns).cumprod()

        # Calculate annualized returns
        years = (common_dates[-1] - common_dates[0]).days / 365.25
        portfolio_return = (portfolio_cumulative.iloc[-1]) ** (1 / years) - 1
        benchmark_return = (benchmark_cumulative.iloc[-1]) ** (1 / years) - 1

        # Calculate beta and alpha using regression
        model = stats.linregress(benchmark_returns, weighted_returns.sum(axis=1))
        beta = model.slope
        alpha = model.intercept * 252  # Annualize alpha
        r_squared = model.rvalue ** 2

        factor_attribution = {
            'market_beta': float(beta),
            'alpha': float(alpha),
            'r_squared': float(r_squared),
            'portfolio_return': float(portfolio_return),
            'benchmark_return': float(benchmark_return)
        }

        attribution = {
            'sector': sector_attribution,
            'factor': factor_attribution,
            'total_return': float(total_return)
        }

        logger.info("Attribution calculation completed:")
        logger.debug(f"Sector attribution: {sector_attribution}")
        logger.debug(f"Factor attribution: {factor_attribution}")
        logger.debug(f"Total return: {total_return}")

        return clean_for_json(attribution)

    except Exception as e:
        logger.error(f"Error in calculate_attribution: {str(e)}", exc_info=True)
        raise

def calculate_correlation_matrix(portfolio_data):
    """Calculate correlation matrix with proper handling of returns"""
    try:
        # Calculate returns first
        returns = portfolio_data.pct_change().dropna()

        # Calculate correlations using returns, not prices
        pearson_corr = returns.corr(method='pearson')
        spearman_corr = returns.corr(method='spearman')

        # Create mask for diagonal elements
        mask = ~np.eye(len(pearson_corr), dtype=bool)

        # Calculate statistics excluding self-correlations
        pearson_stats = {
            'min': float(np.min(pearson_corr.values[mask])),
            'max': float(np.max(pearson_corr.values[mask])),
            'mean': float(np.mean(pearson_corr.values[mask]))
        }

        spearman_stats = {
            'min': float(np.min(spearman_corr.values[mask])),
            'max': float(np.max(spearman_corr.values[mask])),
            'mean': float(np.mean(spearman_corr.values[mask]))
        }

        correlations = {
            'pearson': {
                'matrix': pearson_corr.to_dict(),
                'min': pearson_stats['min'],
                'max': pearson_stats['max'],
                'mean': pearson_stats['mean']
            },
            'spearman': {
                'matrix': spearman_corr.to_dict(),
                'min': spearman_stats['min'],
                'max': spearman_stats['max'],
                'mean': spearman_stats['mean']
            }
        }

        logger.info("Correlation calculation completed:")
        logger.debug(f"Pearson stats: {pearson_stats}")
        logger.debug(f"Spearman stats: {spearman_stats}")

        return clean_for_json(correlations)

    except Exception as e:
        logger.error(f"Error in calculate_correlation_matrix: {str(e)}", exc_info=True)
        raise

def calculate_attribution(portfolio_data, weights, benchmark_data):
    try:
        logger.info("Starting attribution calculation...")
        logger.debug(f"Portfolio data shape: {portfolio_data.shape}")
        logger.debug(f"Weights length: {len(weights)}")
        logger.debug(f"Benchmark data shape: {benchmark_data.shape}")

        # Align dates
        common_dates = portfolio_data.index.intersection(benchmark_data.index)
        portfolio_data = portfolio_data.reindex(common_dates)
        benchmark_data = benchmark_data.reindex(common_dates)

        # Calculate returns
        portfolio_returns = portfolio_data.pct_change().fillna(0)
        benchmark_returns = benchmark_data.pct_change().fillna(0)

        # Calculate weighted returns
        weighted_returns = portfolio_returns.multiply(weights, axis=1)

        # Sector Attribution
        sectors = {ticker: get_sector_info(ticker) for ticker in portfolio_data.columns}

        sector_attribution = {}
        total_return = 0

        for sector in set(sectors.values()):
            sector_tickers = [t for t, s in sectors.items() if s == sector]
            if sector_tickers:
                # Calculate sector weight
                sector_weights = [weights[list(portfolio_data.columns).index(t)] for t in sector_tickers]
                sector_weight = sum(sector_weights)

                # Calculate sector returns
                sector_returns = weighted_returns[sector_tickers].sum(axis=1)
                annual_return = (1 + sector_returns).prod() ** (252 / len(sector_returns)) - 1
                contribution = annual_return * sector_weight
                total_return += contribution

                sector_attribution[sector] = {
                    'weight': float(sector_weight),
                    'return': float(annual_return),
                    'contribution': float(contribution)
                }

        # Factor Attribution
        portfolio_cumulative = (1 + weighted_returns.sum(axis=1)).cumprod()
        benchmark_cumulative = (1 + benchmark_returns).cumprod()

        # Calculate annualized returns
        years = (common_dates[-1] - common_dates[0]).days / 365.25
        portfolio_return = (portfolio_cumulative.iloc[-1]) ** (1 / years) - 1
        benchmark_return = (benchmark_cumulative.iloc[-1]) ** (1 / years) - 1

        # Calculate beta and alpha using regression
        model = stats.linregress(benchmark_returns, weighted_returns.sum(axis=1))
        beta = model.slope
        alpha = model.intercept * 252  # Annualize alpha
        r_squared = model.rvalue ** 2

        factor_attribution = {
            'market_beta': float(beta),
            'alpha': float(alpha),
            'r_squared': float(r_squared),
            'portfolio_return': float(portfolio_return),
            'benchmark_return': float(benchmark_return)
        }

        attribution = {
            'sector': sector_attribution,
            'factor': factor_attribution,
            'total_return': float(total_return)
        }

        logger.info("Attribution calculation completed:")
        logger.debug(f"Sector attribution: {sector_attribution}")
        logger.debug(f"Factor attribution: {factor_attribution}")
        logger.debug(f"Total return: {total_return}")

        return clean_for_json(attribution)

    except Exception as e:
        logger.error(f"Error in calculate_attribution: {str(e)}", exc_info=True)
        raise

def compare_portfolios(portfolios, start_date):
    try:
        logger.info("Starting portfolio comparison...")
        results = {}
        all_data = pd.DataFrame()

        unique_tickers = set()
        for portfolio in portfolios:
            unique_tickers.update([stock['ticker'] for stock in portfolio['portfolio']])

        for ticker in unique_tickers:
            stock_data = fetch_stock_data(ticker, start_date)
            if stock_data is not None and not stock_data.empty:
                all_data[ticker] = stock_data['Adj Close']
            else:
                logger.warning(f"No data available for {ticker} during comparison.")

        for idx, portfolio_data in enumerate(portfolios):
            portfolio_name = portfolio_data.get('name', f'Portfolio {idx + 1}')
            portfolio_stocks = portfolio_data['portfolio']

            tickers = [stock['ticker'] for stock in portfolio_stocks]
            weights = [float(stock['weight']) / 100 for stock in portfolio_stocks]
            portfolio_price_data = all_data[tickers].dropna()

            portfolio_returns = portfolio_price_data.pct_change().dropna()
            portfolio_cumulative = (1 + portfolio_returns.dot(weights)).cumprod()

            metrics = calculate_portfolio_metrics(portfolio_price_data, weights)

            results[portfolio_name] = {
                'metrics': metrics,
                'cumulative_returns': portfolio_cumulative.tolist(),
                'weights': dict(zip(tickers, weights))
            }

        logger.info("Portfolio comparison completed.")
        return {
            'success': True,
            'comparison_results': results,
            'dates': all_data.index.strftime('%Y-%m-%d').tolist()
        }

    except Exception as e:
        logger.error(f"Error in compare_portfolios: {str(e)}", exc_info=True)
        return {
            'success': False,
            'error': str(e)
        }

# API Endpoints

@app.route('/api/validate-ticker', methods=['POST'])
def validate_ticker_endpoint():
    data = request.json
    ticker = data.get('ticker', '')
    is_valid = validate_ticker(ticker)
    logger.info(f"Ticker validation for '{ticker}': {is_valid}")
    return jsonify({
        'success': True,
        'valid': is_valid
    })

@app.route('/api/backtest', methods=['POST'])
def backtest():
    try:
        logger.info("=== Starting Backtest ===")
        data = request.json
        logger.info(f"Request data: {data}")

        portfolio = data['portfolio']
        start_date = data.get('startDate')
        benchmark = data.get('benchmark', 'SPY')

        logger.info(f"Processing portfolio: {portfolio}")
        logger.info(f"Start date: {start_date}")
        logger.info(f"Benchmark: {benchmark}")

        # Get portfolio data
        portfolio_data = pd.DataFrame()
        start_date = pd.to_datetime(start_date)

        # Fetch data using fetch_stock_data and calculate weights
        stock_data_dict = {}
        weights = []
        for stock in portfolio:
            ticker = stock['ticker'].strip().upper()
            weight = float(stock['weight']) / 100

            stock_data = fetch_stock_data(ticker, start_date)
            if stock_data is None or stock_data.empty:
                logger.error(f"No data available for {ticker}")
                return jsonify({
                    'success': False,
                    'error': f'No data available for {ticker}'
                }), 400

            stock_data_dict[ticker] = stock_data['Adj Close']
            weights.append(weight)

        # Validate that weights sum to 1
        total_weight = sum(weights)
        if not np.isclose(total_weight, 1.0):
            logger.error(f"Portfolio weights sum to {total_weight * 100:.2f}%, which is not 100%.")
            return jsonify({
                'success': False,
                'error': f'Portfolio weights must sum to 100%. Current sum: {total_weight * 100:.2f}%'
            }), 400

        # Fetch benchmark data
        benchmark_data = fetch_stock_data(benchmark, start_date)
        if benchmark_data is None or benchmark_data.empty:
            logger.error(f"No data available for benchmark {benchmark}")
            return jsonify({
                'success': False,
                'error': f'No data available for benchmark {benchmark}'
            }), 400
        benchmark_data = benchmark_data['Adj Close']

        # Create aligned portfolio data
        common_dates = None
        for ticker, data in stock_data_dict.items():
            if common_dates is None:
                common_dates = data.index
            else:
                common_dates = common_dates.intersection(data.index)

        # Add benchmark dates to intersection
        common_dates = common_dates.intersection(benchmark_data.index)

        # Align all data to common dates
        for ticker in stock_data_dict:
            portfolio_data[ticker] = stock_data_dict[ticker].reindex(common_dates)
        benchmark_data = benchmark_data.reindex(common_dates)

        # Drop any remaining NaNs
        portfolio_data = portfolio_data.dropna()
        benchmark_data = benchmark_data.dropna()

        if portfolio_data.empty or benchmark_data.empty:
            logger.error("Insufficient overlapping data between portfolio and benchmark.")
            return jsonify({
                'success': False,
                'error': 'Insufficient overlapping data between portfolio and benchmark.'
            }), 400

        # Calculate returns
        portfolio_returns = portfolio_data.pct_change().fillna(0)
        benchmark_returns = benchmark_data.pct_change().fillna(0)  # Already a Series

        # Calculate metrics
        portfolio_metrics = calculate_portfolio_metrics(
            portfolio_data=portfolio_data,
            weights=weights,
            benchmark_returns=benchmark_returns
        )

        # Create benchmark DataFrame with proper dimensions
        benchmark_df = pd.DataFrame()
        benchmark_df[benchmark] = benchmark_data

        benchmark_metrics = calculate_portfolio_metrics(
            portfolio_data=benchmark_df,
            weights=[1.0],
            benchmark_returns=None
        )

        # Calculate historical performance
        initial_investment = 10000
        portfolio_values = initial_investment * (1 + portfolio_returns.dot(weights)).cumprod()
        benchmark_values = initial_investment * (1 + benchmark_returns).cumprod()

        # Ensure both series have the same index
        common_dates = portfolio_values.index.intersection(benchmark_values.index)
        portfolio_values = portfolio_values.reindex(common_dates)
        benchmark_values = benchmark_values.reindex(common_dates)

        response_data = {
            'success': True,
            'portfolio_metrics': clean_for_json(portfolio_metrics),
            'benchmark_metrics': clean_for_json(benchmark_metrics),
            'historical_performance': {
                'dates': common_dates.strftime('%Y-%m-%d').tolist(),
                'portfolio_values': clean_for_json(portfolio_values.values),  # Use .values to get numpy array
                'benchmark_values': clean_for_json(benchmark_values.values),  # Use .values to get numpy array
                'initial_investment': initial_investment
            }
        }

        logger.info("Response data prepared successfully")
        return jsonify(response_data)

    except Exception as e:
        logger.error(f"Error in backtest: {str(e)}", exc_info=True)
        return jsonify({
            'success': False,
            'error': str(e)
        }), 400

@app.route('/api/portfolio/save', methods=['POST'])
def save_portfolio_endpoint():
    try:
        portfolio_data = request.json
        filename = save_portfolio(portfolio_data)
        return jsonify({
            'success': True,
            'filename': filename
        })
    except Exception as e:
        logger.error(f"Error in save_portfolio_endpoint: {str(e)}", exc_info=True)
        return jsonify({
            'success': False,
            'error': str(e)
        }), 400

@app.route('/api/portfolio/load', methods=['GET'])
def list_portfolios():
    try:
        portfolios = []
        for filename in os.listdir(PORTFOLIO_DIR):
            if filename.endswith('.json'):
                portfolio = load_portfolio(filename)
                if portfolio:
                    portfolios.append({
                        'filename': filename,
                        'portfolio': portfolio
                    })
        logger.info("Portfolios listed successfully")
        return jsonify({
            'success': True,
            'portfolios': portfolios
        })
    except Exception as e:
        logger.error(f"Error in list_portfolios: {str(e)}", exc_info=True)
        return jsonify({
            'success': False,
            'error': str(e)
        }), 400

@app.route('/api/export/csv', methods=['POST'])
def export_csv():
    try:
        results = request.json
        csv_data = generate_csv_report(results)
        logger.info("CSV export successful")
        return jsonify({
            'success': True,
            'csv_data': csv_data
        })
    except Exception as e:
        logger.error(f"Error in export_csv: {str(e)}", exc_info=True)
        return jsonify({
            'success': False,
            'error': str(e)
        }), 400

@app.route('/api/attribution', methods=['POST'])
def get_attribution():
    try:
        logger.info("Starting attribution analysis...")
        data = request.json
        portfolio = data['portfolio']
        start_date = pd.to_datetime(data.get('startDate'))
        benchmark = data.get('benchmark', 'SPY')

        portfolio_data = pd.DataFrame()
        for stock in portfolio:
            ticker = stock['ticker'].strip().upper()
            stock_data = fetch_stock_data(ticker, start_date)
            if stock_data is None or stock_data.empty:
                logger.error(f"No data available for {ticker}")
                return jsonify({
                    'success': False,
                    'error': f'No data available for {ticker}'
                }), 400
            portfolio_data[ticker] = stock_data['Adj Close']

        logger.info("Fetching benchmark data...")
        benchmark_data = fetch_stock_data(benchmark, start_date)['Adj Close']
        if benchmark_data.empty:
            logger.error(f"No data available for benchmark {benchmark}")
            return jsonify({
                'success': False,
                'error': f'No data available for benchmark {benchmark}'
            }), 400

        # Find common dates
        logger.info("Finding common dates...")
        common_dates = portfolio_data.index.intersection(benchmark_data.index)

        if len(common_dates) == 0:
            logger.error("No overlapping dates found for the selected securities")
            return jsonify({
                'success': False,
                'error': 'No overlapping dates found for the selected securities'
            }), 400

        portfolio_data = portfolio_data.reindex(common_dates).dropna()
        benchmark_data = benchmark_data.reindex(common_dates).dropna()

        if portfolio_data.empty or benchmark_data.empty:
            logger.error("Insufficient overlapping data after reindexing.")
            return jsonify({
                'success': False,
                'error': 'Insufficient overlapping data after reindexing.'
            }), 400

        weights = [float(stock['weight']) / 100 for stock in portfolio]

        # Validate that weights sum to 1
        total_weight = sum(weights)
        if not np.isclose(total_weight, 1.0):
            logger.error(f"Portfolio weights sum to {total_weight * 100:.2f}%, which is not 100%.")
            return jsonify({
                'success': False,
                'error': f'Portfolio weights must sum to 100%. Current sum: {total_weight * 100:.2f}%'
            }), 400

        logger.info("Calculating attribution...")
        attribution = calculate_attribution(
            portfolio_data=portfolio_data,
            weights=np.array(weights, dtype=float),
            benchmark_data=benchmark_data
        )

        return jsonify({
            'success': True,
            'attribution': attribution
        })

    except Exception as e:
        logger.error(f"Error in get_attribution: {str(e)}", exc_info=True)
        return jsonify({
            'success': False,
            'error': str(e)
        }), 400

@app.route('/api/rebalance', methods=['POST'])
def analyze_rebalancing():
    try:
        data = request.json
        portfolio = data['portfolio']
        start_date = data.get('startDate')
        start_date = pd.to_datetime(start_date)

        portfolio_data = pd.DataFrame()
        weights = []

        for stock in portfolio:
            ticker = stock['ticker'].strip().upper()
            weight = float(stock['weight']) / 100
            stock_data = fetch_stock_data(ticker, start_date)
            if stock_data is None or stock_data.empty:
                logger.error(f"No data available for {ticker}")
                return jsonify({
                    'success': False,
                    'error': f'No data available for {ticker}'
                }), 400
            portfolio_data[ticker] = stock_data['Adj Close']
            weights.append(weight)

        # Validate that weights sum to 1
        total_weight = sum(weights)
        if not np.isclose(total_weight, 1.0):
            logger.error(f"Portfolio weights sum to {total_weight * 100:.2f}%, which is not 100%.")
            return jsonify({
                'success': False,
                'error': f'Portfolio weights must sum to 100%. Current sum: {total_weight * 100:.2f}%'
            }), 400

        weights = np.array(weights)

        rebalancing_results = calculate_rebalancing_impact(portfolio_data, weights)

        return jsonify({
            'success': True,
            'rebalancing_results': rebalancing_results
        })

    except Exception as e:
        logger.error(f"Error in analyze_rebalancing: {str(e)}", exc_info=True)
        return jsonify({
            'success': False,
            'error': str(e)
        }), 400

@app.route('/api/cache/clear', methods=['POST'])
def clear_cache():
    try:
        for file in os.listdir(CACHE_DIR):
            if file.endswith('.pkl'):
                os.remove(os.path.join(CACHE_DIR, file))
                logger.debug(f"Cache file removed: {file}")
        logger.info("Cache cleared successfully")
        return jsonify({
            'success': True,
            'message': 'Cache cleared successfully'
        })
    except Exception as e:
        logger.error(f"Error in clear_cache: {str(e)}", exc_info=True)
        return jsonify({
            'success': False,
            'error': str(e)
        }), 400

@app.route('/api/cache/status', methods=['GET'])
def cache_status():
    try:
        cache_files = [f for f in os.listdir(CACHE_DIR) if f.endswith('.pkl')]
        total_size = sum(os.path.getsize(os.path.join(CACHE_DIR, f)) for f in cache_files)

        logger.info("Cache status retrieved successfully")
        return jsonify({
            'success': True,
            'status': {
                'cached_items': len(cache_files),
                'total_size_mb': round(total_size / (1024 * 1024), 2),
                'cache_dir': CACHE_DIR
            }
        })
    except Exception as e:
        logger.error(f"Error in cache_status: {str(e)}", exc_info=True)
        return jsonify({
            'success': False,
            'error': str(e)
        }), 400

@app.route('/api/compare', methods=['POST'])
def compare_portfolios_endpoint():
    try:
        data = request.json
        portfolios = data['portfolios']
        start_date = data.get('startDate')
        start_date = pd.to_datetime(start_date)

        results = compare_portfolios(portfolios, start_date)
        return jsonify(results)

    except Exception as e:
        logger.error(f"Error in compare_portfolios_endpoint: {str(e)}", exc_info=True)
        return jsonify({
            'success': False,
            'error': str(e)
        }), 400

def calculate_correlation_matrix(portfolio_data):
    """Calculate correlation matrix with proper handling of returns"""
    try:
        # Calculate returns first
        returns = portfolio_data.pct_change().dropna()

        # Calculate correlations using returns, not prices
        pearson_corr = returns.corr(method='pearson')
        spearman_corr = returns.corr(method='spearman')

        # Create mask for diagonal elements
        mask = ~np.eye(len(pearson_corr), dtype=bool)

        # Calculate statistics excluding self-correlations
        pearson_stats = {
            'min': float(np.min(pearson_corr.values[mask])),
            'max': float(np.max(pearson_corr.values[mask])),
            'mean': float(np.mean(pearson_corr.values[mask]))
        }

        spearman_stats = {
            'min': float(np.min(spearman_corr.values[mask])),
            'max': float(np.max(spearman_corr.values[mask])),
            'mean': float(np.mean(spearman_corr.values[mask]))
        }

        correlations = {
            'pearson': {
                'matrix': pearson_corr.to_dict(),
                'min': pearson_stats['min'],
                'max': pearson_stats['max'],
                'mean': pearson_stats['mean']
            },
            'spearman': {
                'matrix': spearman_corr.to_dict(),
                'min': spearman_stats['min'],
                'max': spearman_stats['max'],
                'mean': spearman_stats['mean']
            }
        }

        logger.info("Correlation calculation completed.")
        logger.debug(f"Pearson stats: {pearson_stats}")
        logger.debug(f"Spearman stats: {spearman_stats}")

        return clean_for_json(correlations)

    except Exception as e:
        logger.error(f"Error in calculate_correlation_matrix: {str(e)}", exc_info=True)
        raise

@app.route('/api/correlation', methods=['POST'])
def get_correlation():
    try:
        data = request.json
        portfolio = data['portfolio']
        start_date = data.get('startDate')
        start_date = pd.to_datetime(start_date)

        portfolio_data = pd.DataFrame()
        for stock in portfolio:
            ticker = stock['ticker'].strip().upper()
            stock_data = fetch_stock_data(ticker, start_date)
            if stock_data is None or stock_data.empty:
                logger.error(f"No data available for {ticker}")
                return jsonify({
                    'success': False,
                    'error': f'No data available for {ticker}'
                }), 400
            portfolio_data[ticker] = stock_data['Adj Close']

        if portfolio_data.empty:
            logger.error("No data available for correlation calculation")
            return jsonify({
                'success': False,
                'error': 'No data available for correlation calculation'
            }), 400

        correlations = calculate_correlation_matrix(portfolio_data)

        return jsonify({
            'success': True,
            'correlations': correlations
        })

    except Exception as e:
        logger.error(f"Error in get_correlation: {str(e)}", exc_info=True)
        return jsonify({
            'success': False,
            'error': str(e)
        }), 400

if __name__ == '__main__':
    app.run(debug=True)